Image encoding device, image decoding device, and their control methods and programs

Abstract

To efficiently encode RAW image data having an infrared component in addition to three primary colors.SOLUTION: An image encoder encodes an image obtained by an imaging sensor having R, G, B three-primary-color filters in a visible light wavelength region, and an infrared filter, and the image encoder has: a generation unit that generates difference data indicating the difference between image data of any one of the three-primary-color filters determined according to the intensity distribution of the transmission wavelength of the infrared filter, and image data obtained through the infrared filter; and an encoding unit that encodes the respective pieces of image data of the three-primary-color filters and the difference data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image encoding technique. [Background technology]

[0002] Nowadays, imaging devices such as digital cameras and digital camcorders use CCD or CMOS sensors as their imaging elements. These sensors have a color filter array (hereafter referred to as CFA) formed on the sensor surface, and detect one color component per pixel. A typical CFA is a Bayer array, as shown in Figure 3, in which 2 x 2 pixels (one red (R), two greens (G0, G1), and one blue (B)) are periodically arranged. Naturally, a Bayer array sensor produces Bayer array image data (hereafter referred to as RAW data).

[0003] Because human vision is highly sensitive to luminance components, a typical Bayer array is configured to allocate twice as many pixels to the green component, which is abundant in luminance components, as compared to the red and blue components, as shown in Figure 3. Because RAW data only contains information for one color component per pixel, a process called demosaicing is required to generate red, blue, and green information for each pixel. Generally, the RGB signals obtained by demosaicing, or the YUV signals obtained by converting the RGB signals, are encoded and recorded. However, demosaicing results in three color components per pixel, requiring three times the amount of data compared to RAW data. Therefore, a method has been proposed in which the RAW data before demosaicing is directly encoded and recorded.

[0004] For example, Patent Document 1 discloses a method of encoding RAW data after separating it into four planes of R, G0, B, and G1. Patent Document 2 discloses a method of encoding RAW data after separating it into four planes of R, G0, B, and G1 in the same manner as Patent Document 1, and then converting it approximately into luminance (Y) and color difference (Co, Cg, Dg).

[0005] Meanwhile, sensors that can simultaneously capture color and infrared images have been developed for applications such as monitoring and inspection. These sensors differ from Figure 3 in that they have a pixel arrangement that includes IR (infrared) pixels in addition to R, G, and B, as shown in Figure 4. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2003-125209 [Patent Document 2] Japanese Patent Application Laid-Open No. 2006-121669 Summary of the Invention [Problem to be solved by the invention]

[0007] The plane conversion methods disclosed in Patent Documents 1 and 2 are based on the Bayer array shown in FIG. 3, and do not support a pixel arrangement having IR in addition to RGB, as shown in FIG.

[0008] The present invention aims to provide a technique for efficiently encoding RAW data of a pixel array that includes components outside the visible light range in addition to RGB. [Means for solving the problem]

[0009] In order to solve this problem, for example, an image coding device of the present invention has the following arrangement: An image encoding device that encodes an image obtained by an image sensor having three primary color filters of R, G, and B in the visible light wavelength region and an infrared filter, a generating means for generating difference data indicating a difference between image data of one of the three primary color filters determined according to the intensity distribution of the transmission wavelength of the infrared filter and image data obtained through the infrared filter; encoding means for encoding the image data of each of the three primary color filters and the differential data; With death, The generating means When the central transmission wavelength of the infrared filter is around 780 nm, the difference data is generated using image data of the R component of the three primary colors; When the central transmission wavelength of the infrared filter is around 870 nm, the difference data is generated using image data of the B component of the three primary colors; When the central transmission wavelength of the infrared filter is around 940 nm, the differential data is generated using image data of the G component of the three primary colors. . [Effects of the Invention]

[0010] According to the present invention, RAW image data having an infrared component in addition to the three primary colors can be efficiently coded. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a block diagram showing an example of the configuration of an image encoding device according to a first embodiment. [Figure 2] FIG. 1 is a block diagram showing an example of the configuration of an image decoding apparatus according to a first embodiment. [Figure 3] FIG. [Figure 4] A diagram showing an RGB-IR array. [Figure 5] 3A and 3B are diagrams for explaining examples of infrared rays having three different wavelength intensity distributions according to the first embodiment, and the relationships with the wavelengths of ultraviolet rays and visible light. [Figure 6] FIG. 1 is a diagram illustrating a wavelet transform. [Figure 7] FIG. 10 is a diagram for explaining a case in which the effective wavelength range of the IR used is intermediate between IR1 and IR2 in the second embodiment. [Figure 8] A diagram showing an RGB-NIR-FIR array. [Figure 9] 10 is a diagram for explaining a case in which the effective wavelength region of the IR used is between IR1 and IR2 and closer to IR1 in the second embodiment. FIG. [Figure 10] FIG. 11 is a diagram for explaining an IRH plane according to the third embodiment. [Figure 11] FIG. 10 is a hardware configuration diagram of an information processing device according to a modified example of the first embodiment. [Figure 12] 10 is a flowchart showing the procedure of an image encoding process. [Figure 13] 10 is a flowchart showing the procedure of an image decoding process. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the claimed invention. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0013] [First embodiment] FIG. 1 is a block diagram showing the configuration of an image encoding device according to the first embodiment.

[0014] 1, the image encoding device includes an imaging unit 100, a RAW encoding unit 110, and a control unit 150 that controls these units. The RAW encoding unit 110 includes a plane conversion unit 101, a frequency conversion unit 102, a quantization unit 103, and an entropy encoding unit 104. The control unit 150 is made up of a CPU, a ROM that stores programs executed by the CPU, and a RAM that the CPU uses as a work area.

[0015] In addition, in this embodiment, JPEG2000 is used as an example of the encoding method, but the type of encoding method is not particularly important.

[0016] In the above configuration, the encoding process of an input image will be described with reference to FIG.

[0017] The imaging unit 100 is a typical imaging optical unit composed of an optical lens, an aperture mechanism, a shutter mechanism, an imager, etc. The imager is a type that captures images by color separation, such as a CCD or MOS type having a color filter for color separation on its surface. This imager separates the color of the formed optical image and converts it into an electrical signal according to the amount of light. In this embodiment, the color filter for color separation of the imager will be described as a filter with an RGB-IR (Infrared Ray) arrangement as shown in FIG. 4. In the figure, R, G, and B represent three primary color filters in the visible light wavelength region, and IR represents an infrared filter.

[0018] The plane conversion unit 101 separates RAW image data with an RGB-IR pixel array into an R plane, a G plane, a B plane, and an IR plane, as shown in FIG. 4. The R plane is a plane consisting of a single R component, the B plane is a plane consisting of a single B component, the G plane is a plane consisting of a single G component, and the IR plane is a plane consisting of a single IR component. If the number of pixels in the horizontal direction of the RAW image is W and the number of pixels in the vertical direction is H, the R plane, G plane, B plane, and IR plane have the same size, with W / 2 pixels in the horizontal direction and H / 2 pixels in the vertical direction. The plane conversion unit 101 then outputs the R, G, and B planes obtained by the separation. For the IR plane, the plane conversion unit 101 generates and outputs an IRH plane, which indicates differential data between the IR plane and the color planes of the other color components, in accordance with instructions from the control unit 150.

[0019] Figure 5 shows an example of infrared light with three different wavelength intensity distributions, as well as the relationship with the wavelengths of ultraviolet light and visible light. In Figure 5, the filter with intensity (high transmittance) in the B wavelength region is a B component filter, the filter with intensity in the G wavelength region is a G component filter, and the filter with intensity in the R wavelength region is an R component filter.

[0020] The IR wavelength region includes three wavelength regions IR1, IR2, and IR3. A filter having intensity in the wavelength region IR1 is defined as a filter for the IR1 component. IR1 has a wavelength intensity distribution in a region adjacent to the R wavelength region. The wavelength region IR2 is an infrared region that is the next closest to the R wavelength region after the wavelength region IR1, and a filter having intensity in the wavelength region IR2 is defined as a filter for the IR2 component. The wavelength region IR3 has a wavelength intensity distribution in the wavelength region farthest from the R component within the IR wavelength region. A filter having intensity in this wavelength region IR3 is defined as a filter for the IR3 component. In this embodiment, the central transmission wavelength of the wavelength region IR1 is approximately 780 nm, the central transmission wavelength of the wavelength region IR2 is approximately 870 nm, and the central transmission wavelength of the wavelength region IR3 is approximately 940 nm. In this embodiment, the full width at half maximum of the wavelength intensity distribution is approximately 50 nm. The wavelength region IR1 is highly correlated with R, the wavelength region IR2 with B, and the wavelength region IR3 with G.

[0021] Assume that the control unit 150 acquires, via an operation unit (not shown), for example, information indicating that the IR filter of the imaging unit 100 has intensity in the wavelength region of the wavelength intensity distribution IR1. In this case, the control unit 150 sets the plane conversion unit 101 to calculate the difference between the IR plane and the R plane. In response to this, the plane conversion unit 101 calculates the difference plane, i.e., the IRH plane, according to the following equation (1). IRH = R -IR × ir_gain1 …(1)

[0022] As explained above, IR1 has a high correlation with R. Therefore, the plane conversion unit 101 performs gain adjustment using a gain irga1 to match the IR level to R. The plane conversion unit 101 then calculates the IRH plane, which indicates the difference between the R plane and the adjusted IR plane "IR × ir_gain1." Although the values ​​that make up the IRH plane (difference pixel values) have positive and negative signs, many small values ​​close to zero appear, and a high compression rate can be expected.

[0023] Also, assume that information indicating that the IR filter of the imaging unit 100 has intensity in the wavelength region of the wavelength intensity distribution IR2 is set to the control unit 150. In this case, the control unit 150 sets the plane division unit 101 to calculate the difference between the IR plane and the B plane. In response to this, the plane conversion unit 101 calculates the IRH plane according to the following equation (2). IRH = B -IR × ir_gain2 …(2) As explained above, IR2 has a high correlation with B. Therefore, the plane conversion unit 101 performs gain adjustment using a gain irga2 to match the IR level to B. The plane conversion unit 101 then calculates the IRH plane, which represents the difference between the B plane and the adjusted IR plane "IR × ir_gain2." Although the values ​​that make up the IRH plane (difference pixel values) have positive and negative signs, many small values ​​close to zero appear, and a high compression rate can be expected.

[0024] Furthermore, when information indicating that the IR filter of the imaging unit 100 has intensity in the wavelength region of the wavelength intensity distribution IR3 is set in the control unit 150, the control unit 150 sets the plane conversion unit 101 to calculate the difference between the IR plane and the G plane. In response to this, the plane conversion unit 101 calculates the IRH plane according to the following equation (3). IRH = G -IR × ir_gain3 …(3) As explained above, IR3 has a high correlation with G. Therefore, the plane conversion unit 101 performs gain adjustment using a gain irga3 to match the IR level to G. The plane conversion unit 101 then calculates the IRH plane, which represents the difference between the G plane and the adjusted IR plane "IR × ir_gain3." Although the values ​​that make up the IRH plane (difference pixel values) have positive and negative signs, many small values ​​close to zero appear, and a high compression rate can be expected.

[0025] As described above, the plane conversion unit 101 determines the color components for which differences are to be calculated based on the effective wavelength range of the IR filter of the imaging unit 100, and generates an IRH plane that represents the differences. Then, the plane conversion unit 101 supplies the R, G, B planes and the IRH plane to the frequency conversion unit 102.

[0026] The frequency transform unit 102 obtains multiple subbands by performing frequency transform (wavelet transform in this embodiment) on each plane supplied from the plane transform unit 101. Then, the frequency transform unit 102 sends the transform coefficients in each subband to the quantization unit 104.

[0027] Wavelet transform will now be described with reference to FIG. 6. FIG. 6 shows an example of wavelet transform, which is also adopted in JPEG2000 and performs subband decomposition at only one level (one time). As shown in FIG. 6, an input plane 400 is divided into low-frequency and high-frequency component data using a vertical low-pass filter 401 and a vertical high-pass filter 402 for the vertical component. The data extracted through the vertical low-pass filter 401 and the vertical high-pass filter 402 is downsampled at a ratio of 2:1 by downsampling circuits 403 and 404, respectively. As a result, the downsampling circuits 403 and 404 output data with half the vertical resolution. The data output from the downsampling circuits 403 and 404 is divided into low-frequency and high-frequency component data using a horizontal low-pass filter 405 and a horizontal high-pass filter 406, and a horizontal low-pass filter 407 and a horizontal high-pass filter 408. The data is further downsampled at a ratio of 2:1 by downsampling circuits 409, 410, 411, and 412, respectively. As a result, an output image 413 can be obtained. The output image 413 is composed of blocks LL, HL, LH, and HH, each obtained through the above filtering process. For simplicity, these blocks will be referred to as LL, HL, LH, and HH as needed below. L indicates the low-frequency region, and H indicates the high-frequency region. For example, HH indicates a subband with high frequency regions in both the horizontal and vertical directions. In the LL region of the output image 413, the resolution of the input image 400 is reduced by half both vertically and horizontally. The HH, HL, and LH regions are high-frequency regions. The LL, HL, LH, and HH regions shown in the figure are generally called subbands. Note that a wavelet transform may be performed recursively on the subband LL.

[0028] The quantization unit 104 quantizes the transform coefficients included in the subbands of each plane data using a predetermined quantization parameter for each subband, and supplies the quantized transform coefficients to the entropy coding unit 105.

[0029] The entropy coding unit 105 performs entropy coding, such as EBCOT (Embedded Block Coding with Optimized Truncation), on the wavelet coefficients and quantization parameters quantized by the quantization unit 104 for each subband. The entropy coding unit 105 then generates a header containing information necessary for decoding. The entropy coding unit 105 then outputs coded data following this header. Information stored in the header includes the size of the RAW image and the number of bits per component. In this embodiment, the header also includes information indicating which of the IR filters, IR1 to IR3, the IR filter of the imaging unit 100 is (or may also include information indicating which color component plane the IRH plane represents as a difference between the IR plane and the IR plane). The output destination of the entropy coding unit 105 is not particularly limited, and may be, for example, a storage medium such as a hard disk drive (HDD) or a memory card.

[0030] Next, a method for decoding the coded image data generated in the above procedure will be explained.

[0031] 2 is a block diagram of an image decoding device according to an embodiment. As shown in the figure, the image decoding device includes an entropy decoding unit 200, an inverse quantization unit 201, a frequency inverse transform unit 202, a plane inverse transform unit 203, a header analysis unit 204, and a control unit 250 that controls the entire device.

[0032] The header analysis unit 204 analyzes the header (file header) of the coded stream to be decoded. Then, the header analysis unit 204 extracts information indicating which of the IR filters, IR1 to IR3, the IR filter of the image pickup device (image pickup unit 100) used when capturing the image to be decoded is, and supplies the information to the plane inverse conversion unit 203.

[0033] The entropy decoding unit 200 decodes the coded data following the header, that is, the coded data of the wavelet coefficients and quantization parameters coded by EBCOT (Embedded Block Coding with Optimized Truncation) or the like, and transfers it to the inverse quantization unit 101.

[0034] The inverse quantization unit 201 generates frequency transform coefficients by inverse quantizing the data (quantized frequency transform coefficients) supplied from the entropy decoding unit 200 using the quantization parameter. Then, the inverse quantization unit 201 supplies the frequency transform coefficients to the frequency inverse transform unit 202.

[0035] The frequency inverse transform unit 202 performs frequency inverse transform (inverse wavelet transform in this embodiment) on the frequency transform coefficients supplied from the inverse quantization unit 201 to generate R, G, B, and IRH planes. Then, the frequency inverse transform unit 202 supplies the R, G, B, and IRH planes to the plane inverse transform unit 203.

[0036] The plane inverse transform unit 203 performs inverse plane transform on the R, G, B, and IRH planes supplied from the frequency inverse transform unit 202, reconstructs RAW image data in an RGB-IR arrangement, and outputs it. Details of the inverse plane transform will be described below.

[0037] The plane inverse conversion unit 203 generates an IR plane from the IRH plane based on information supplied from the header analysis unit 204, i.e., information indicating which of the IR filters IR1 to IR3 is used as the IR filter of the imaging element (imaging unit 100) used to capture the image to be decoded. When the IR1 filter is used The plane inverse transform unit 203 reconstructs the IR plane from the R plane and the IRH plane according to the following equation (4): IR = (R-IRH) / irgain1 …(4) When the IR2 filter is used The plane inverse transform unit 203 reconstructs the IR plane from the R plane and the IRH plane according to the following equation (5): IR = (B-IRH) / irgain2 …(5) When an IR3 filter is used The plane inverse transform unit 203 reconstructs the IR plane from the G plane and the IRH plane according to the following equation (6): IR = (G-IRH) / irgain3 …(6)

[0038] The plane inverse transform unit 203 restores the IR plane using either of the above methods. After that, the plane inverse transform unit 203 generates RAW image data by arranging the pixels in the R, G, B, and IR planes as shown in FIG.

[0039] As described above, according to this embodiment, highly efficient encoding is possible by taking the difference between the IR of RAW image data in an RGB-IR array and a color component with a high correlation before encoding. Note that in this embodiment, gain adjustment is performed on the IR to take the difference, but gain adjustment may also be performed on the RGB before taking the difference.

[0040] In the above description, the image decoding device has gains irgain1, irgain2, and orgain3 used by the image coding device. However, when the image coding device generates coded data, the used gains may be stored in the header.

[0041] [Modification of the first embodiment] An example in which the image encoding process of the first embodiment is implemented by an application program of an information processing device, such as a personal computer, will be described as a modified example of the first embodiment. 11 is a hardware configuration diagram of an information processing device according to a modified example. The information processing device includes a CPU 1101, a ROM 1102, a RAM 1103, an HDD 1104, an operation unit (keyboard, mouse, etc.) 1105, a display unit 1106, and an I / F 1107.

[0042] In the above configuration, when the device is powered on, the CPU 1101 executes a boot program in the ROM 1102, loads an OS (operating system) from the HDD 1104 into the RAM 1103, and transfers processing to the OS, thereby enabling various processes to be performed via the operation unit 1105. Then, in accordance with instructions from the user via the operation unit 1105, an image encoding application or an image decoding application is loaded from the HDD 1104 into the RAM 1103 and the application is executed under the OS, thereby causing the device to function as an image encoding device or an image decoding device.

[0043] Here, the processing procedure of the CPU 1101 when the information processing device functions as an image encoding device will be described with reference to the flowchart in Fig. 12. It is assumed that the RAW image data file of the RGB-IR array to be encoded has been received from an external imaging device via the I / F 1107 and has already been stored in the HDD 1104. It is also assumed that this file has been selected as the file to be encoded.

[0044] In S1201, the CPU 1101 reads the RAW image data file to be coded from the HDD 1104 and inputs the RAW image data. Then, in S1202, the CPU 1101 generates R, G, B, and IR planes from the input RAW image data and stores them in the RAM 103.

[0045] In step S1203, the CPU 1101 determines whether the IR filter of the imaging unit of the imaging device used when capturing the RAW image data is one of the IR1 to IR3 filters. If information specifying the IR filter is stored in the file header of the RAW image data, the CPU 1101 determines whether the IR filter is one of the IR1 to IR3 filters. Alternatively, the CPU 1101 may determine whether the IR filter is one of the IR1 to IR3 filters in response to a user selection operation.

[0046] If the CPU 1101 determines that the IR1 filter is being used, it branches the process to S1204; if it determines that the IR2 filter is being used, it branches the process to S1205; and if it determines that the IR3 filter is being used, it branches the process to S1205.

[0047] If the process branches to S1204, the CPU 1101 generates the IRH plane according to the above-described formula (1). If the process branches to S1205, the CPU 1101 generates the IRH plane according to the above-described formula (2). If the process branches to S1206, the CPU 1101 generates the IRH plane according to the above-described formula (3).

[0048] At S1207, the CPU 1101 performs frequency transform (wavelet transform in this embodiment) on each of the R, G, B, and IRH planes to generate transform coefficients. Then, at S1208, the CPU 1101 performs quantization processing on the transform coefficients of each plane. At S1209, the CPU 1101 entropy codes the quantized transform coefficients to generate coded data.

[0049] Finally, in S1210, the CPU 1101 creates a file header in the HDD 1104 that includes information necessary for decoding, such as information indicating the type of IR filter, and creates an encoded image data file by storing the encoded data of each plane (it is preferable that the encoded data of the IRH is last) in the HDD 1104 so that it follows the file header. do.

[0050] Next, the processing procedure of the CPU 1101 when the information processing device functions as an image decoding device will be described with reference to the flowchart in Fig. 13. It is assumed that the encoded image data file to be decoded in the HDD 1104 has already been selected by the user. An example will be described in which the RAW image data obtained by decoding is stored as a file in the HDD 1104. When an infrared image is to be displayed on the display unit 1106, the IR plane obtained by decoding can be displayed as a monochrome image. When an RGB color image is to be displayed, an image of the three color components R, G, and B per pixel can be generated from the R, G, and B planes obtained by decoding, and then displayed.

[0051] At S1301, the CPU 1101 analyzes the header of the encoded image data file to be decoded and determines which of the IR1 to IR3 filters was used to capture the image. Then, at S1302, the CPU 1101 reads encoded data from the encoded image data file to be decoded and performs entropy decoding to obtain quantized transform coefficients for each of the R, G, B, and IRH planes. At S1303, the CPU 1101 performs inverse quantization to obtain frequency transform coefficients for each of the R, G, B, and IRH planes. Then, at S1304, the CPU 1101 performs inverse frequency transform (inverse wavelet transform in this embodiment) on the frequency transform coefficients for each of the R, G, B, and IRH planes to restore the R, G, B, and IRH planes.

[0052] In S1305, the CPU 1101 determines which of IR1 to IR3 is the IR filter for the image to be decoded, based on the analysis result of S1301. If the CPU 1101 determines that the IR1 filter is being used, it branches the process to S1306, if it determines that the IR2 filter is being used, it branches the process to S1307, and if it determines that the IR3 filter is being used, it branches the process to S1308. If the process branches to S1306, the CPU 1101 generates an IR plane according to the previously shown formula (4). If the process branches to S1307, the CPU 1101 generates an IR plane according to the previously shown formula (5). If the process branches to S1308, the CPU 1101 generates an IR plane according to the previously shown formula (3).

[0053] In S1309, the CPU 1101 reconstructs RAW image data in an RGB-IR array by arranging the IR plane generated in any of S1306 to S1308 and the pixels of the R, G, and B planes restored in S1304 as shown in Figure 4.

[0054] Then, in S1310, the CPU 1101 stores the reconstructed RAW image data as a file on the HDD, and at this time, stores information indicating the type of IR filter in the file header.

[0055] As described above, the processing corresponding to the first embodiment can be realized by a CPU executing a computer program. It should be noted that the second and third embodiments described below can also be realized by a CPU executing a computer program.

[0056] [Second embodiment] Next, an image encoding device according to a second embodiment will be described with reference to Figures 1, 4, and 7. In this second embodiment, the configuration of the image encoding device is the same as that of the first embodiment, but the conversion process by the plane conversion unit 101 is different from that of the first embodiment. Note that other operations are assumed to be the same as those of the first embodiment, and a description thereof will be omitted.

[0057] In this second embodiment, we will explain an example in which the effective wavelength range of IR in the RGB-IR array is midway between IR1 and IR2 (the central wavelength of the intensity distribution of the transmitted wavelength is near 820 nm, which is almost in the middle of the range from 780 nm to 870 nm) as shown in Figure 7.

[0058] The plane conversion unit 101 generates R, G, and IR planes from RAW image data of an RGB-IR pixel array as shown in FIG. 4, and then calculates the IRH plane according to the following equation (7). IRH=(R+B) / 2- IR×irgain4 …(7) Here, irgain4 is the gain for adjusting the IR level to "(R+B) / 2".

[0059] The subsequent encoding process is the same as in the first embodiment, and therefore a description thereof will be omitted. However, the header of the generated encoded image data file will contain information indicating that the IR filter has the characteristics shown in Figure 4.

[0060] Furthermore, the configuration related to the decoding process may be the same as that of the first embodiment shown in Fig. 2. That is, when decoding, if the information related to IR stored in the header represents Fig. 7, the decoding device regenerates the IR plane from the restored R, B, and IRH planes according to the following equation (8). IR={(R+B) / 2-IRH} / irgain4 …(8) When the effective wavelength range of IR in RAW image data in an RGB-IR array falls between IR1 and IR2, plane conversion is performed taking into account the ratio between R, which has a strong correlation with IR1, and B, which has a strong correlation with IR2. In this embodiment, an example has been described in which the effective wavelength range is between IR1 and IR2, but when the effective wavelength range is closer to IR1 as in Figure 9, plane conversion is performed by multiplying R by a stronger ratio, for example, as in the following equation (9). IRH=(R×0.7+B×0.3)-IR×irgain5 …(9) In this case, when decoding, the IR plane can be regenerated according to the following equation (10) using the previously decoded R, B, and IRH planes. IR={(R×0.7+B×0.3)-IRH} / irgain5 …(10) The weighting coefficients shown in equations (9) and (10) differ depending on whether the effective wavelength region is closer to IR1 or IR2. Therefore, if the effective wavelength region is closer to IR2, a stronger weighting coefficient is applied to B to perform plane conversion.

[0061] Furthermore, in the second embodiment, the case where the effective wavelength range of IR is between IR1 and IR2 has been described, but the same concept can be applied to the case where the effective wavelength range of IR is between IR2 and IR3, that is, the case where the central transmission wavelength of the infrared filter is within the range of 870 nm to 940 nm.

[0062] [Third embodiment] An image encoding device according to the third embodiment will be described with reference to FIGS. 1, 8, and 10. In the third embodiment, the configuration of the image encoding device is the same as that of the first embodiment, but the conversion process performed by the plane conversion unit 101 is different from that of the first embodiment. Also, unlike the first and second embodiments, the image sensor uses two types of filters: a near-infrared filter (NIR: Near Infrared Ray) and a far-infrared filter (FIR: Far Infrared Ray). The use of an NIR filter improves visibility in dark places, and the use of an FIR filter also serves as a human presence sensor for detecting intrusion, which is expected to further improve surveillance performance.

[0063] Other operations are the same as those in the first embodiment, and therefore will not be described here. In the third embodiment, an example is shown in which IR1 is used as the effective wavelength region of NIR.

[0064] Figure 8 shows the pixel arrangement of the sensor in this third embodiment. As with Figure 4, the pixel arrangement has IR (infrared) pixels in addition to R, G, and B, but it differs in that near-infrared (NIR) and far-infrared (FIR) pixels are alternately arranged.

[0065] When the plane conversion unit 101 receives information from the control unit 150 indicating that this sensor is used, it separates the RAW image data in the RGB-IR array into R, G, B planes and an FIR+NIR plane, and then calculates IRH, which represents the difference plane, according to the following equations (11) and (12). When the IR pixel is located in the NIR position IRHr = R - NIR × irgain1 … (11) When the IR pixel is located at the FIR position IRHg = G-FIR × irgain6 …(12)

[0066] The plane conversion unit 101 determines the color components to be subtracted depending on whether NIR or FIR is used as described above, forms an IRH plane in which the calculated IRHr and IRg are mixed as shown in Fig. 10, and supplies R, G, B, and IRH to the frequency conversion unit 102. The subsequent processing is the same as in the first embodiment, and therefore will not be described here.

[0067] During decoding, the IR components are regenerated according to the following equations (13) and (14) using the R, G, and IRH planes previously obtained by decoding. If the decoding pixel position is NIR, NIR = {R-IRHr} / irgain1 … (13) If the decoding pixel position is FIR, FIR={G-IRHg} / irgain6 …(14)

[0068] As described above, in the case of a pixel array in which NIR and FIR are mixed, the color component from which the difference is taken is changed depending on whether the IR pixel is NIR or FIR. Also, instead of forming a separate plane, an IRH plane is formed in which the results of different calculations are mixed, and processing after plane conversion is performed. In this way, by forming an IRH plane in which the results of different calculations are mixed, the number of pixels that must be processed for R, G, B, and IRH is the same. Therefore, when processing after plane conversion is performed in parallel, processing can be made uniform.

[0069] (Other Examples) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0070] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0071] 100...imaging unit, 101...plane conversion unit, 102...frequency conversion unit, 103...quantization unit, 104...entropy coding unit, 110...RAW coding unit, 150...control unit, 204...header analysis unit, 200...entropy decoding unit, 201...inverse quantization unit, 202...inverse frequency conversion unit, 203...plane inverse conversion unit, 250...control unit

Claims

1. An image encoding device that encodes an image obtained by an image sensor having R, G, and B primary color filters in the visible light wavelength region and an infrared filter, a generating means for generating difference data indicating a difference between image data of one of the three primary color filters determined according to an intensity distribution of a transmission wavelength of the infrared filter and image data obtained through the infrared filter; encoding means for encoding image data for each of the three primary color filters and the differential data; and The generating means When the central transmission wavelength of the infrared filter is around 780 nm, the difference data is generated using image data of the R component of the three primary colors; When the central transmission wavelength of the infrared filter is around 870 nm, the difference data is generated using image data of the B component of the three primary colors; When the central transmission wavelength of the infrared filter is around 940 nm, the differential data is generated using image data of the G component of the three primary colors. An image encoding device comprising:

2. An image encoding device that encodes an image obtained by an imaging sensor having three primary color filters of R, G, and B in the visible light wavelength region and an infrared filter, a generating means for generating difference data indicating a difference between image data of one of the three primary color filters determined according to an intensity distribution of a transmission wavelength of the infrared filter and image data obtained through the infrared filter; encoding means for encoding image data for each of the three primary color filters and the differential data; and The generating means When the central transmission wavelength of the infrared filter is within a range of 780 nm to 870 nm, the difference data is generated using image data of a weighted average of the image data of the R component and the image data of the B component of the three primary colors; When the central transmission wavelength of the infrared filter is within a range of 870 nm to 940 nm, the difference data is generated using image data of a weighted average of the image data of the B component and the G component of the three primary colors. An image encoding device comprising:

3. an adjusting means for adjusting the image data obtained by the infrared filter with a gain corresponding to the type of filter used to generate the difference data among the three primary color filters; The generating means generates the difference data using the image data obtained by the infrared filter after adjustment by the adjusting means.

2. The image encoding device according to claim 1.

4. the infrared filters in the image sensor include two types of filters: a near-infrared filter and a far-infrared filter; The generating means generates the difference data by calculating a difference between the pixels obtained by transmitting through the near-infrared filter and the pixels obtained by transmitting through the R component filter, and by calculating a difference between the pixels obtained by transmitting through the far-infrared filter and the pixels obtained by transmitting through the G component filter.

4. The image encoding device according to claim 1 or 3.

5. An image decoding device that decodes coded data of RAW image data obtained by an image sensor having R, G, and B primary color filters in the visible light wavelength region and an infrared filter, an acquisition means for analyzing a header of the encoded data and acquiring information representing an intensity distribution of a transmission wavelength of the infrared filter; a decoding means for decoding the coded data to generate image data for each of the three primary color filters and differential data for the infrared filter; a restoration means for restoring image data obtained from the infrared filter from image data of any one of the three primary color filters specified by the information obtained by the acquisition means and the difference data; an output means for outputting the image data obtained by the decoding means through each of the three primary color filters and the image data obtained by the restoration means through the infrared filter as RAW image data; and The restoration means is When the acquiring means acquires information that the central transmission wavelength of the infrared filter is near 780 nm, the image data acquired from the infrared filter is restored from the image data of the R component of the three primary colors and the difference data; When the acquiring means acquires information that the central transmission wavelength of the infrared filter is near 870 nm, the image data acquired from the infrared filter is restored from the image data of the B component of the three primary colors and the difference data; When the information that the transmission center wavelength of the infrared filter is near 940 nm is acquired by the acquisition means, the image data obtained from the infrared filter is restored from the image data of the G component of the three primary colors and the difference data. An image decoding device comprising:

6. A control method for an image encoding device that encodes an image obtained by an image sensor having R, G, and B primary color filters in the visible light wavelength region and an infrared filter, comprising: a generating step of generating difference data indicating a difference between image data of any one of the three primary color filters determined according to the intensity distribution of the transmission wavelength of the infrared filter and image data obtained through the infrared filter; an encoding step of encoding image data for each of the three primary color filters and the difference data; and In the generating step, When the central transmission wavelength of the infrared filter is around 780 nm, the difference data is generated using image data of the R component of the three primary colors; When the central transmission wavelength of the infrared filter is around 870 nm, the difference data is generated using image data of the B component of the three primary colors; When the central transmission wavelength of the infrared filter is around 940 nm, the differential data is generated using image data of the G component of the three primary colors.

2. A control method for an image encoding device comprising:

7. A control method for an image encoding device that encodes an image obtained by an imaging sensor having three primary color filters of R, G, and B in the visible light wavelength region and an infrared filter, comprising: a generating step of generating difference data indicating a difference between image data of any one of the three primary color filters determined according to the intensity distribution of the transmission wavelength of the infrared filter and image data obtained through the infrared filter; an encoding step of encoding image data for each of the three primary color filters and the difference data; and In the generating step, When the central transmission wavelength of the infrared filter is within a range of 780 nm to 870 nm, the difference data is generated using image data of a weighted average of the image data of the R component and the image data of the B component of the three primary colors; When the central transmission wavelength of the infrared filter is within a range of 870 nm to 940 nm, the difference data is generated using image data of a weighted average of the image data of the B component and the G component of the three primary colors.

2. A control method for an image encoding device comprising:

8. A control method for an image decoding device that decodes coded data of RAW image data obtained by an imaging sensor having R, G, and B primary color filters in the visible light wavelength region and an infrared filter, comprising: an acquiring step of analyzing a header of the encoded data and acquiring information representing an intensity distribution of a transmission wavelength of the infrared filter; a decoding step of decoding the encoded data to generate image data for each of the three primary color filters and differential data for the infrared filter; a restoration step of restoring image data obtained from the infrared filter from image data of any one of the three primary color filters specified by the information obtained in the acquisition step and the difference data; an output step of outputting, as RAW image data, the image data obtained through each of the three primary color filters in the decoding step and the image data obtained through the infrared filter in the restoration step; and In the restoration step, In the acquiring step, when information indicating that the transmission center wavelength of the infrared filter is near 780 nm is acquired, image data obtained from the infrared filter is restored from image data of the R component of the three primary colors and the difference data; In the acquiring step, when information indicating that the transmission center wavelength of the infrared filter is near 870 nm is acquired, image data obtained from the infrared filter is restored from the image data of the B component of the three primary colors and the difference data; In the acquisition step, when information indicating that the transmission center wavelength of the infrared filter is near 940 nm is acquired, the image data obtained from the infrared filter is restored from the image data of the G component of the three primary colors and the difference data.

2. A control method for an image decoding device comprising:

9. A computer program that, when read and executed by a computer, causes the computer to execute each step of the control method for an image encoding device according to claim 6.

10. A computer program that, when read and executed by a computer, causes the computer to execute each step of the control method for an image decoding device according to claim 8.

Images