Image processing device and method

Abstract

The present disclosure relates to an image processing device and method by which a decrease in coding efficiency can be suppressed. A raw image, which is an output from a sensor having a pixel array constituted by a plurality of types of pixels for detecting incident lights having mutually different optical characteristics including at least a polarization angle characteristic, is reconfigured by use of the optical characteristics of the incident lights including at least the polarization angle characteristic, thereby generating a polarization angle characteristic reconfigured image constituted by the pixel values of the pixels having detected the incident lights for which at least the characteristic values of the polarization angle characteristic are identical with each other, and the generated polarization angle characteristic reconfigured image is encoded. The present disclosure can be applied, for example, to an image processing device, an encoding device, a decoding device, electronic equipment, an image processing method, or a program.

Description

[Technical Field]

[0001] The present disclosure relates to an image processing device and method, and more particularly to an image processing device and method that can suppress a decrease in encoding efficiency. [Background technology]

[0002] In the past, there were camera systems in which RAW images output from an image sensor were transmitted to an image processing LSI (Large Scale Integration) for development and image processing. In recent years, with the advancement of semiconductor manufacturing technology and element configuration technology, the image sensors used in such camera systems have become higher in definition, frame rate, and dynamic range, and the data volume of RAW images output from the image sensor has increased.

[0003] To address this issue, a method was devised to encode (compress) RAW images and transmit them to an image processing LSI. In encoding RAW images, a coding method that uses prediction based on correlation within an image (between pixels) or between images, such as JPEG (Joint Photographic Experts Group), MPEG (Moving Picture Experts Group), AVC (Advanced Video Coding), HEVC (High Efficiency Video Coding), or VVC (Versatile Video Coding), can be used to further reduce the reduction in encoding efficiency.

[0004] Another method has been proposed in which a RAW image is separated into the R, G, and B colors of a color filter and the image is coded for each color (see, for example, Patent Document 1). This improves prediction accuracy within an image (between pixels) and further reduces the reduction in coding efficiency.

[0005] Recently, polarization sensors have been developed that detect incident light that has been given a polarization angle by a polarization filter. A polarization filter transmits only light with a predetermined polarization angle. In other words, light that passes through the polarization filter is given that polarization angle as an optical characteristic. The polarization filter can set the polarization angle to be given for each pixel. For example, a polarization filter can give multiple polarization angles as an optical characteristic to the incident light of the entire pixel array of the polarization sensor. In other words, the polarization filter can give one of multiple polarization angles as an optical characteristic to the incident light of each pixel of the pixel array. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2003-125209 Summary of the Invention [Problem to be solved by the invention]

[0007] However, with the method described in Patent Document 1, it is difficult to separate a RAW image for each polarization angle imparted to incident light. In other words, there is a risk that the image for each color may contain pixel values ​​of incident light imparted with different types of polarization angles. Therefore, when encoding the image for each color, there is a risk that prediction accuracy within the image (between pixels) may decrease, resulting in a decrease in encoding efficiency.

[0008] The present disclosure has been made in light of such circumstances, and makes it possible to suppress a decrease in subjective image quality while suppressing a decrease in coding efficiency. [Means for solving the problem]

[0009] According to one aspect of the present technology, there is provided an image processing device, comprising: a sensor having a pixel array configured with a plurality of types of pixels that detect incident light having different optical characteristics including at least polarization angle characteristics; By reconstructing the image based on intra-block positions, which are pixel positions within a pixel block configured by a predetermined number of pixels having a plurality of types of light receiving sensitivity, an intra-block position reconstructed image is generated that is configured by pixel values ​​of the pixels whose intra-block positions are the same, and the intra-block position reconstructed image is calculated based on the polarization angle characteristicand a coding unit that codes the polarization angle characteristic reconstructed image.

[0010] An image processing method according to one aspect of the present technology includes: By reconstructing the image based on intra-block positions, which are pixel positions within a pixel block configured by a predetermined number of pixels having a plurality of types of light receiving sensitivity, an intra-block position reconstructed image is generated that is configured by pixel values ​​of the pixels whose intra-block positions are the same, and the intra-block position reconstructed image is calculated based on the polarization angle characteristic and generating a polarization angle characteristic reconstructed image composed of pixel values ​​of the pixels that detected the incident light having at least the same characteristic values ​​of the polarization angle characteristic by reconstructing the polarization angle characteristic by the above method, and encoding the generated polarization angle characteristic reconstructed image.

[0011] According to another aspect of the present technology, there is provided an image processing device, wherein by decoding encoded data, a RAW image output from a sensor having a pixel array configured with a plurality of types of pixels that detect incident light having different optical characteristics including at least polarization angle characteristics is generated. The light receiving sensitivity is reconstructed by the pixel position in the pixel block, which is the pixel position in the pixel block composed of a predetermined number of the pixels of a plurality of types, and the polarization angle characteristic is further reconstructed by the pixel position in the pixel block. and a development processing unit that develops the polarization angle characteristic reconstructed image.

[0012] In an image processing method according to another aspect of the present technology, by decoding encoded data, a RAW image that is an output of a sensor having a pixel array configured with a plurality of types of pixels that detect incident light having different optical characteristics including at least polarization angle characteristics is obtained by: The light receiving sensitivity is reconstructed by the pixel position in the pixel block, which is the pixel position in the pixel block composed of a predetermined number of the pixels of a plurality of types, and the polarization angle characteristic is further reconstructed by the pixel position in the pixel block. and generating a polarization angle characteristic reconstructed image composed of pixel values ​​of the pixels that detected the incident light, the pixel values ​​having at least the same characteristic values ​​of the polarization angle characteristic being reconstructed by the above method, and developing the generated polarization angle characteristic reconstructed image.

[0013] In an image processing device and method according to one aspect of the present technology, a RAW image is an output of a sensor having a pixel array configured with a plurality of types of pixels that detect incident light having different optical characteristics including at least polarization angle characteristics, The light receiving sensitivity is reconstructed by the intra-block position, which is the pixel position within a pixel block composed of a predetermined number of pixels of a plurality of types, to generate an intra-block position reconstructed image composed of pixel values ​​of pixels having the same intra-block position, and the intra-block position reconstructed image is used to generate a polarization angle characteristic. By reconstructing the polarization angle characteristics by the above method, a polarization angle characteristic reconstructed image is generated that is composed of pixel values ​​of pixels that detect incident light having at least the same characteristic values ​​of the polarization angle characteristics, and the generated polarization angle characteristic reconstructed image is encoded.

[0014] In the image processing device and method according to another aspect of the present technology, the encoded data is decoded, and thereby a RAW image, which is an output of a sensor having a pixel array configured with a plurality of types of pixels that detect incident light having different optical characteristics including at least polarization angle characteristics, is generated. The light receiving sensitivity is reconstructed by the pixel position in the pixel block, which is the pixel position in the pixel block composed of a predetermined number of pixels of multiple types, and the polarization angle characteristic is also reconstructed by the pixel position in the pixel block. A polarization angle characteristic reconstructed image is generated, which is composed of pixel values ​​of pixels that detect incident light having at least the same characteristic values ​​of the polarization angle characteristics as reconstructed by the above method, and the generated polarization angle characteristic reconstructed image is developed. [Brief explanation of the drawings]

[0015] [Figure 1] FIG. 10 is a diagram illustrating an example of transmitting a RAW image. [Figure 2] FIG. 10 is a diagram illustrating an example of encoding and transmitting a RAW image. [Figure 3] FIG. 10 is a diagram showing an example of reconstructing an image based on polarization angles. [Figure 4] FIG. 10 is a diagram showing an example of the configuration of a RAW image of a polarization sensor. [Figure 5] FIG. 10 is a diagram showing an example of a transmission wavelength characteristic reconstructed image. [Figure 6] FIG. 10 is a diagram showing an example of a polarization angle characteristic reconstructed image. [Figure 7] FIG. 10 is a diagram showing an example of reconstruction using only polarization angles. [Figure 8] FIG. 10 is a diagram showing an example of a developed image. [Figure 9] FIG. 2 is a block diagram showing an example of the main configuration of a polarization sensor unit. [Figure 10]10 is a flowchart illustrating an example of a detection processing method. [Figure 11] 10 is a flowchart illustrating an example of an image processing method. [Figure 12] FIG. 1 is a diagram illustrating an example of the configuration of a RAW image of a multispectral image sensor. [Figure 13] FIG. 10 is a diagram showing an example of reconstructing an image using transmission wavelength characteristics and lattice points. [Figure 14] FIG. 10 is a diagram illustrating an example of lattice points. [Figure 15] FIG. 10 is a diagram showing an example of a transmission wavelength characteristic grid point reconstruction image. [Figure 16] FIG. 10 is a diagram showing an example of a developed image. [Figure 17] FIG. 2 is a block diagram showing an example of the main configuration of a multispectral image sensor unit. [Figure 18] 10 is a flowchart illustrating an example of a detection processing method. [Figure 19] 10 is a flowchart illustrating an example of an image processing method. [Figure 20] FIG. 1 is a diagram showing an example of the configuration of a RAW image of an image plane phase difference AF image sensor. [Figure 21] FIG. 10 is a diagram showing an example of reconstructing an image based on intra-block positions. [Figure 22] FIG. 10 is a diagram showing an example of an intra-block position reconstructed image. [Figure 23] FIG. 10 is a diagram showing an example of a developed image. [Figure 24] FIG. 2 is a block diagram showing an example of the main configuration of an image plane phase difference AF image sensor unit. [Figure 25] 10 is a flowchart illustrating an example of a detection processing method. [Figure 26] 10 is a flowchart illustrating an example of an image processing method. [Figure 27] FIG. 1 is a block diagram illustrating an example of the main configuration of a computer. DETAILED DESCRIPTION OF THE INVENTION

[0016] Hereinafter, modes for carrying out the present disclosure (hereinafter referred to as embodiments) will be described in the following order. 1. RAW image transfer 2. First embodiment (image reconstruction based on polarization angle characteristics) 3. Second embodiment (multispectral image sensor) 4. Third embodiment (image plane phase difference AF image sensor) 5. Combination 6. Supplementary Notes

[0017] <1. RAW image transmission> <References supporting technical content and technical terminology> The scope of disclosure of the present technology includes not only the contents described in the embodiments but also the contents described in the following non-patent documents that were publicly known at the time of filing, as well as the contents of other documents referenced in the following non-patent documents.

[0018] Patent Document 1: (mentioned above) Non-patent document 1: TELECOMMUNICATION STANDARDIZATION SECTOR OF ITU, "Versatile video coding", SERIES H: AUDIOVISUAL AND MULTIMEDIA SYSTEMS Infrastructure of audiovisual services - Coding of moving video, Recommendation ITU-T H.266, 08 / 2020 Non-patent document 2: Recommendation ITU-T H.264 (04 / 2017) "Advanced video coding for generic audiovisual services", April 2017 Non-patent document 3: Recommendation ITU-T H.265 (02 / 18) "High efficiency video coding", February 2018

[0019] That is, the content described in the above-mentioned non-patent documents, the content of other documents referred to in the above-mentioned non-patent documents, etc. also serve as the basis for judging the support requirements.

[0020] <RAW Image Processing System> Conventionally, there has been a camera system such as shown in FIG. 1, in which a RAW image output from an imaging device is transmitted to an image processing LSI (Large Scale Integration), and development processing and image processing are performed. In the camera system 10 shown in FIG. 1, a transmission unit 22 transmits a RAW image output from an image sensor 21 from a detection unit 11 to an image processing unit 12. A reception unit 31 acquires the RAW image. A development processing unit 32 performs development processing on the acquired RAW image. An image processing unit 33 performs predetermined image processing on the image after the development processing (also referred to as the post-development image).

[0021] In recent years, with the improvement of semiconductor manufacturing technology and element configuration technology, imaging devices used in such camera systems have advanced in high definition, high frame rate, high dynamic range, etc., and the data volume of RAW images output from the imaging devices has been increasing.

[0022] For example, with the improvement of semiconductor manufacturing technology and element configuration technology, imaging devices have become more highly refined, and elements with a pixel count per image exceeding tens of millions to over 100 million are also being used. Also, for example, when used for video shooting and recording applications, the imaging device captures and outputs high-definition images as described above at a rate of dozens to hundreds of images per second. Therefore, the amount of processing data per unit time increases in proportion to the frame rate. Also, due to the spread of devices handling high dynamic range color spaces, the dynamic range per pixel has increased, for example, from 12 bits to 14 bits.

[0023] As described above, the amount of data handled by image sensors and the camera systems that process the image data is increasing. This has raised concerns that the power consumption of each device along the data processing path, such as the image sensor, image processing LSI, and display device, will increase. This has become a major challenge in designing and implementing products that incorporate camera systems.

[0024] In addition, transmitting large volumes of data at high speed between independent devices, such as an image sensor and an image processing LSI, places strict constraints on timing delays and wiring layout from the perspective of electronic circuit hardware design and implementation, making it difficult to design the interface circuits and mounting boards for each device, resulting in increased development and manufacturing costs. Furthermore, the induction of electromagnetic noise, which can cause malfunctions in peripheral devices, has become significant, forcing products to take various countermeasures, posing the risk of increasing manufacturing costs.

[0025] Therefore, a method has been devised in which a RAW image is encoded (compressed) and transmitted to an image processing LSI. For example, as shown in FIG. 2, an encoding unit 51 encodes (compresses) a RAW image output from an image sensor 21. A transmitting unit 22 transmits encoded data of the RAW image generated by the encoding unit 51 from a detecting unit 11 to an image processing unit 12. A receiving unit 31 acquires the encoded data. A decoding unit 52 decodes the acquired encoded data and generates (restores) a RAW image. A development processing unit 32 performs development processing on the generated (restored) RAW image. An image processing unit 33 performs predetermined image processing on the image after the development processing (also referred to as a developed image).

[0026] By encoding RAW images during transmission in this manner, the transmission data rate between devices can be reduced, and problems caused by high-speed, large-volume data transmission between devices can be alleviated. Note that, in encoding RAW images, by applying an encoding method that uses prediction utilizing correlation within an image (between pixels) or correlation between images, such as JPEG (Joint Photographic Experts Group), MPEG (Moving Picture Experts Group), AVC (Advanced Video Coding), HEVC (High Efficiency Video Coding), or VVC (Versatile Video Coding), a reduction in encoding efficiency can be further suppressed.

[0027] For example, in the case of a color image sensor, color filters are provided in the pixel array to obtain pixel values ​​in the red (R) wavelength range, pixel values ​​in the green (G) wavelength range, and pixel values ​​in the blue (B) wavelength range. The color filters transmit incident light to each pixel in the pixel array and limit its wavelength range. This wavelength range is set for each pixel. In other words, the color filters are composed of a filter for one pixel that transmits the red (R) wavelength range, a filter for one pixel that transmits the green (G) wavelength range, and a filter for one pixel that transmits the blue (B) wavelength range, all arranged in an array in a predetermined arrangement pattern.

[0028] Therefore, for example, in a pixel provided with a filter that transmits the red (R) wavelength range, incident light in the red (R) wavelength range is detected, and a pixel value in the red (R) wavelength range is obtained. Also, in a pixel provided with a filter that transmits the green (G) wavelength range, incident light in the green (G) wavelength range is detected, and a pixel value in the green (G) wavelength range is obtained. Furthermore, in a pixel provided with a filter that transmits the blue (B) wavelength range, incident light in the blue (B) wavelength range is detected, and a pixel value in the blue (B) wavelength range is obtained.

[0029] For example, if filters for each wavelength band are arranged in a Bayer array pattern, the RAW image output from this color image sensor will be composed of pixel values ​​for each wavelength band distributed in a pattern corresponding to the Bayer array. For example, in the case of a Bayer array, the RAW image will be composed of rows in which pixel values ​​for the red (R) wavelength band and pixel values ​​for the green (Gr) wavelength band are alternately arranged, and rows in which pixel values ​​for the green (Gb) wavelength band and pixel values ​​for the blue (B) wavelength band are alternately arranged. In other words, pixel values ​​for wavelength bands of the same color will not be adjacent. Note that the arrangement pattern of filters for each wavelength band is arbitrary and is not limited to the Bayer array, but generally, pixel values ​​for wavelength bands of the same color will not be adjacent.

[0030] Therefore, such RAW images obtained by sequentially reading out from the image sensor have weak correlation between adjacent pixel values ​​and high spatial frequency, which means that in the case of coding methods that use predictions that utilize correlation within an image (between pixels) as described above, there is a risk that prediction accuracy will decrease and coding efficiency will decrease.

[0031] Therefore, a method has been devised in which each pixel value of a RAW image is classified into the wavelength ranges of red (R), green (G), and blue (B) of a color filter, an image composed of pixel values ​​of one type of wavelength range (i.e., an image of each wavelength range) is generated, and the image of each wavelength range is encoded (see, for example, Patent Document 1). This makes it possible to improve prediction accuracy within an image (between pixels), and further suppress a decrease in encoding efficiency.

[0032] Recently, polarization sensors have been developed that detect incident light that has been given a polarization angle by a polarization filter. A polarization filter transmits only light with a specific polarization angle. In other words, light that passes through the polarization filter is given that polarization angle as an optical characteristic. Such polarization filters are provided in the pixel array of the polarization sensor, and a polarization angle is given to the incident light at each pixel as an optical characteristic.

[0033] This polarization angle is set for each pixel. The polarizing filter may impart multiple types of polarization angles as optical characteristics to the incident light of the entire pixel array. For example, the polarizing filter may be configured with a one-pixel filter that transmits light with a polarization angle of 0 degrees, a one-pixel filter that transmits light with a polarization angle of 45 degrees, a one-pixel filter that transmits light with a polarization angle of 90 degrees, and a one-pixel filter that transmits light with a polarization angle of 135 degrees, all arranged in an array in a predetermined arrangement pattern.

[0034] In this case, for example, a pixel provided with a filter that transmits light with a polarization angle of 0 degrees will detect incident light with a polarization angle of 0 degrees. A pixel provided with a filter that transmits light with a polarization angle of 45 degrees will detect incident light with a polarization angle of 45 degrees. A pixel provided with a filter that transmits light with a polarization angle of 90 degrees will detect incident light with a polarization angle of 90 degrees. A pixel provided with a filter that transmits light with a polarization angle of 135 degrees will detect incident light with a polarization angle of 135 degrees.

[0035] Therefore, the RAW image output from this polarization sensor is composed of pixel values ​​corresponding to each polarization angle distributed according to the arrangement pattern of the filters for each polarization angle. Therefore, a RAW image generally includes portions where pixel values ​​corresponding to different polarization angles are adjacent to each other. The correlation between pixel values ​​corresponding to different polarization angles is weak, resulting in high spatial frequencies. Therefore, if a coding method using prediction utilizing intra-image (inter-pixel) correlation as described above is applied to coding the RAW image output from such a polarization sensor, there is a risk of reduced prediction accuracy and reduced coding efficiency.

[0036] Furthermore, the method described in Patent Document 1 can only generate images for each wavelength range, and it is difficult to separate RAW images for each polarization angle imparted to incident light (i.e., to generate images for each polarization angle). In other words, even if the method described in Patent Document 1 is applied, it is difficult to suppress a decrease in the coding efficiency of the RAW image output from the polarization sensor.

[0037] 2. First Embodiment <2-1. Image reconstruction based on polarization angle characteristics> Therefore, as shown in the top row of the table in FIG. 3, an image is reconstructed based on the polarization angle (method 1).

[0038] In this specification, "reconstruction" refers to extracting and summarizing pixel values ​​of pixels having the same characteristic value for a predetermined optical property from one or more images. "Summarizing" pixel values ​​refers to arranging the pixel values ​​on a two-dimensional plane (i.e., imaging).

[0039] In this specification, the term "image" refers to two-dimensional array (distribution) data (also referred to as 2D data) of pixel values ​​obtained in a pixel array of a sensor. In other words, an image does not have to be data obtained by detecting visible light. For example, an image may be data obtained by detecting invisible light, or may be depth values, etc.

[0040] <2-1-1. Reconstruction of RAW images based on polarization angle characteristics> For example, an encoder that encodes a RAW image may reconstruct the RAW image using optical characteristic parameters including at least the polarization angle characteristics, as shown in the second row from the top of the table in FIG. 3, and encode the reconstructed RAW image (method 1-1).

[0041] For example, in an image processing method, a RAW image, which is the output of a sensor having a pixel array composed of multiple types of pixels that detect incident light having different optical characteristics including at least the polarization angle characteristics, may be reconstructed using the optical characteristics including at least the polarization angle characteristics of the incident light to generate a polarization angle characteristics reconstructed image composed of pixel values ​​of pixels that detected incident light having the same characteristic values ​​of at least the polarization angle characteristics, and the generated polarization angle characteristics reconstructed image may be encoded.

[0042] For example, an image processing device may include a reconstruction unit that reconstructs a RAW image, which is the output of a sensor having a pixel array composed of multiple types of pixels that detect incident light having different optical characteristics including at least the polarization angle characteristics, using the optical characteristics including at least the polarization angle characteristics of the incident light, to generate a polarization angle characteristics reconstructed image composed of pixel values ​​of pixels that detected incident light having the same characteristic values ​​of at least the polarization angle characteristics, and an encoding unit that encodes the polarization angle characteristics reconstructed image.

[0043] That is, the encoder reconstructs pixel values ​​of the RAW image output from the polarization sensor based on the polarization angle characteristics to generate a polarization angle characteristic reconstructed image made up of pixel values ​​having the same characteristic value (polarization angle) for the polarization angle characteristics. Then, the encoder encodes each of the polarization angle characteristic reconstructed images generated for each characteristic value (polarization angle).

[0044] By doing so, the encoder can suppress a decrease in the encoding efficiency of the RAW image. Therefore, the encoder can suppress an increase in the amount of transmission data, power consumption, and electromagnetic noise. This also allows the encoder to suppress an increase in manufacturing costs and device size. Furthermore, by suppressing a decrease in encoding efficiency, the subjective image quality of the decoded image can be improved when compared with the same amount of transmission data. In other words, the encoder can suppress a decrease in the subjective image quality of the decoded image.

[0045] In this embodiment, the polarization angle characteristic reconstructed image refers to an image reconstructed using one or more optical characteristic parameters including the polarization angle characteristic. In other words, the polarization angle characteristic reconstructed image includes an image reconstructed using only the polarization angle characteristic and an image reconstructed using the polarization angle characteristic and other optical characteristics.

[0046] <Transmission wavelength characteristics 1> The polarization sensor may be a monochrome polarization sensor that detects the luminance value at each polarization angle, or a color polarization sensor that detects the luminance value at each polarization angle for each color (wavelength range).

[0047] In other words, the above-mentioned optical characteristics (parameters) may include optical characteristics other than polarization angle characteristics. For example, as shown in the third row from the top of the table in FIG. 3, the optical characteristics (parameters) may further include transmission wavelength characteristics (method 1-1-1). The transmission wavelength characteristics are the wavelength characteristics of incident light detected by the pixels of the polarization sensor. For example, in a color image sensor, this indicates the color (wavelength range) (red (R), green (G), blue (B), etc.) of the color filter through which the incident light has passed.

[0048] An example of a RAW image output from a color polarization sensor is shown in Figure 4. RAW image 101 shown in Figure 4 is an example of a RAW image output from a polarization sensor in which a polarization filter and a color filter are provided in the pixel array. Each square in RAW image 101 represents a pixel value, with white pixels representing pixel values ​​in the blue (B) wavelength range, gray pixels representing pixel values ​​in the green (G) wavelength range, and diagonally shaded pixels representing pixel values ​​in the red (R) wavelength range.

[0049] The encoder may reconstruct such a RAW image 101 based on the polarization angle characteristics, based on the transmission wavelength characteristics, or based on both the polarization angle characteristics and the transmission wavelength characteristics.

[0050] <Reorganization order> For example, as shown in the fourth row from the top of the table in FIG. 3, the encoder may perform reconstruction based on the transmission wavelength characteristics and further perform reconstruction based on the polarization angle characteristics (method 1-1-1-1).

[0051] In other words, the encoder may reconstruct the RAW image using the transmission wavelength characteristics to generate a transmission wavelength characteristics reconstructed image composed of pixel values ​​of pixels that detect incident light having the same characteristic values ​​of the transmission wavelength characteristics, and then reconstruct the transmission wavelength characteristics reconstructed image using the polarization angle characteristics to generate a polarization angle characteristics reconstructed image.

[0052] For example, an encoder may reconstruct the RAW image 101 of FIG. 4 using the transmission wavelength characteristics to generate a transmission wavelength characteristic reconstructed image as shown in FIG. 5. In FIG. 5, a transmission wavelength characteristic reconstructed image 111 is a reconstructed image made up of pixel values ​​in the blue (B) wavelength range. A transmission wavelength characteristic reconstructed image 112 is a reconstructed image made up of pixel values ​​in the green (Gb) wavelength range. A transmission wavelength characteristic reconstructed image 113 is a reconstructed image made up of pixel values ​​in the green (Gr) wavelength range. A transmission wavelength characteristic reconstructed image 114 is a reconstructed image made up of pixel values ​​in the red (R) wavelength range.

[0053] The encoder may further reconstruct each transmission wavelength characteristic reconstructed image using the polarization angle characteristic to generate polarization angle characteristic reconstructed images as shown in FIG. 6. In FIG. 6, polarization angle characteristic reconstructed image 111A is a reconstructed image made up of pixel values ​​at a polarization angle of 0 degrees in the blue (B) wavelength range. Polarization angle characteristic reconstructed image 111B is a reconstructed image made up of pixel values ​​at a polarization angle of 45 degrees in the blue (B) wavelength range. Polarization angle characteristic reconstructed image 111C is a reconstructed image made up of pixel values ​​at a polarization angle of 90 degrees in the blue (B) wavelength range. Polarization angle characteristic reconstructed image 111D is a reconstructed image made up of pixel values ​​at a polarization angle of 135 degrees in the blue (B) wavelength range.

[0054] Furthermore, polarization angle characteristic reconstructed image 112A is a reconstructed image made up of pixel values ​​at a polarization angle of 0 degrees in the green (Gb) wavelength range. Polarization angle characteristic reconstructed image 112B is a reconstructed image made up of pixel values ​​at a polarization angle of 45 degrees in the green (Gb) wavelength range. Polarization angle characteristic reconstructed image 112C is a reconstructed image made up of pixel values ​​at a polarization angle of 90 degrees in the green (Gb) wavelength range. Polarization angle characteristic reconstructed image 112D is a reconstructed image made up of pixel values ​​at a polarization angle of 135 degrees in the green (Gb) wavelength range. Furthermore, polarization angle characteristic reconstructed image 113A is a reconstructed image made up of pixel values ​​at a polarization angle of 0 degrees in the green (Gr) wavelength range. Polarization angle characteristic reconstructed image 113B is a reconstructed image made up of pixel values ​​at a polarization angle of 45 degrees in the green (Gr) wavelength range. Polarization angle characteristic reconstructed image 113C is a reconstructed image made up of pixel values ​​at a polarization angle of 90 degrees in the green (Gr) wavelength range. The polarization angle characteristic reconstructed image 113D is a reconstructed image configured from pixel values ​​in the green (Gr) wavelength range at a polarization angle of 135 degrees.

[0055] Furthermore, polarization angle characteristic reconstructed image 114A is a reconstructed image made up of pixel values ​​at a polarization angle of 0 degrees in the red (R) wavelength range. Polarization angle characteristic reconstructed image 114B is a reconstructed image made up of pixel values ​​at a polarization angle of 45 degrees in the red (R) wavelength range. Polarization angle characteristic reconstructed image 114C is a reconstructed image made up of pixel values ​​at a polarization angle of 90 degrees in the red (R) wavelength range. Polarization angle characteristic reconstructed image 114D is a reconstructed image made up of pixel values ​​at a polarization angle of 135 degrees in the red (R) wavelength range.

[0056] That is, each polarization angle characteristic reconstructed image shown in FIG. 6 is an image obtained by reconstructing the RAW image 101 using the transmission wavelength characteristic and the polarization angle characteristic.

[0057] By repeating the reconstruction for each optical characteristic in this way, the encoder can easily reconstruct a plurality of optical characteristics. In addition, by doing so, the encoder can perform control such as, for example, reconstructing only the optical characteristics that need to be reconstructed and omitting the reconstruction for the optical characteristics that do not need to be reconstructed.

[0058] The order in which the optical characteristics are reconstructed is arbitrary and is not limited to the above example. For example, the encoder may first reconstruct the image based on the polarization angle characteristics, and then reconstruct the image based on the transmission wavelength characteristics.

[0059] <Encoding order sort> The encoder encodes the polarization angle characteristic reconstructed image generated as described above. In this case, the encoder may sort the polarization angle characteristic reconstructed images in an order in which the same transmission wavelength characteristic values ​​are consecutive, as shown in the fifth row from the top of the table in Fig. 3, and encode each polarization angle characteristic reconstructed image in that order (Method 1-1-1-2).

[0060] That is, the encoder may encode the plurality of polarization angle characteristic reconstructed images in an order based on the characteristic values ​​of the transmission wavelength characteristics. More specifically, the encoder may encode the plurality of polarization angle characteristic reconstructed images in an order such that the same characteristic values ​​of the transmission wavelength characteristics appear consecutively.

[0061] By doing so, the encoder can increase predictions that utilize the correlation between polarization angle characteristic reconstructed images with the same characteristic value. Generally, the correlation between polarization angle characteristic reconstructed images with the same characteristic value is high, so by doing so, the encoder can suppress a decrease in prediction accuracy between images (polarization angle characteristic reconstructed images). In other words, the encoder can further suppress a decrease in encoding efficiency of RAW images.

[0062] The encoder may encode multiple polarization angle characteristic reconstructed images reconstructed based on the polarization angle characteristic and other optical characteristics in an order based on the characteristic values ​​of the polarization angle characteristic. More specifically, the encoder may encode multiple polarization angle characteristic reconstructed images in an order in which the same characteristic value for the polarization angle characteristic appears consecutively. In this case, the encoder can also suppress a decrease in prediction accuracy between images (polarization angle characteristic reconstructed images). In other words, the encoder can further suppress a decrease in encoding efficiency of RAW images.

[0063] <Encoding method> The encoder may encode the reconstructed image by applying prediction that utilizes correlation between pixels or between images, as shown in the sixth row from the top of the table in FIG. 3, for example (method 1-1-1-3).

[0064] That is, the encoder may encode the polarization angle characteristic reconstructed image using prediction that utilizes intra-image correlation or inter-image correlation, thereby further suppressing a decrease in the encoding efficiency of the RAW image, as described above.

[0065] <2-1-2. Decoding and development of polarization angle characteristic reconstructed image> For example, a decoder that decodes the coded data of a RAW image may decode the coded data and develop the resulting polarization angle characteristic reconstructed image, as shown in the seventh row from the top of the table in Figure 3 (Method 1-2).

[0066] For example, in an image processing method, encoded data may be decoded to generate a polarization angle characteristic reconstructed image composed of pixel values ​​of pixels that detect incident light having at least the same characteristic values ​​of the polarization angle characteristic, reconstructed from a RAW image that is the output of a sensor having a pixel array composed of multiple types of pixels that detect incident light having different optical characteristics including at least the polarization angle characteristic, using the optical characteristics including at least the polarization angle characteristic of the incident light, and the generated polarization angle characteristic reconstructed image may then be developed.

[0067] For example, an image processing device may include a decoding unit that decodes encoded data to generate a polarization angle characteristic reconstructed image composed of pixel values ​​of pixels that detected incident light having at least the same characteristic values ​​of the polarization angle characteristic, where the RAW image is the output of a sensor having a pixel array composed of multiple types of pixels that detect incident light having different optical characteristics including at least the polarization angle characteristic, and the polarization angle characteristic reconstructed image is reconstructed using the optical characteristics including at least the polarization angle characteristic of the incident light, and a development processing unit that develops the polarization angle characteristic reconstructed image.

[0068] That is, the decoder decodes the encoded data of the polarization angle characteristic reconstructed image generated by the encoder as described above, develops the obtained polarization angle characteristic reconstructed image, and generates a developed image. Therefore, the decoder can correctly generate a developed image from the encoded data generated by the encoder. Therefore, the decoder can suppress a decrease in the encoding efficiency of the RAW image. Therefore, the decoder can suppress an increase in the amount of transmission data, power consumption, and electromagnetic noise. Furthermore, this allows the decoder to suppress an increase in manufacturing costs and device size. Furthermore, suppressing a decrease in encoding efficiency can improve the subjective image quality of the decoded image when compared with the same amount of transmission data. That is, the decoder can suppress a decrease in the subjective image quality of the decoded image.

[0069] In this specification, the development process refers to a process of separating the reconstructed image obtained by decoding into each transmission wavelength characteristic (for example, each color) and interpolating missing pixel values. For example, in the case of processing a RAW image output from an image sensor to which a Bayer array color filter is applied, the development process is also called a demosaic process.

[0070] <Transmission wavelength characteristics 2> The polarization sensor that outputs a RAW image may be a monochrome polarization sensor that detects the brightness value at each polarization angle, or a color polarization sensor that detects the brightness value at each polarization angle for each color (wavelength range).

[0071] In other words, the above-mentioned optical characteristics (parameters) may include optical characteristics other than the polarization angle characteristics. For example, as shown in the eighth row from the top of the table in Fig. 3, the optical characteristics (parameters) may further include the transmission wavelength characteristics (method 1-2-1). The transmission wavelength characteristics are the same as those described on the encoding side.

[0072] <Reconstruction process> For example, as shown in the ninth row from the top of the table in FIG. 3, the decoder may reconstruct the polarization angle characteristic reconstructed image obtained by decoding the encoded data using the polarization angle characteristic (method 1-2-1-1).

[0073] In other words, the decoder may reconstruct a polarization angle characteristic reconstructed image obtained by decoding the encoded data using optical characteristics including the polarization angle characteristic and the transmission wavelength characteristic, using the polarization angle characteristic, and then develop the reconstructed polarization angle characteristic reconstructed image.

[0074] For example, if the encoder reconstructs and encodes a RAW image using a plurality of optical characteristics including the polarization angle characteristic, the decoder decodes the encoded data to obtain a polarization angle characteristic reconstructed image reconstructed using a plurality of optical characteristics including the polarization angle characteristic. For example, if the encoder reconstructs and encodes a RAW image using the polarization angle characteristic and the transmission wavelength characteristic, the decoder decodes the encoded data to obtain a polarization angle characteristic reconstructed image as shown in FIG.

[0075] In this case, the decoder may reconstruct the polarization angle characteristic reconstructed image before developing it, and generate a polarization angle characteristic reconstructed image reconstructed only using the polarization angle characteristic. For example, by performing this reconstruction on the polarization angle characteristic reconstructed image shown in Figure 6, the decoder generates a polarization angle characteristic reconstructed image as shown in Figure 7.

[0076] 7, polarization angle characteristic reconstructed image 121 is a reconstructed image made up of pixel values ​​at a polarization angle of 0 degrees. The decoder generates polarization angle characteristic reconstructed image 121 by reconstructing polarization angle characteristic reconstructed image 111A, polarization angle characteristic reconstructed image 112A, polarization angle characteristic reconstructed image 113A, and polarization angle characteristic reconstructed image 114A shown in FIG.

[0077] 6. Polarization angle characteristic reconstructed image 122 is a reconstructed image made up of pixel values ​​at a polarization angle of 45 degrees. The decoder generates polarization angle characteristic reconstructed image 122 by reconstructing polarization angle characteristic reconstructed image 111B, polarization angle characteristic reconstructed image 112B, polarization angle characteristic reconstructed image 113B, and polarization angle characteristic reconstructed image 114B shown in FIG.

[0078] 6. Furthermore, polarization angle characteristic reconstructed image 123 is a reconstructed image made up of pixel values ​​at a polarization angle of 90 degrees. The decoder generates polarization angle characteristic reconstructed image 123 by reconstructing polarization angle characteristic reconstructed image 111C, polarization angle characteristic reconstructed image 112C, polarization angle characteristic reconstructed image 113C, and polarization angle characteristic reconstructed image 114C shown in FIG.

[0079] 6. Polarization angle characteristic reconstructed image 124 is a reconstructed image made up of pixel values ​​at a polarization angle of 135 degrees. The decoder generates polarization angle characteristic reconstructed image 124 by reconstructing polarization angle characteristic reconstructed image 111D, polarization angle characteristic reconstructed image 112D, polarization angle characteristic reconstructed image 113D, and polarization angle characteristic reconstructed image 114D shown in FIG.

[0080] Therefore, the polarization angle characteristic reconstructed images 121 to 124 are equivalent to reconstructed images obtained by reconstructing a RAW image using only the polarization angle characteristic. That is, the polarization angle characteristic reconstructed images 121 to 124 include pixel values ​​in the blue (B) wavelength range, pixel values ​​in the green (Gb) wavelength range, pixel values ​​in the green (Gr) wavelength range, and pixel values ​​in the red (R) wavelength range, respectively.

[0081] The decoder performs development processing on each of the polarization angle characteristic reconstructed images 121 to 124, and generates a developed image as shown in FIG.

[0082] 8, developed image 121A is a developed image in the blue (B) wavelength range. Developed image 121B is a developed image in the green (G) wavelength range. Developed image 121C is a developed image in the red (R) wavelength range. The decoder performs a development process on the polarization angle characteristic reconstructed image 121 to generate developed images 121A to 121C.

[0083] Moreover, developed image 122A is a developed image in the blue (B) wavelength range. Developed image 122B is a developed image in the green (G) wavelength range. Developed image 122C is a developed image in the red (R) wavelength range. The decoder performs a development process on the polarization angle characteristic reconstructed image 122 to generate developed images 122A to 122C.

[0084] Furthermore, developed image 123A is a developed image in the blue (B) wavelength range. Developed image 123B is a developed image in the green (G) wavelength range. Developed image 123C is a developed image in the red (R) wavelength range. The decoder performs a development process on the polarization angle characteristic reconstructed image 123 to generate developed images 123A to 123C.

[0085] Moreover, developed image 124A is a developed image in the blue (B) wavelength range. Developed image 124B is a developed image in the green (G) wavelength range. Developed image 124C is a developed image in the red (R) wavelength range. The decoder performs a development process on the polarization angle characteristic reconstructed image 124 to generate developed images 124A to 124C.

[0086] In this way, the decoder can generate a developed image for each characteristic value of the polarization angle characteristic.

[0087] <Image processing> The decoder may also perform image processing on the generated developed image, for example, as shown in the tenth row from the top of FIG. 3 (method 1-2-2).

[0088] The content of this image processing is arbitrary. For example, as shown in the eleventh row from the top of Fig. 3, the decoder may perform noise reduction processing to reduce noise in the generated developed image (Method 1-2-2-1). Also, as shown in the bottom row of Fig. 3, the decoder may enlarge or reduce the generated developed image (Method 1-2-2-2). By doing so, the decoder can suppress degradation of the subjective image quality of the decoded image.

[0089] <2-2. Polarization sensor unit> Next, a configuration to which the above-described present technology is applied will be described. Fig. 9 is a block diagram showing an example of the configuration of a polarization sensor unit, which is one aspect of an image processing device to which the present technology is applied. The polarization sensor unit 300 shown in Fig. 9 is a unit that uses a polarization sensor to generate a RAW image from which pixel values ​​for each polarization angle of incident light can be acquired, and performs image processing on the RAW image.

[0090] Note that Fig. 9 shows the main processing units (blocks) and data flows, and is not limited to all that are shown in Fig. 9. In other words, in the polarization sensor unit 300, there may be processing units that are not shown as blocks in Fig. 9, and there may be processing and data flows that are not shown as arrows or the like in Fig. 9.

[0091] 9, the polarization sensor unit 300 has a detection unit 301 and an image processing unit 302. The detection unit 301 performs detection and outputs a RAW image (encoded data obtained by reconstructing and encoding the RAW image) as the detection result of the detection target.

[0092] The detector 301 includes a polarization sensor 311 , a reconstructor 312 , a reconstructed image encoder 313 , and a transmitter 314 .

[0093] The polarization sensor 311 has a pixel array provided with a filter that imparts optical characteristics including at least polarization angle characteristics to incident light, and detects the incident light that has been imparted with optical characteristics including at least polarization angle characteristics at each pixel of the pixel array. That is, the polarization sensor 311 includes at least a polarization filter in the pixel array. For example, the polarization sensor 311 may include a polarization filter and a color filter in the pixel array. That is, the optical characteristics imparted to the incident light may further include a transmission wavelength characteristic, which is the wavelength characteristic of the incident light.

[0094] In other words, the polarization sensor 311 has a pixel array made up of multiple types of pixels that detect incident light with different optical characteristics, including at least polarization angle characteristics, and outputs the detection results of the incident light as a RAW image.

[0095] The reconstruction unit 312 acquires the RAW image output from the polarization sensor 311 and reconstructs it using optical characteristics including at least the polarization angle characteristics, thereby generating a polarization angle characteristic reconstructed image composed of pixel values ​​of pixels that detect incident light having at least the same characteristic values ​​of the polarization angle characteristics.

[0096] For example, when the polarization sensor 311 imparts a polarization angle and transmission wavelength characteristics to incident light (for example, when the polarization sensor 311 has a polarization filter and a color filter in the pixel array), the reconstruction unit 312 may reconstruct the RAW image output from the polarization sensor 311 using the transmission wavelength characteristics to generate a transmission wavelength characteristics reconstructed image made up of pixel values ​​of pixels that detected incident light having the same characteristic values ​​of the transmission wavelength characteristics, and then reconstruct the transmission wavelength characteristics reconstructed image using the polarization angle characteristics to generate a polarization angle characteristics reconstructed image.

[0097] 9, the reconstruction unit 312 may have a transmission wavelength characteristics reconstruction unit 321 and a polarization angle characteristics reconstruction unit 322. The transmission wavelength characteristics reconstruction unit 321 reconstructs the RAW image using the transmission wavelength characteristics to generate a transmission wavelength characteristics reconstructed image. The polarization angle characteristics reconstruction unit 322 reconstructs the transmission wavelength characteristics reconstructed image using the polarization angle characteristics to generate a polarization angle characteristics reconstructed image.

[0098] The reconstructed image encoding unit 313 encodes the polarization angle characteristic reconstructed image generated by the reconstruction unit 312 (polarization angle characteristic reconstruction unit 322) using a predetermined encoding method to generate encoded data. For example, the reconstructed image encoding unit 313 may encode the polarization angle characteristic reconstructed image using prediction that utilizes intra-image correlation or inter-image correlation. Furthermore, the reconstructed image encoding unit 313 may encode multiple polarization angle characteristic reconstructed images in an order based on the characteristic values ​​of the transmission wavelength characteristics.

[0099] The transmission unit 314 transmits the coded data generated by the reconstructed image coding unit 313 to the image processing unit 302 .

[0100] The reconstructing unit 312 and the reconstructed image encoding unit 313 can also be regarded as an encoding unit 331 .

[0101] The image processing unit 302 performs image processing on the RAW image generated by the detection unit 301. As shown in FIG. 9 , the image processing unit 302 includes a receiving unit 351, a reconstructed image decoding unit 352, a polarization angle characteristic reconstructing unit 353, a development processing unit 354, and an image processing unit 355.

[0102] The receiving unit 351 acquires the coded data transmitted from the detecting unit 301 (transmitting unit 314 ), and supplies the coded data to the reconstructed image decoding unit 352 .

[0103] The reconstructed image decoding unit 352 decodes the encoded data supplied from the receiving unit 351 and generates (restores) a polarization angle characteristic reconstructed image. This polarization angle characteristic reconstructed image is the reconstructed image generated by the reconstruction unit 312 of the detection unit 301 as described above. In other words, by decoding the encoded data, the reconstructed image decoding unit 352 generates a polarization angle characteristic reconstructed image composed of pixel values ​​of pixels that detected incident light having the same characteristic values ​​of at least the polarization angle characteristic, reconstructed from a RAW image that is the output of a sensor having a pixel array composed of multiple types of pixels that detect incident light having different optical characteristics including at least the polarization angle characteristic, using optical characteristics including at least the polarization angle characteristic of the incident light.

[0104] The polarization angle characteristic reconstruction unit 353 reconstructs, by using the polarization angle characteristic, the polarization angle characteristic reconstructed image generated (restored) by the reconstructed image decoding unit 352. In other words, the polarization angle characteristic reconstruction unit 353 reconstructs, by using the polarization angle characteristic, the polarization angle characteristic reconstructed image that has been obtained by decoding the encoded data by the reconstructed image decoding unit 352 and reconstructed by using the optical characteristics including the polarization angle characteristic and the transmission wavelength characteristic.

[0105] The development processing unit 354 develops the polarization angle characteristic reconstructed image reconstructed by the polarization angle characteristic reconstruction unit 353 based on the polarization angle characteristic, and generates a developed image.

[0106] The image processing unit 355 performs image processing on the developed image generated by the development processing unit 354. The content of this image processing is arbitrary. For example, the image processing unit 355 may perform processing to reduce noise as part of its image processing. The image processing unit 355 may also perform enlargement or reduction of the developed image as part of its image processing. The image processing unit 355 outputs the developed image that has been subjected to image processing to the outside of the image processing unit 302, i.e., to the outside of the polarization sensor unit 300.

[0107] The reconstructed image decoding unit 352, the polarization angle characteristics reconstruction unit 353, and the development processing unit 354 can also be regarded as a decoding unit 361. The polarization angle characteristics reconstruction unit 353 may be omitted.

[0108] With the above-described configuration, the polarization sensor unit 300 can further suppress a decrease in the coding efficiency of the RAW image.

[0109] <Detection process flow> Next, a description will be given of the processing executed by the polarization sensor unit 300. An example of the flow of the detection processing executed by the detection section 301 will be described with reference to the flowchart of FIG.

[0110] When the detection process starts, the polarization sensor 311 detects incident light and generates a RAW image in step S301.

[0111] In step S302, the reconstruction unit 312 (transmission wavelength characteristic reconstruction unit 321) reconstructs the RAW image generated in step S301 using the transmission wavelength characteristic to generate a transmission wavelength characteristic reconstructed image. That is, the number of reconstructed images generated is the number of types of characteristic values, i.e., N1. The arrangement of pixels in each reconstructed image is assumed to maintain the matrix order in the RAW image.

[0112] In step S303, the reconstruction unit 312 (polarization angle characteristic reconstruction unit 322) reconstructs the transmission wavelength characteristic reconstructed image generated in step S302 using the polarization angle characteristic to generate a polarization angle characteristic reconstructed image. That is, the number of reconstructed images generated is the number of types of characteristic values, i.e., N2. The arrangement of pixels in each reconstructed image maintains the order of the matrix in each reconstructed image generated in step S302.

[0113] The same process is repeated up to optical property M. In total, N=N1xN2x...xNM images are generated.

[0114] In other words, the reconstruction unit 312 reconstructs a RAW image, which is the output of a sensor having a pixel array composed of multiple types of pixels that detect incident light having different optical characteristics including at least the polarization angle characteristics, using optical characteristics including at least the polarization angle characteristics, thereby generating a polarization angle characteristic reconstructed image composed of pixel values ​​of pixels that detect incident light having at least the same characteristic values ​​of the polarization angle characteristics.

[0115] In step S304, the reconstructed image encoding unit 313 arranges the polarization angle characteristic reconstructed images generated in step S303 in encoding order.

[0116] In step S305, the reconstructed image encoding unit 313 encodes the polarization angle characteristic reconstructed images in the order sorted in step S304 to generate encoded data. The reconstructed image encoding unit 313 encodes the generated N images using an image compression method. Image compression may be performed using a method that uses correlation within the same image or a method that uses correlation between images. When correlation between images is used, the order of compression processing is such that images having a common characteristic value for a certain optical characteristic are encoded consecutively. This strengthens the correlation between neighboring images, improving encoding performance.

[0117] That is, the reconstructed image encoding unit 313 encodes the polarization angle characteristic reconstructed image generated in step S303.

[0118] In step S306, the transmission unit 314 transmits the encoded data generated by the process in step S305 to the image processing unit 302 (reception unit 351).

[0119] When the process of step S306 is completed, the image processing ends.

[0120] <Image processing flow> Next, an example of the flow of image processing executed by the image processing unit 302 will be described with reference to the flowchart of FIG.

[0121] When image processing starts, in step S351, the receiving unit 351 receives the coded data transmitted in step S306 of FIG.

[0122] In step S352, the reconstructed image decoding unit 352 decodes the encoded data and generates (restores) N polarization angle characteristic reconstructed images.

[0123] In other words, by decoding the encoded data, the reconstructed image decoding unit 352 generates a polarization angle characteristic reconstructed image composed of pixel values ​​of pixels that detected incident light having at least the same characteristic values ​​of the polarization angle characteristics, reconstructed from a RAW image, which is the output of a sensor having a pixel array composed of multiple types of pixels that detect incident light having different optical characteristics including at least the polarization angle characteristics, using optical characteristics including at least the polarization angle characteristics.

[0124] In step S353, the polarization angle characteristics reconstruction unit 353 reconstructs the image based on the polarization angle characteristics. That is, the polarization angle characteristics reconstruction unit 353 selects and arranges pixels from the N polarization angle characteristics reconstructed images restored in step S352, as necessary, so as to be suitable for subsequent image processing, and reconstructs the image.

[0125] In step S354, the development processing unit 354 performs development processing on the polarization angle characteristic reconstructed image to generate a developed image.

[0126] In step S355, the image processing unit 355 performs image processing such as noise reduction and scaling.

[0127] When the process of step S355 is completed, the image processing ends.

[0128] By performing each process as described above, the polarization sensor unit 300 can further suppress a decrease in the coding efficiency of the RAW image.

[0129] 3. Second Embodiment <3-1. Image reconstruction using transmission wavelength characteristics and lattice points (positions)> <Multispectral image sensor> Image sensors that capture images at various wavelengths have also been developed. There are cameras that capture images of a subject at multiple different wavelengths (for example, called spectroscopic cameras or multispectral cameras). There are also image sensors (called multispectral image sensors, etc.) that are designed to capture images at more wavelengths than the conventional RGB three-color type by arranging one type of filter with various transmission wavelengths for each photodiode.

[0130] For example, as shown in FIG. 12, some multispectral image sensors have pixel groups consisting of a certain number of adjacent pixels having different transmission wavelengths, and these pixel groups are arranged periodically.

[0131] A RAW image 401 shown in Fig. 12 is an example of a RAW image generated by a multispectral image sensor. In Fig. 12, each square in the RAW image 401 represents a pixel. A to H assigned to each pixel represent example characteristic values ​​of the transmission wavelength characteristics. As shown in Fig. 12, the pixels in the RAW image 401 are arranged in an arrangement pattern in which pixel groups in each wavelength range A to H are grouped together, as indicated by the thick lines. Furthermore, adjacent unit groups in the row direction are shifted in position in the column direction.

[0132] Therefore, even in the same wavelength range, there are two types of pixels that are shifted in rows and columns, such as pixel A0 and pixel A1 shown in Figure 12. Because pixel A0 and pixel A1 are shifted in rows and columns, the correlation between these two pixels is low.

[0133] When a RAW image from a multispectral image sensor with such a pixel layout is separated by color and reconstructed as described in Patent Document 1, the pixel A0 and pixel A1 described above are combined into a single reconstructed image in a state where they are adjacent to each other, which may reduce the coding efficiency of the reconstructed image.

[0134] <Using lattice points (positions)> Therefore, as shown in the top row of the table in FIG. 13, an image is reconstructed using the transmission wavelength characteristics and the lattice points (pixel positions) (method 2).

[0135] For example, in the example of Figure 12, if pixels A0 and A1 are to be processed, connecting the rows and columns of pixel A0 and the rows and columns of pixel A1 with the dotted lines or dashed-dotted lines shown in Figure 14 will form a rectangular lattice (square lattice) formed by the dotted lines or dashed-dotted lines. For example, each vertex (lattice point) of the square lattice formed by the dotted lines is pixel A0, and each vertex (lattice point) of the square lattice formed by the dashed-dotted lines is pixel A1. Therefore, pixels A0 and A1 are identified and reconstructed based on the lattice points of this square lattice.

[0136] <3-1-1. Reconstruction based on transmission wavelength characteristics and grid points (positions) of RAW images> For example, an encoder that encodes a RAW image may reconstruct the RAW image using transmission wavelength characteristics and lattice points, as shown in the second row from the top of the table in FIG. 13, and encode the reconstructed RAW image (method 2-1).

[0137] For example, in an image processing method, a RAW image, which is the output of a sensor having a pixel array made up of multiple types of pixels that detect incident light having mutually different wavelength characteristics, may be reconstructed using the transmission wavelength characteristics, which are the wavelength characteristics of the incident light, and the positions of the pixels, to generate a transmission wavelength characteristic lattice point reconstructed image made up of pixel values ​​of pixels that are located at the same row or column positions in the pixel array and that have detected incident light having mutually identical characteristic values ​​of the transmission wavelength characteristics, and the generated transmission wavelength characteristic lattice point reconstructed image may be encoded.

[0138] For example, an image processing device may include a reconstruction unit that reconstructs a RAW image, which is the output of a sensor having a pixel array composed of multiple types of pixels that detect incident light having mutually different wavelength characteristics, using the transmitted wavelength characteristics that are the wavelength characteristics of the incident light and the positions of the pixels, to generate a transmitted wavelength characteristic lattice point reconstructed image composed of pixel values ​​of pixels that are located at the same positions in the row direction or column direction in the pixel array and that have detected incident light having mutually identical characteristic values ​​of the transmitted wavelength characteristics, and an encoding unit that encodes the transmitted wavelength characteristic lattice point reconstructed image.

[0139] That is, the encoder reconstructs pixel values ​​of the RAW image output from the multispectral image sensor based on the transmission wavelength characteristics and the pixel positions, thereby generating a transmission wavelength characteristic lattice point reconstructed image composed of pixel values ​​of pixels located at the same row or column positions in the pixel array that have detected incident light having the same transmission wavelength characteristic values. For example, the encoder reconstructs the RAW image 401 in Fig. 14 to generate transmission wavelength characteristic lattice point reconstructed images 411 to 426 as shown in Fig. 15. By performing reconstruction in this manner, the encoder can separate and reconstruct pixel values ​​(distinguish between 0 and 1) based on not only the transmission wavelength characteristics (A to H) but also the pixel positions, as shown in Fig. 15.

[0140] The encoder then encodes the transmission wavelength characteristics and the transmission wavelength characteristic grid point reconstruction image generated for each pixel position.

[0141] By doing so, the encoder can generate a reconstructed image so as to suppress a decrease in prediction accuracy due to a shift in pixel position (row or column). Therefore, the encoder can suppress a decrease in encoding efficiency of the RAW image. Therefore, the encoder can suppress an increase in transmission data volume, power consumption, and electromagnetic noise. This also allows the encoder to suppress an increase in manufacturing costs and device size. Furthermore, suppressing a decrease in encoding efficiency can improve the subjective image quality of a decoded image when compared with the same transmission data volume. In other words, the encoder can suppress a decrease in subjective image quality of a decoded image.

[0142] In this embodiment, the transmission wavelength characteristic lattice point reconstructed image refers to an image reconstructed from the transmission wavelength characteristics and lattice points.

[0143] <Encoding order sort> The encoder may also arrange the transmission wavelength characteristic lattice point reconstruction images so that the same characteristic values ​​for the transmission wavelength characteristics are consecutive, as shown in the third row from the top of the table in Figure 13, and encode each transmission wavelength characteristic lattice point reconstruction image in that order (method 2-1-1).

[0144] That is, the encoder may encode the plurality of transmission wavelength characteristic lattice point reconstructed images in an order based on the characteristic values ​​of the transmission wavelength characteristics. For example, by encoding the transmission wavelength characteristic lattice point reconstructed images 411 to 426 shown in Fig. 15 in this order, it is possible to consecutively encode transmission wavelength characteristic lattice point reconstructed images having the same transmission wavelength characteristics (for example, the transmission wavelength characteristic lattice point reconstructed image 411 and the transmission wavelength characteristic lattice point reconstructed image 412).

[0145] By doing so, the encoder can increase prediction utilizing the correlation between transmission wavelength characteristic lattice point reconstructed images of the same characteristic value. Generally, the correlation between transmission wavelength characteristic lattice point reconstructed images of the same characteristic value is high, so by doing so, the encoder can suppress a decrease in prediction accuracy between images (transmission wavelength characteristic lattice point reconstructed images). In other words, the encoder can further suppress a decrease in encoding efficiency of RAW images.

[0146] <Encoding method> The encoder may encode the reconstructed image by applying prediction that utilizes correlation between pixels or between images, as shown in the fourth row from the top of the table in FIG. 13, for example (method 2-1-2).

[0147] That is, the encoder may encode the transmission wavelength characteristic lattice point reconstructed image using prediction that utilizes intra-image correlation or inter-image correlation, thereby enabling the encoder to further suppress a decrease in the encoding efficiency of the RAW image, as described above.

[0148] <3-1-2. Decoding and development of transmission wavelength characteristic grid point reconstruction image> For example, a decoder that decodes the encoded data of a RAW image may decode the encoded data and develop the resulting transmission wavelength characteristic grid point reconstruction image, as shown in the fifth row from the top of the table in Figure 13 (Method 2-2).

[0149] For example, in an image processing method, by decoding the encoded data, a RAW image, which is the output of a sensor having a pixel array composed of multiple types of pixels that detect incident light having mutually different wavelength characteristics, is reconstructed using transmission wavelength characteristics, which are the wavelength characteristics of the incident light, and the positions of the pixels, to generate a transmission wavelength characteristic lattice point reconstructed image composed of pixel values ​​of pixels that are located at the same row or column positions in the pixel array and that detect incident light having mutually identical characteristic values ​​of the transmission wavelength characteristics, and the generated transmission wavelength characteristic lattice point reconstructed image may be developed.

[0150] For example, an image processing device may include a decoding unit that decodes encoded data to generate a transmission wavelength characteristic lattice point reconstructed image composed of pixel values ​​of pixels located at the same row or column positions in the pixel array that have detected incident light having the same characteristic values ​​of the transmission wavelength characteristic, in which a RAW image, which is the output of a sensor having a pixel array composed of multiple types of pixels that detect incident light having different wavelength characteristics, is reconstructed using transmission wavelength characteristics that are the wavelength characteristics of the incident light and the positions of the pixels, and a development processing unit that develops the transmission wavelength characteristic lattice point reconstructed image.

[0151] That is, the decoder decodes the encoded data of the transmission wavelength characteristic lattice point reconstructed image generated by the encoder as described above, and develops the obtained transmission wavelength characteristic lattice point reconstructed image to generate a developed image. For example, when the encoded data is decoded to obtain the transmission wavelength characteristic lattice point reconstructed image shown in Fig. 15, the decoder performs the development process to generate developed images 431 to 438 (images for each wavelength range) as shown in Fig. 16.

[0152] Therefore, the decoder can correctly generate a developed image from the encoded data generated by the encoder. Therefore, the decoder can suppress a decrease in the encoding efficiency of the RAW image. Therefore, the decoder can suppress an increase in the amount of transmission data, power consumption, and electromagnetic noise. This also allows the decoder to suppress an increase in manufacturing costs and device size. Furthermore, suppressing a decrease in encoding efficiency can improve the subjective image quality of the decoded image when compared with the same amount of transmission data. In other words, the decoder can suppress a decrease in the subjective image quality of the decoded image.

[0153] <Image processing> The decoder may also perform image processing on the developed image generated by developing the transmission wavelength characteristic grid point reconstruction image, as shown in the sixth row from the top of FIG. 13 (method 2-2-1), for example.

[0154] The content of this image processing is arbitrary. For example, as shown in the seventh row from the top of Fig. 13, the decoder may perform noise reduction processing to reduce noise in the generated developed image (Method 2-2-1-1). Also, as shown in the eighth row from the top of Fig. 13, the decoder may enlarge or reduce the generated developed image (Method 2-2-1-2). By doing so, the decoder can suppress degradation of the subjective image quality of the decoded image.

[0155] <Wavelength expansion processing> Furthermore, the decoder may perform wavelength extension processing on the developed image generated by developing the transmission wavelength characteristic grid point reconstruction image, for example, as shown in the bottom row of FIG. 13 (method 2-2-2).

[0156] <3-2. Multispectral image sensor unit> Next, a configuration to which the present technology described above is applied in this embodiment will be described. Fig. 17 is a block diagram showing an example of the configuration of a multispectral image sensor unit, which is one aspect of an image processing device to which the present technology is applied. The multispectral image sensor unit 500 shown in Fig. 17 is a unit that uses a multispectral image sensor to generate a RAW image from which transmitted wavelengths and pixel values ​​for each pixel position can be acquired, and performs image processing on the RAW image.

[0157] Note that Fig. 17 shows the main processing units (blocks), data flows, etc., and does not necessarily show everything. In other words, in the multispectral image sensor unit 500, there may be processing units that are not shown as blocks in Fig. 17, and there may be processing and data flows that are not shown as arrows, etc. in Fig. 17.

[0158] 17, the multispectral image sensor unit 500 includes a detection unit 501 and an image processing unit 502. The detection unit 501 performs detection and outputs a RAW image (encoded data obtained by reconstructing and encoding the RAW image) as the detection result of the detection target.

[0159] The detection unit 501 includes a multispectral image sensor 511 , a reconstruction unit 512 , a reconstruction image encoding unit 513 , and a transmission unit 514 .

[0160] The multispectral image sensor 511 is a sensor having a pixel array composed of multiple types of pixels that detect incident light with different wavelength characteristics. As shown in the example of FIG. 12, the multispectral image sensor 511 has an arrangement pattern in which pixel groups, each consisting of a predetermined number of pixels, are arranged as unit groups. Adjacent unit groups in the row direction are shifted in position in the column direction. The multispectral image sensor 511 outputs the detection results of the incident light as a RAW image.

[0161] The reconstruction unit 512 acquires the RAW image output from the multispectral image sensor 511, and reconstructs the RAW image using the transmission wavelength characteristic, which is the wavelength characteristic of the incident light, and the pixel position, thereby generating a transmission wavelength characteristic lattice point reconstruction image composed of pixel values ​​of pixels located at the same row or column positions in the pixel array that detect incident light having the same characteristic values ​​of the transmission wavelength characteristic.

[0162] The reconstructed image encoding unit 513 encodes the transmission wavelength characteristic lattice point reconstructed image generated by the reconstruction unit 512 using a predetermined encoding method to generate encoded data. For example, the reconstructed image encoding unit 513 may encode the transmission wavelength characteristic lattice point reconstructed image using prediction utilizing intra-image correlation or inter-image correlation. Furthermore, the reconstructed image encoding unit 513 may encode multiple transmission wavelength characteristic lattice point reconstructed images in an order based on the characteristic values ​​of the transmission wavelength characteristics.

[0163] The transmission unit 514 transmits the coded data generated by the reconstructed image coding unit 513 to the image processing unit 502 .

[0164] The reconstructing unit 512 and the reconstructed image encoding unit 513 can also be regarded as an encoding unit 531 .

[0165] The image processing unit 502 performs image processing on the RAW image generated by the detection unit 501. As shown in FIG. 17 , the image processing unit 502 includes a receiving unit 551, a reconstructed image decoding unit 552, a development processing unit 553, an image processing unit 554, and a wavelength extension processing unit 555.

[0166] The receiving unit 551 acquires the coded data transmitted from the detecting unit 501 (transmitting unit 514 ), and supplies it to the reconstructed image decoding unit 552 .

[0167] The reconstructed image decoding unit 552 decodes the encoded data supplied from the receiving unit 551 and generates (restores) a transmission wavelength characteristic lattice point reconstructed image. This transmission wavelength characteristic lattice point reconstructed image is the reconstructed image generated by the reconstruction unit 512 of the detection unit 501 as described above. In other words, by decoding the encoded data, the reconstructed image decoding unit 552 generates a transmission wavelength characteristic lattice point reconstructed image composed of pixel values ​​of pixels located at the same row or column positions in the pixel array that detected incident light having the same characteristic values ​​of the transmission wavelength characteristic, in which a RAW image that is the output of a sensor having a pixel array configured with multiple types of pixels that detect incident light having different wavelength characteristics is reconstructed using the transmission wavelength characteristic that is the wavelength characteristic of the incident light and the pixel position.

[0168] The development processing unit 553 develops the transmission wavelength characteristic lattice point reconstructed image generated by the reconstructed image decoding unit 552 to generate a developed image.

[0169] The image processing unit 554 performs image processing on the developed image generated by the development processing unit 553. The content of this image processing is arbitrary. For example, the image processing unit 554 may perform processing to reduce noise as the image processing. Furthermore, the image processing unit 554 may perform enlargement or reduction of the developed image as the image processing.

[0170] The wavelength extension processing unit 555 performs wavelength extension processing on the developed image that has been appropriately image processed by the image processing unit 554. The wavelength extension processing unit 555 outputs the processed developed image to the outside of the image processing unit 502, i.e., to the outside of the multispectral image sensor unit 500.

[0171] The reconstructed image decoding unit 552 and the development processing unit 553 can also be regarded as a decoding unit 561 .

[0172] With the above-described configuration, the multispectral image sensor unit 500 can further suppress a decrease in the coding efficiency of the RAW image.

[0173] <Detection process flow> Next, a description will be given of the processing executed by the multispectral image sensor unit 500. An example of the flow of the detection processing executed by the detection section 501 will be described with reference to the flowchart of FIG.

[0174] When the detection process starts, the multispectral image sensor 511 detects incident light and generates a RAW image in step S501.

[0175] In step S502, the reconstruction unit 512 reconstructs the RAW image generated in step S501 using the transmission wavelength characteristics and lattice points to generate a transmission wavelength characteristics lattice point reconstruction image. That is, the reconstruction unit 512 reconstructs an image from the RAW image using only pixels that have the same characteristic value and are located at lattice points of the same square lattice. The arrangement of pixels in each transmission wavelength characteristics lattice point reconstruction image maintains the order of the matrix in the RAW image. When the arrangement of pixels with the same characteristic value is covered by n different square lattices, a total of nxM images are generated.

[0176] In other words, the reconstruction unit 512 reconstructs a RAW image, which is the output of a sensor having a pixel array composed of multiple types of pixels that detect incident light having different wavelength characteristics, using the transmission wavelength characteristic, which is the wavelength characteristic of the incident light, and the position of the pixel, to generate a transmission wavelength characteristic lattice point reconstruction image composed of pixel values ​​of pixels that are located at the same row or column positions in the pixel array and that have detected incident light having the same characteristic values ​​of the transmission wavelength characteristic.

[0177] In step S503, the reconstructed image encoding unit 513 arranges the transmission wavelength characteristic grid point reconstructed images generated in step S502 in the encoding order.

[0178] In step S504, the reconstructed image encoding unit 513 encodes the transmission wavelength characteristic grid point reconstructed image in the order sorted in step S503 to generate encoded data. The reconstructed image encoding unit 513 encodes the generated nxN images using an image compression method. Image compression may be performed using a method that uses correlation within the same image or a method that uses correlation between images. When correlation between images is used, the order of compression processing is such that images having a common characteristic value for a certain optical characteristic are encoded consecutively. This strengthens the correlation between neighboring images, improving encoding performance.

[0179] In step S505, the transmission unit 314 transmits the encoded data generated by the process in step S504 to the image processing unit 502 (reception unit 551).

[0180] When the process of step S505 is completed, the image processing ends.

[0181] <Image processing flow> Next, an example of the flow of image processing executed by the image processing unit 502 will be described with reference to the flowchart of FIG.

[0182] When image processing starts, in step S551, the receiving unit 551 receives the coded data transmitted in step S505 of FIG.

[0183] In step S552, the reconstructed image decoding unit 552 decodes the encoded data and generates (restores) nxN transmission wavelength characteristic lattice point reconstructed images.

[0184] In other words, by decoding the encoded data, the reconstructed image decoding unit 552 generates a transmission wavelength characteristic lattice point reconstructed image composed of pixel values ​​of pixels located at the same row or column positions in the pixel array that have detected incident light having the same characteristic values ​​of the transmission wavelength characteristic, in which the RAW image, which is the output of a sensor having a pixel array composed of multiple types of pixels that detect incident light having different wavelength characteristics, is reconstructed using the transmission wavelength characteristic, which is the wavelength characteristic of the incident light, and the positions of the pixels.

[0185] In step S553, the development processing unit 553 executes development processing on the transmission wavelength characteristic grid point reconstruction image generated in step S552, and generates a developed image.

[0186] In step S554, the image processing unit 554 performs image processing such as noise reduction and scaling.

[0187] In step S555, the wavelength extension processing unit 555 executes wavelength extension processing on the developed image that has been subjected to image processing in step S554.

[0188] When the process of step S555 is completed, the image processing ends.

[0189] By performing each process as described above, the multispectral image sensor unit 500 can further suppress a decrease in the coding efficiency of the RAW image.

[0190] 4. Third Embodiment <4-1. Image reconstruction based on intra-block position> <Sensor with phase difference detection pixels> Even in image sensors designed to capture visible images, there are cases where pixel value levels fluctuate locally at a certain spatial period due to the device structure and various other factors. For example, in a technology called image plane phase detection autofocus, phase difference detection pixels are arranged at a certain spatial period on the image sensor.

[0191] An example of a pixel block formed by a predetermined number of pixels in an on-chip phase-difference AF image sensor, which is an image sensor having such phase-difference detection pixels and performs on-chip phase-difference focusing processing using the phase-difference detection pixels, is shown in Fig. 20. As shown in Fig. 20, phase-difference detection pixels are provided at predetermined positions in a pixel block 601 of such an on-chip phase-difference AF image sensor. In the example of Fig. 20, gray pixels W11 and W33 are phase-difference detection pixels.

[0192] Generally, due to their structure, phase difference detection pixels have lower light sensitivity than normal pixels for obtaining pixel values ​​that constitute a RAW image. Therefore, there is a risk that the correlation between phase difference detection pixels and normal pixels is low, resulting in a high spatial frequency.

[0193] When a RAW image of an on-chip phase difference AF image sensor (an image sensor having phase difference detection pixels) with such a pixel layout is separated by color and reconstructed as described in Patent Document 1, the pixel values ​​of the phase difference detection pixels will be included in the reconstructed image, which could reduce the coding efficiency of the reconstructed image.

[0194] <Using position within a block> Therefore, as shown in the top row of the table in FIG. 21, the image is reconstructed based on the position within the block (method 3).

[0195] For example, the pixel block shown in Fig. 20 is divided into individual pixels and reconstructed into different reconstructed images. That is, pixels at the same positions in each pixel block are collected to generate reconstructed images.

[0196] <4-1-1. Reconstruction of RAW image by position within block> For example, an encoder that encodes a RAW image may reconstruct the RAW image based on its position within a block and encode it, as shown in the second row from the top of the table in FIG. 21 (method 3-1).

[0197] For example, in an image processing method, a RAW image, which is the output of a sensor having a pixel array made up of multiple types of pixels with different light-receiving sensitivities, may be reconstructed by intra-block positions, which are pixel positions within a pixel block made up of a predetermined number of pixels including multiple types of pixels, to generate an intra-block position reconstructed image made up of pixel values ​​of pixels whose intra-block positions are identical, and then the generated intra-block position reconstructed image may be encoded.

[0198] For example, an image processing device may include a reconstruction unit that reconstructs a RAW image, which is the output of a sensor having a pixel array composed of multiple types of pixels with different light-receiving sensitivities, using intra-block positions, which are pixel positions within a pixel block composed of a predetermined number of pixels including multiple types of pixels, to generate an intra-block position reconstructed image composed of pixel values ​​of pixels whose intra-block positions are identical, and an encoding unit that encodes the intra-block position reconstructed image.

[0199] That is, the encoder reconstructs, according to the position within the block, pixel values ​​of a RAW image output from a sensor having a pixel array made up of multiple types of pixels with different light-receiving sensitivities, thereby generating an intra-block-position reconstructed image made up of pixel values ​​of pixels at the same intra-block position. For example, the encoder reconstructs a RAW image having a pixel block 601 as shown in Fig. 20, and generates intra-block-position reconstructed images 611-1 to 611-62, as well as intra-block-position reconstructed images 612-1 and 612-2, which are reconstructed images for each intra-block position, as shown in Fig. 22.

[0200] 22, intra-block position reconstructed images 611-1 to 611-62 are reconstructed images of normal pixels. Intra-block position reconstructed images 612-1 and 612-2 are reconstructed images of phase difference detection pixels. In other words, by doing this, the encoder can reconstruct pixel values ​​with different light receiving sensitivities into different reconstructed images.

[0201] Then, the encoder encodes the reconstructed image at each position within the block.

[0202] By doing so, the encoder can generate a reconstructed image such that pixels with different light receiving sensitivities are assigned to different images. Therefore, the encoder can suppress a decrease in prediction accuracy due to differences in light receiving sensitivities. This allows the encoder to suppress a decrease in encoding efficiency of the RAW image. Therefore, the encoder can suppress an increase in transmission data volume, power consumption, and electromagnetic noise. This also allows the encoder to suppress an increase in manufacturing costs and device size. Furthermore, suppressing a decrease in encoding efficiency can improve the subjective image quality of a decoded image when compared with the same transmission data volume. In other words, the encoder can suppress a decrease in subjective image quality of a decoded image.

[0203] In this embodiment, the intra-block position reconstructed image refers to an image reconstructed based on an intra-block position.

[0204] <Encoding order sort> The encoder may also arrange the intra-block position reconstructed images so that the same characteristic values ​​for the light sensitivity characteristics are consecutive, as shown in the third row from the top of the table in Figure 21, and encode each intra-block position reconstructed image in that order (method 3-1-1).

[0205] That is, the encoder may encode the plurality of intra-block-position reconstructed images in an order based on the light sensitivity. For example, the encoder may successively encode the intra-block-position reconstructed images 611-1 to 611-62 shown in Fig. 22, and then encode the intra-block-position reconstructed images 612-1 and 612-2. In this manner, the encoder can successively encode intra-block-position reconstructed images having the same light sensitivity characteristics.

[0206] By doing so, the encoder can increase predictions that utilize the correlation between intra-block-position reconstructed images with the same characteristic value. Generally, there is a high correlation between intra-block-position reconstructed images with the same light-receiving sensitivity, so by doing so, the encoder can suppress a decrease in prediction accuracy between images (intra-block-position reconstructed images). In other words, the encoder can further suppress a decrease in the encoding efficiency of RAW images.

[0207] <Encoding method> The encoder may encode the reconstructed image by applying prediction that utilizes correlation between pixels or between images, as shown in the fourth row from the top of the table in FIG. 21 (method 3-1-2).

[0208] That is, the encoder may encode the intra-block position reconstructed image using prediction that utilizes intra-image correlation or inter-image correlation, thereby enabling the encoder to further suppress a decrease in the encoding efficiency of the RAW image, as described above.

[0209] <4-1-2. Decoding and development of intra-block position reconstructed image> For example, a decoder that decodes the coded data of a RAW image may decode the coded data and develop the resulting intra-block position reconstructed image, as shown in the fifth row from the top of the table in Figure 21 (Method 3-2).

[0210] For example, in an image processing method, by decoding encoded data, a RAW image, which is the output of a sensor having a pixel array composed of multiple types of pixels with different light receiving sensitivities, is reconstructed by intra-block positions, which are pixel positions within a pixel block composed of a predetermined number of pixels including multiple types of pixels, to generate an intra-block position reconstructed image composed of pixel values ​​of pixels whose intra-block positions are identical, and the generated intra-block position reconstructed image may then be developed.

[0211] For example, an image processing device may include a decoding unit that decodes encoded data to generate an intra-block position reconstructed image composed of pixel values ​​of pixels whose intra-block positions are identical, where the RAW image, which is the output of a sensor having a pixel array composed of multiple types of pixels with different light-receiving sensitivities, is reconstructed by intra-block positions, which are pixel positions within a pixel block composed of a predetermined number of pixels including multiple types of pixels, and a development processing unit that develops the intra-block position reconstructed image.

[0212] That is, the decoder decodes the coded data of the intra-block position reconstructed image generated by the encoder as described above, and develops the obtained intra-block position reconstructed image to generate a developed image. For example, when the coded data is decoded to obtain the intra-block position reconstructed image shown in Fig. 22, the decoder performs the development process to generate developed images 631 to 633 (images for each light receiving sensitivity) shown in Fig. 23.

[0213] Therefore, the decoder can correctly generate a developed image from the encoded data generated by the encoder. Therefore, the decoder can suppress a decrease in the encoding efficiency of the RAW image. Therefore, the decoder can suppress an increase in the amount of transmission data, power consumption, and electromagnetic noise. This also allows the decoder to suppress an increase in manufacturing costs and device size. Furthermore, suppressing a decrease in encoding efficiency can improve the subjective image quality of the decoded image when compared with the same amount of transmission data. In other words, the decoder can suppress a decrease in the subjective image quality of the decoded image.

[0214] <Image processing> The decoder may also perform image processing on the developed image generated by developing the intra-block position reconstructed image, as shown in the sixth row from the top of FIG. 21 (method 3-2-1), for example.

[0215] The content of this image processing is arbitrary. For example, as shown in the seventh row from the top of Fig. 21, the decoder may perform noise reduction processing to reduce noise in the generated developed image (Method 3-2-1-1). Also, as shown in the eighth row from the top of Fig. 21, the decoder may enlarge or reduce the generated developed image (Method 3-2-1-2). By doing so, the decoder can suppress degradation of the subjective image quality of the decoded image.

[0216] <Focus processing> Furthermore, the decoder may perform focusing processing of the image sensor using a developed image of the phase difference detection pixel generated by developing the intra-block position reconstructed image, for example, as shown in the bottom row of Figure 21 (method 3-2-2).

[0217] <4-2. Phase-detection AF image sensor unit> Next, a configuration to which the present technology described above is applied in this embodiment will be described. Fig. 24 is a block diagram showing an example of the configuration of an on-chip phase difference AF image sensor unit, which is one aspect of an image processing device to which the present technology is applied. The on-chip phase difference AF image sensor unit 700 shown in Fig. 24 is a unit that generates a RAW image using an on-chip phase difference AF image sensor and performs image processing on the RAW image. Note that the on-chip phase difference AF image sensor has phase difference detection pixels. The on-chip phase difference AF image sensor unit 700 can perform on-chip phase difference focusing processing using the phase difference detection pixels.

[0218] Note that Fig. 24 shows the main processing units (blocks), data flows, etc., and does not necessarily include everything shown in Fig. 24. In other words, in the on-chip phase difference AF image sensor unit 700, there may be processing units that are not shown as blocks in Fig. 24, and there may be processing or data flows that are not shown as arrows or the like in Fig. 24.

[0219] 24, the on-chip phase difference AF image sensor unit 700 has a detection unit 701 and an image processing unit 702. The detection unit 701 performs detection and outputs a RAW image (encoded data obtained by reconstructing and encoding the RAW image) as the detection result of the detection target.

[0220] The detection unit 701 includes an image sensor 711 , a reconstruction unit 712 , a reconstruction image encoding unit 713 , and a transmission unit 714 .

[0221] The image sensor 711 is a sensor having a pixel array configured with pixels that detect incident light. The image sensor 711 outputs the detection result of the incident light as a RAW image. The pixel array also includes phase difference detection pixels. In other words, the image sensor 711 is a sensor having a pixel array configured with multiple types of pixels that differ from each other in light receiving sensitivity. The image sensor 711 can perform focusing processing using an image plane phase difference method using pixel values ​​of the phase difference detection pixels.

[0222] The reconstruction unit 712 acquires the RAW image output from the image sensor 711 and reconstructs the RAW image based on intra-block positions, which are pixel positions within a pixel block consisting of a predetermined number of pixels including multiple types of pixels, to generate an intra-block position reconstructed image consisting of pixel values ​​of pixels whose intra-block positions are the same.

[0223] The reconstructed image encoding unit 713 encodes the intra-block-position reconstructed image generated by the reconstruction unit 712 using a predetermined encoding method to generate encoded data. For example, the reconstructed image encoding unit 713 may encode the intra-block-position reconstructed image using prediction utilizing intra-image correlation or inter-image correlation. Furthermore, the reconstructed image encoding unit 713 may encode multiple intra-block-position reconstructed images in an order based on light-receiving sensitivity.

[0224] The transmission unit 714 transmits the coded data generated by the reconstructed image coding unit 713 to the image processing unit 702 .

[0225] The reconstructor 712 and the reconstructed image encoder 713 can also be regarded as an encoder 731 .

[0226] The image processing unit 702 performs image processing on the RAW image generated by the detection unit 701. As shown in FIG. 24 , the image processing unit 702 includes a receiving unit 751, a reconstructed image decoding unit 752, a development processing unit 753, an image processing unit 754, and a focusing processing unit 755.

[0227] The receiving unit 751 acquires the coded data transmitted from the detecting unit 701 (transmitting unit 714 ), and supplies it to the reconstructed image decoding unit 752 .

[0228] The reconstructed image decoding unit 752 decodes the coded data supplied from the receiving unit 751 and generates (restores) an intra-block-position reconstructed image. This intra-block-position reconstructed image is the reconstructed image generated by the reconstruction unit 712 of the detection unit 701 as described above. In other words, the reconstructed image decoding unit 752 generates an intra-block-position reconstructed image composed of pixel values ​​of pixels at the same intra-block positions, which are pixel positions within a pixel block composed of a predetermined number of pixels including multiple types of pixels, reconstructed from a RAW image, which is the output of a sensor having a pixel array composed of multiple types of pixels with different light-receiving sensitivities, by intra-block positions.

[0229] The development processing unit 753 develops the intra-block position reconstructed image generated by the reconstructed image decoding unit 752 to generate a developed image.

[0230] The image processing unit 754 performs image processing on the developed image generated by the development processing unit 753. The content of this image processing is arbitrary. For example, the image processing unit 754 may perform processing to reduce noise as the image processing. Furthermore, the image processing unit 754 may perform enlargement or reduction of the developed image as the image processing.

[0231] The focusing processing unit 755 controls the image sensor 711 using the developed image of the phase difference detection pixels to perform focusing processing to focus the image sensor 711 on the subject. The focusing processing unit 755 outputs the developed image to the outside of the image processing unit 702, i.e., to the outside of the on-chip phase difference AF image sensor unit 700.

[0232] The reconstructed image decoding unit 752 and the development processing unit 753 can also be regarded as a decoding unit 761 .

[0233] With the above-described configuration, the on-chip phase difference AF image sensor unit 700 can further suppress a decrease in the coding efficiency of RAW images.

[0234] <Detection process flow> Next, a description will be given of the processing executed by the on-chip phase difference AF image sensor unit 700. An example of the flow of the detection processing executed by the detection unit 701 will be described with reference to the flowchart in FIG.

[0235] When the detection process starts, the image sensor 711 detects incident light and generates a RAW image in step S701.

[0236] In step S702, the reconstruction unit 712 reconstructs the RAW image generated in step S701 based on intra-block positions to generate an intra-block position reconstructed image. That is, the reconstruction unit 712 reconstructs from the RAW image based on intra-block positions, which are pixel positions within a pixel block composed of a predetermined number of pixels including multiple types of pixels, to generate an intra-block position reconstructed image composed of pixel values ​​of pixels having the same intra-block positions. That is, for a pixel group of M rows and N columns that constitutes a period, an image is reconstructed using only pixels located in the same row and column within the pixel group. The pixel arrangement in each reconstructed image maintains the matrix order in the RAW image. A total of MxN images are generated.

[0237] In other words, the reconstruction unit 712 reconstructs a RAW image, which is the output of a sensor having a pixel array composed of multiple types of pixels with different light receiving sensitivities, using intra-block positions, which are pixel positions within a pixel block composed of a predetermined number of pixels including multiple types of pixels, to generate an intra-block position reconstructed image composed of pixel values ​​of pixels whose intra-block positions are the same.

[0238] In step S703, the reconstructed image encoding unit 713 arranges the intra-block position reconstructed images generated in step S702 in encoding order.

[0239] In step S704, the reconstructed image encoding unit 713 encodes the reconstructed images in the block in the order sorted in step S703, generating encoded data. The reconstructed image encoding unit 713 encodes the generated MxN images using an image compression method. Image compression may be performed using a method that uses correlation within the same image or a method that uses correlation between images. When correlation between images is used, the order of compression processing is such that images having a common characteristic value for a certain optical characteristic are encoded consecutively. This strengthens the correlation between neighboring images, improving encoding performance.

[0240] In step S705, the transmission unit 714 transmits the coded data generated by the process in step S704 to the image processing unit 702 (reception unit 751).

[0241] When the process of step S705 is completed, the image processing ends.

[0242] <Image processing flow> Next, an example of the flow of image processing executed by the image processing unit 702 will be described with reference to the flowchart of FIG.

[0243] When image processing starts, in step S751, the receiving unit 751 receives the coded data transmitted in step S705 of FIG.

[0244] In step S752, the reconstructed image decoding unit 752 decodes the coded data and generates (restores) MxN intra-block position reconstructed images.

[0245] In other words, by decoding the encoded data, the reconstructed image decoding unit 752 generates an intra-block position reconstructed image composed of pixel values ​​of pixels whose intra-block positions are identical, in which a RAW image, which is the output of a sensor having a pixel array composed of multiple types of pixels with different light receiving sensitivities, is reconstructed by intra-block positions, which are pixel positions within a pixel block composed of a predetermined number of pixels including multiple types of pixels.

[0246] In step S753, the development processing unit 753 executes development processing on the intra-block position reconstructed image generated in step S752, and generates a developed image.

[0247] In step S754, the image processing unit 754 performs image processing such as noise reduction and scaling.

[0248] In step S755, the focusing processing unit 755 executes focusing processing to focus the image sensor 711 on the subject, using the developed image configured by the phase difference detection pixels generated in step S753.

[0249] When the process of step S755 is completed, the image processing ends.

[0250] By performing each process as described above, the on-chip phase difference AF image sensor unit 700 can further suppress a decrease in the coding efficiency of the RAW image.

[0251] <5. Combination> The present technology described above in the first embodiment, the second embodiment, and the third embodiment may be applied in appropriate combination.

[0252] <Combination of the first and second embodiments> For example, in the first embodiment, an encoder was described that reconstructs a RAW image, which is the output of a sensor having a pixel array made up of multiple types of pixels that detect incident light having different optical characteristics including at least the polarization angle characteristics, using the optical characteristics including at least the polarization angle characteristics of the incident light, to generate a polarization angle characteristics reconstructed image made up of pixel values ​​of pixels that detect incident light having the same characteristic values ​​of at least the polarization angle characteristics, and encodes the generated polarization angle characteristics reconstructed image.

[0253] As described in the second embodiment, this encoder may reconstruct a RAW image using the transmission wavelength characteristics and pixel positions to generate a transmission wavelength characteristics lattice point reconstructed image composed of pixel values ​​of pixels located at the same row or column positions in the pixel array and detecting incident light having the same characteristic values ​​of the transmission wavelength characteristics, and may generate a polarization angle characteristics reconstructed image by reconstructing the transmission wavelength characteristics lattice point reconstructed image using the polarization angle characteristics.

[0254] 9, the reconstruction unit 312 may have the reconstruction unit 512 of FIG. 17 instead of the transmission wavelength characteristic reconstruction unit 321, and the reconstruction unit 512 may reconstruct the RAW image using the transmission wavelength characteristic, which is the wavelength characteristic of the incident light, and the pixel position, to generate a transmission wavelength characteristic lattice point reconstructed image composed of pixel values ​​of pixels located at the same row or column positions in the pixel array that have detected incident light having the same characteristic values ​​of the transmission wavelength characteristic. Then, the polarization angle characteristic reconstruction unit 322 may reconstruct the transmission wavelength characteristic lattice point reconstructed image using the polarization angle characteristic to generate a polarization angle characteristic reconstructed image.

[0255] 10, the reconstruction unit 512 may reconstruct the RAW image using the transmission wavelength characteristics and lattice points to generate a transmission wavelength characteristics lattice point reconstructed image, similar to step S502 in Fig. 18. Then, in step S303 in Fig. 10, the reconstruction unit 312 (polarization angle characteristics reconstruction unit 322) may reconstruct the transmission wavelength characteristics lattice point reconstructed image using the polarization angle characteristics to generate a polarization angle characteristics reconstructed image.

[0256] Even in such a case, the polarization sensor unit 300 can further suppress a decrease in the coding efficiency of the RAW image, as in the first embodiment.

[0257] In this case, the encoder may also encode the multiple polarization angle characteristic reconstructed images in the order based on the characteristic values ​​of the transmission wavelength characteristics. That is, in the polarization sensor unit 300 of Fig. 9, the reconstructed image encoding unit 313 may encode the multiple polarization angle characteristic reconstructed images in the order based on the characteristic values ​​of the transmission wavelength characteristics.

[0258] Furthermore, for example, in the first embodiment, a decoder was described that decodes encoded data to generate a polarization angle characteristic reconstructed image composed of pixel values ​​of pixels that detect incident light having at least the same characteristic values ​​of the polarization angle characteristic, reconstructed from a RAW image, which is the output of a sensor having a pixel array composed of multiple types of pixels that detect incident light having different optical characteristics including at least the polarization angle characteristic, using the optical characteristics including at least the polarization angle characteristic of the incident light, and then develops the generated polarization angle characteristic reconstructed image.

[0259] In this decoder, as described in the second embodiment, the above-mentioned optical characteristics may further include transmission wavelength characteristics, which are wavelength characteristics of incident light, and the above-mentioned polarization angle characteristic reconstructed image may be an image obtained by reconstructing a RAW image using the transmission wavelength characteristics and pixel positions, and then reconstructing it using the polarization angle characteristics.

[0260] For example, in the polarization sensor unit 300 of FIG. 9, the polarization angle characteristic reconstructed image generated by the reconstructed image decoding unit 352 by decoding the encoded data may be an image obtained by reconstructing a RAW image based on the transmission wavelength characteristics and pixel positions, and then further reconstructing the image based on the polarization angle characteristics.

[0261] Also, for example, in step S352 of FIG. 11, the polarization angle characteristic reconstructed image generated by the reconstructed image decoding unit 352 by decoding the encoded data may be an image obtained by reconstructing a RAW image based on the transmission wavelength characteristics and pixel positions, and then further reconstructing the image based on the polarization angle characteristics.

[0262] Even in such a case, the polarization sensor unit 300 can further suppress a decrease in the coding efficiency of the RAW image, as in the first embodiment.

[0263] In this case, too, the decoder may perform wavelength expansion processing on the developed image, as explained in the second embodiment.

[0264] For example, the image processing unit 302 of the polarization sensor unit 300 in Fig. 9 may have a wavelength extension processing unit 555 shown in Fig. 17 after the image processing unit 355. In addition, in the image processing in Fig. 11, the focusing processing unit 755 may execute the processing of step S555 in Fig. 19 after the processing of step S355.

[0265] <Combination of the first and third embodiments> Furthermore, as described in the third embodiment, the encoder described in the first embodiment may reconstruct a RAW image based on intra-block positions, which are pixel positions within a pixel block composed of a predetermined number of pixels with multiple types of light sensitivity, to generate an intra-block position reconstructed image composed of pixel values ​​of pixels having the same intra-block positions, and then reconstruct the intra-block reconstructed image based on polarization angle characteristics to generate a polarization angle characteristic reconstructed image.

[0266] 9, the reconstruction unit 312 may have the reconstruction unit 712 of FIG. 24 instead of the transmission wavelength characteristics reconstruction unit 321, and the reconstruction unit 712 may reconstruct the RAW image based on intra-block positions, which are pixel positions within a pixel block composed of a predetermined number of pixels with multiple types of light sensitivity, to generate an intra-block position reconstructed image composed of pixel values ​​of pixels at the same intra-block positions. Then, the polarization angle characteristics reconstruction unit 322 may reconstruct the intra-block position reconstructed image based on the polarization angle characteristics to generate a polarization angle characteristics reconstructed image.

[0267] 10, the reconstruction unit 512 may reconstruct the RAW image based on the intra-block position to generate an intra-block position reconstructed image, similar to step S702 in Fig. 25. Then, in step S303 in Fig. 10, the reconstruction unit 312 (polarization angle characteristics reconstruction unit 322) may reconstruct the intra-block position reconstructed image based on the polarization angle characteristics to generate a polarization angle characteristics reconstructed image.

[0268] Even in such a case, the polarization sensor unit 300 can further suppress a decrease in the coding efficiency of the RAW image, as in the first embodiment.

[0269] In this case, the encoder may encode the multiple polarization angle characteristic reconstructed images in the order based on the light receiving sensitivity. That is, in the polarization sensor unit 300 of Fig. 9, the reconstructed image encoding unit 313 may encode the multiple polarization angle characteristic reconstructed images in the order based on the light receiving sensitivity.

[0270] Furthermore, for example, in the decoder described above in the first embodiment, as explained in the third embodiment, the polarization angle characteristic reconstructed image may be an image in which the RAW image is reconstructed based on the intra-block positions, which are pixel positions within a pixel block composed of a predetermined number of pixels with multiple types of light sensitivity, and further reconstructed based on the polarization angle characteristics.

[0271] For example, in the polarization sensor unit 300 of Figure 9, the polarization angle characteristic reconstructed image generated by the reconstructed image decoding unit 352 by decoding the encoded data may be an image obtained by reconstructing a RAW image based on its position within a block and then further reconstructing it based on the polarization angle characteristic.

[0272] Also, for example, in step S352 of FIG. 11, the polarization angle characteristic reconstructed image generated by the reconstructed image decoding unit 352 by decoding the encoded data may be an image obtained by reconstructing a RAW image based on its position within a block and then reconstructing it based on the polarization angle characteristic.

[0273] Even in such a case, the polarization sensor unit 300 can further suppress a decrease in the coding efficiency of the RAW image, as in the first embodiment.

[0274] In this case, too, as explained in the third embodiment, the decoder may perform the focusing process based on the developed image.

[0275] For example, the image processing unit 302 of the polarization sensor unit 300 in Fig. 9 may have a focusing processing unit 755 shown in Fig. 24 after the image processing unit 355. In addition, in the image processing in Fig. 11, the focusing processing unit 755 may execute the processing of step S755 in Fig. 26 after the processing of step S355.

[0276] <Combination of the second and third embodiments> In addition, in the second embodiment, an encoder has been described that reconstructs a RAW image, which is the output of a sensor having a pixel array composed of multiple types of pixels that detect incident light having mutually different wavelength characteristics, using the transmission wavelength characteristics that are the wavelength characteristics of the incident light and the positions of the pixels, thereby generating a transmission wavelength characteristics lattice point reconstructed image composed of pixel values ​​of pixels that are located at the same positions in the row direction or column direction in the pixel array and that have detected incident light having mutually identical characteristic values ​​of the transmission wavelength characteristics, and encodes the generated transmission wavelength characteristics lattice point reconstructed image.

[0277] As described in the third embodiment, this encoder may reconstruct a RAW image based on intra-block positions, which are pixel positions within a pixel block composed of a predetermined number of pixels with multiple types of light sensitivity, to generate an intra-block position reconstructed image composed of pixel values ​​of pixels having the same intra-block positions, and may generate a transmission wavelength characteristic lattice point reconstructed image by reconstructing the intra-block reconstructed image based on the transmission wavelength characteristics and the pixel positions.

[0278] For example, in the multispectral image sensor unit 500 of Fig. 17, the reconstruction unit 512 may reconstruct a RAW image by intra-block positions, which are pixel positions within a pixel block composed of a predetermined number of pixels with multiple types of light sensitivity, as in the reconstruction unit 712 of Fig. 24, to generate an intra-block position reconstructed image composed of pixel values ​​of pixels having the same intra-block positions. Furthermore, the reconstruction unit 512 may reconstruct the intra-block reconstructed image by the transmission wavelength characteristics and the pixel positions, to generate a transmission wavelength characteristic lattice point reconstructed image.

[0279] 18, the reconstruction unit 512 may reconstruct the RAW image based on the intra-block position to generate an intra-block position reconstructed image, similar to step S702 in Fig. 25. Then, the reconstruction unit 512 may further reconstruct the intra-block reconstructed image based on the transmission wavelength characteristics and the pixel positions to generate a transmission wavelength characteristic grid point reconstructed image.

[0280] Even in such a case, the multispectral image sensor unit 500 can further suppress a decrease in the coding efficiency of the RAW image, similar to the second embodiment.

[0281] In this case, the encoder may encode the plurality of transmission wavelength characteristic lattice point reconstructed images in the order based on the light sensitivity. That is, in the multispectral image sensor unit 500 of Fig. 17, the reconstructed image encoding unit 513 may encode the plurality of transmission wavelength characteristic lattice point reconstructed images in the order based on the light sensitivity.

[0282] Furthermore, for example, in the decoder described above in the second embodiment, as explained in the third embodiment, the transmission wavelength characteristic lattice point reconstruction image may be an image in which the RAW image is reconstructed based on the intra-block positions, which are pixel positions within a pixel block composed of a predetermined number of pixels with multiple types of light sensitivity, and further reconstructed based on the transmission wavelength characteristics and the pixel positions.

[0283] For example, in the multispectral image sensor unit 500 of FIG. 17, the transmission wavelength characteristic lattice point reconstruction image generated by the reconstruction image decoding unit 552 by decoding the encoded data may be an image obtained by reconstructing a RAW image based on the position within the block and further reconstructing it based on the transmission wavelength characteristic and the position of the pixel.

[0284] Also, for example, in step S552 of FIG. 19, the transmission wavelength characteristic lattice point reconstructed image generated by the reconstructed image decoding unit 552 by decoding the encoded data may be an image obtained by reconstructing a RAW image based on its position within a block and further reconstructing it based on the transmission wavelength characteristic and the pixel position.

[0285] Even in such a case, the multispectral image sensor unit 500 can further suppress a decrease in the coding efficiency of the RAW image, similar to the second embodiment.

[0286] In this case, too, as explained in the third embodiment, the decoder may perform the focusing process based on the developed image.

[0287] For example, the image processing unit 502 of the multispectral image sensor unit 500 in Fig. 17 may have a focusing processing unit 755 shown in Fig. 24 after the image processing unit 554. In addition, in the image processing in Fig. 19, the focusing processing unit 755 may execute the processing of step S755 in Fig. 26 after the processing of step S555.

[0288] <Combination of the first, second and third embodiments> Of course, the present technology described above in the first embodiment, the present technology described above in the second embodiment, and the present technology described above in the third embodiment may be applied in combination.

[0289] <6. Notes> <Computer> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs constituting the software are installed on a computer. Here, the term "computer" includes computers built into dedicated hardware, and general-purpose personal computers, etc., that can execute various functions by installing various programs.

[0290] FIG. 27 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0291] In a computer 900 shown in FIG. 27, a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903 are interconnected via a bus 904.

[0292] An input / output interface 910 is also connected to the bus 904. To the input / output interface 910, an input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915 are connected.

[0293] The input unit 911 includes, for example, a keyboard, a mouse, a microphone, a touch panel, an input terminal, etc. The output unit 912 includes, for example, a display, a speaker, an output terminal, etc. The storage unit 913 includes, for example, a hard disk, a RAM disk, a non-volatile memory, etc. The communication unit 914 includes, for example, a network interface. The drive 915 drives removable media 921 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0294] In a computer configured as above, the CPU 901 executes the above-described series of processes by, for example, loading a program stored in the storage unit 913 into the RAM 903 via the input / output interface 910 and the bus 904 and executing the program. The RAM 903 also stores data necessary for the CPU 901 to execute various processes as appropriate.

[0295] The program executed by the computer can be applied by recording it on removable media 921 such as package media, for example. In this case, the program can be installed in storage unit 913 via input / output interface 910 by inserting removable media 921 into drive 915.

[0296] This program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, digital satellite broadcasting, etc. In this case, the program can be received by the communication unit 914 and installed in the storage unit 913.

[0297] Alternatively, this program can be installed in advance in the ROM 902 or the storage unit 913 .

[0298] <Applicable targets of this technology> The present technology can be applied to any configuration.

[0299] For example, this technology can be applied to various electronic devices, such as transmitters and receivers (e.g., television sets and mobile phones) used in satellite broadcasting, cable TV and other wired broadcasting, distribution over the Internet, and distribution to terminals via cellular communications, or devices (e.g., hard disk recorders and cameras) that record images on media such as optical disks, magnetic disks, and flash memories, or play images from these storage media.

[0300] Furthermore, for example, the present technology can also be implemented as a part of an apparatus, such as a processor (e.g., a video processor) as a system LSI (Large Scale Integration), a module (e.g., a video module) using multiple processors, a unit (e.g., a video unit) using multiple modules, or a set in which other functions are added to a unit (e.g., a video set).

[0301] Furthermore, for example, the present technology can also be applied to a network system configured with multiple devices. For example, the present technology may be implemented as cloud computing in which multiple devices share and collaborate on processing via a network. For example, the present technology may be implemented in a cloud service that provides image (video)-related services to any terminal, such as a computer, AV (Audio Visual) equipment, a portable information processing terminal, or an IoT (Internet of Things) device.

[0302] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0303] <Fields and applications where this technology can be applied> Systems, devices, processing units, etc. to which the present technology is applied can be used in any field, for example, transportation, medical care, crime prevention, agriculture, livestock farming, mining, beauty, factories, home appliances, weather, nature monitoring, etc. In addition, their uses are also arbitrary.

[0304] For example, the present technology can be applied to systems and devices used to provide viewing content, etc. Furthermore, for example, the present technology can also be applied to systems and devices used for transportation, such as monitoring traffic conditions and controlling automatic driving. Furthermore, for example, the present technology can also be applied to systems and devices used for security. Furthermore, for example, the present technology can also be applied to systems and devices used for automatic control of machines, etc. Furthermore, for example, the present technology can also be applied to systems and devices used for agriculture and livestock farming. Furthermore, for example, the present technology can also be applied to systems and devices used to monitor natural conditions, such as volcanoes, forests, and oceans, and wildlife. Furthermore, for example, the present technology can also be applied to systems and devices used for sports.

[0305] <Other> The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present technology.

[0306] For example, a configuration described as one device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, configurations described above as multiple devices (or processing units) may be combined and configured as one device (or processing unit). Of course, configurations other than those described above may be added to the configuration of each device (or each processing unit). Furthermore, as long as the configuration and operation of the entire system are substantially the same, part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0307] Furthermore, for example, the above-described program may be executed in any device, as long as the device has the necessary functions (functional blocks, etc.) and is able to obtain the necessary information.

[0308] Also, for example, each step of a single flowchart may be executed by one device, or may be shared and executed by multiple devices. Furthermore, when one step includes multiple processes, the multiple processes may be executed by one device, or may be shared and executed by multiple devices. In other words, multiple processes included in one step can be executed as multiple step processes. Conversely, processes described as multiple steps can be executed collectively as one step.

[0309] Furthermore, the program executed by the computer may have the following features. For example, the processing of the steps of writing the program may be executed in chronological order according to the order described in this specification. The processing of the steps of writing the program may also be executed in parallel. Furthermore, the processing of the steps of writing the program may be executed individually at the necessary timing, such as when called. In other words, as long as no contradiction occurs, the processing of each step may be executed in an order different from the order described above. Furthermore, the processing of the steps of writing the program may be executed in parallel with the processing of another program. Furthermore, the processing of the steps of writing the program may be executed in combination with the processing of another program.

[0310] Furthermore, for example, multiple technologies related to the present technology can be implemented independently and independently, as long as no contradiction occurs. Of course, any multiple technologies can also be implemented in combination. For example, part or all of the present technology described in any embodiment can be implemented in combination with part or all of the present technology described in another embodiment. Furthermore, part or all of any of the above-described present technologies can be implemented in combination with other technologies not described above.

[0311] The present technology can also be configured as follows. (1) a reconstruction unit that reconstructs a RAW image, which is an output of a sensor having a pixel array composed of multiple types of pixels that detect incident light having different optical characteristics including at least a polarization angle characteristic, using the optical characteristics including at least the polarization angle characteristic of the incident light, to generate a polarization angle characteristic reconstruction image composed of pixel values ​​of the pixels that detected the incident light having the same characteristic values ​​of at least the polarization angle characteristic; an encoding unit that encodes the polarization angle characteristic reconstructed image; An image processing device comprising: (2) The optical characteristics further include a transmission wavelength characteristic, which is a wavelength characteristic of the incident light. The image processing device according to (1). (3) The reconstruction unit reconstructs the RAW image using the transmission wavelength characteristics to generate a transmission wavelength characteristics reconstruction image configured with pixel values ​​of the pixels that detect the incident light having the same characteristic values ​​of the transmission wavelength characteristics, and reconstructs the transmission wavelength characteristics reconstruction image using the polarization angle characteristics to generate the polarization angle characteristics reconstruction image. (2) An image processing device according to the present invention. (4) The encoding unit encodes the plurality of polarization angle characteristic reconstructed images in an order based on the characteristic values ​​of the transmission wavelength characteristics. An image processing device according to (2) or (3). (5) The encoding unit encodes the polarization angle characteristic reconstructed image using prediction utilizing intra-image correlation or inter-image correlation. An image processing device according to any one of (2) to (4). (6) The reconstruction unit reconstructs the RAW image using the transmission wavelength characteristics and the positions of the pixels to generate a transmission wavelength characteristics lattice point reconstruction image configured with pixel values ​​of the pixels that detect the incident light having the same characteristic values ​​of the transmission wavelength characteristics and are located at the same positions in the row direction or column direction in the pixel array, and generates the polarization angle characteristics reconstruction image by reconstructing the transmission wavelength characteristics lattice point reconstruction image using the polarization angle characteristics. An image processing device according to any one of (2) to (5). (7) The encoding unit encodes the plurality of polarization angle characteristic reconstructed images in an order based on the characteristic value of the transmission wavelength characteristic. (6) An image processing device according to (6). (8) The reconstruction unit reconstructs the RAW image based on intra-block positions, which are pixel positions within a pixel block configured by a predetermined number of the pixels having a plurality of types of light receiving sensitivity, to generate an intra-block position reconstructed image configured by pixel values ​​of the pixels whose intra-block positions are identical to each other, and generates the polarization angle characteristic reconstructed image by reconstructing the intra-block reconstructed image based on the polarization angle characteristic. An image processing device according to any one of (1) to (7). (9) The encoding unit encodes the plurality of polarization angle characteristic reconstructed images in an order based on the light receiving sensitivity. (8) An image processing device according to (8). (10) A RAW image, which is an output of a sensor having a pixel array composed of a plurality of types of pixels that detect incident light having different optical characteristics including at least a polarization angle characteristic, is reconstructed by the optical characteristics including at least the polarization angle characteristic of the incident light, thereby generating a polarization angle characteristic reconstructed image composed of pixel values ​​of the pixels that detected the incident light having the same characteristic values ​​of at least the polarization angle characteristic; The generated polarization angle characteristic reconstruction image is encoded. Image processing methods.

[0312] (11) A decoding unit that decodes encoded data to generate a polarization angle characteristic reconstructed image, which is an output of a sensor having a pixel array composed of multiple types of pixels that detect incident light having different optical characteristics including at least the polarization angle characteristic, reconstructed by the optical characteristics including at least the polarization angle characteristic of the incident light, and is composed of pixel values ​​of the pixels that detected the incident light having the same characteristic values ​​of at least the polarization angle characteristic. a development processing unit that develops the polarization angle characteristic reconstructed image; An image processing device comprising: (12) The optical characteristics further include a transmission wavelength characteristic, which is a wavelength characteristic of the incident light. The image processing device according to (11). (13) A reconstruction unit is further provided which reconstructs the polarization angle characteristic reconstructed image, which is obtained by decoding the encoded data by the decoding unit and is reconstructed based on optical characteristics including the polarization angle characteristic and the transmission wavelength characteristic, based on the polarization angle characteristic, The development processing unit develops the polarization angle characteristic reconstructed image reconstructed by the reconstruction unit. (12) An image processing device according to (12). (14) The polarization angle characteristic reconstructed image may further include an image processing unit that performs processing to reduce noise in a developed image generated by the development processing unit after the development processing. An image processing device according to any one of (11) to (13). (15) The polarization angle characteristic reconstructed image is further processed by the developing unit to generate a developed image, and the developed image is enlarged or reduced. An image processing device according to any one of (11) to (14). (16) The optical characteristics further include a transmission wavelength characteristic, which is a wavelength characteristic of the incident light, The polarization angle characteristic reconstructed image is an image obtained by reconstructing the RAW image based on the transmission wavelength characteristics and the pixel positions, and then reconstructing the image based on the polarization angle characteristics. An image processing device according to any one of (11) to (15). (17) The polarization angle characteristic reconstruction image may further include a wavelength extension processing unit that performs wavelength extension processing on the developed image generated by the development processing unit. (16) An image processing device according to (16). (18) The polarization angle characteristic reconstructed image is an image obtained by reconstructing the RAW image based on pixel positions within a pixel block, which is configured by a predetermined number of pixels having a plurality of types of light receiving sensitivity, and further reconstructing the image based on the polarization angle characteristic. An image processing device according to any one of (11) to (17). (19) The polarization angle characteristic reconstructed image is further processed by the developing unit to generate a developed image. (18) An image processing device according to (18). (20) By decoding the encoded data, a RAW image, which is an output of a sensor having a pixel array composed of a plurality of types of pixels that detect incident light having different optical characteristics including at least the polarization angle characteristics, is reconstructed by the optical characteristics including at least the polarization angle characteristics of the incident light, and a polarization angle characteristic reconstructed image is generated that is composed of pixel values ​​of the pixels that detected the incident light having the same characteristic values ​​of at least the polarization angle characteristics, The generated polarization angle characteristic reconstructed image is developed. Image processing methods.

[0313] (31) A reconstruction unit that reconstructs a RAW image, which is an output of a sensor having a pixel array configured with a plurality of types of pixels that detect incident light having mutually different wavelength characteristics, by using a transmission wavelength characteristic that is the wavelength characteristic of the incident light and the positions of the pixels, to generate a transmission wavelength characteristic grid point reconstruction image configured by pixel values ​​of the pixels that detect the incident light having mutually identical characteristic values ​​of the transmission wavelength characteristic and that are located at the same positions in the row direction or column direction in the pixel array; an encoding unit that encodes the transmission wavelength characteristic grid point reconstruction image; An image processing device comprising: (32) The encoding unit encodes the plurality of transmission wavelength characteristic grid point reconstruction images in an order based on the characteristic values ​​of the transmission wavelength characteristics. (31) An image processing device according to (31). (33) The encoding unit encodes the transmission wavelength characteristic grid point reconstruction image using prediction utilizing intra-image correlation or inter-image correlation. The image processing device according to (31) or (32). (34) The reconstruction unit reconstructs the RAW image by intra-block positions, which are pixel positions within a pixel block configured by a predetermined number of the pixels having a plurality of types of light receiving sensitivity, to generate an intra-block position reconstructed image configured by pixel values ​​of the pixels having the same intra-block positions, and generates the transmission wavelength characteristic lattice point reconstructed image by reconstructing the intra-block reconstructed image by the transmission wavelength characteristic and the pixel positions. An image processing device according to any one of (31) to (33). (35) The encoding unit encodes the plurality of transmission wavelength characteristic grid point reconstruction images in an order based on the light receiving sensitivity. (34) An image processing device according to (34). (36) A RAW image, which is an output of a sensor having a pixel array composed of a plurality of types of pixels that detect incident light having mutually different wavelength characteristics, is reconstructed by using a transmission wavelength characteristic, which is the wavelength characteristic of the incident light, and the positions of the pixels, to generate a transmission wavelength characteristic grid point reconstruction image composed of pixel values ​​of the pixels that detect the incident light having mutually identical characteristic values ​​of the transmission wavelength characteristic and are located at the same positions in the row direction or column direction in the pixel array; The generated transmission wavelength characteristic grid point reconstruction image is encoded. Image processing methods.

[0314] (41) A decoding unit that generates a transmission wavelength characteristic lattice point reconstruction image by decoding encoded data, in which a RAW image, which is an output of a sensor having a pixel array composed of multiple types of pixels that detect incident light having different wavelength characteristics, is reconstructed using transmission wavelength characteristics, which are the wavelength characteristics of the incident light, and positions of the pixels, and which detects the incident light having the same characteristic values ​​of the transmission wavelength characteristics and is located at the same positions in the row direction or column direction in the pixel array; a development processing unit that develops the transmission wavelength characteristic grid point reconstruction image; An image processing device comprising: (42) The present invention further includes an image processing unit that performs image processing on a developed image generated by the development processing unit after the transmission wavelength characteristic grid point reconstruction image is developed. (41) An image processing device according to (41). (43) The image processing unit performs a process for reducing noise on the developed image. (42) An image processing device according to (42). (44) The image processing unit enlarges or reduces the developed image. The image processing device according to (42) or (43). (45) The present invention further includes a wavelength extension processing unit that performs wavelength extension processing on a developed image generated by developing the transmission wavelength characteristic grid point reconstruction image by the development processing unit. An image processing device according to any one of (41) to (44). (46) The transmission wavelength characteristic grid point reconstruction image is an image obtained by reconstructing the RAW image based on pixel positions within a pixel block, which is composed of a predetermined number of pixels having a plurality of types of light receiving sensitivity, and further reconstructing the image based on the transmission wavelength characteristic and the pixel positions. An image processing device according to any one of (41) to (45). (47) The present invention further includes a focusing processing unit that performs focusing processing based on a developed image generated by developing the transmission wavelength characteristic grid point reconstruction image by the development processing unit. (46) An image processing device according to (46). (48) By decoding the encoded data, a RAW image, which is the output of a sensor having a pixel array composed of multiple types of pixels that detect incident light having different wavelength characteristics, is reconstructed using the transmission wavelength characteristics that are the wavelength characteristics of the incident light and the positions of the pixels, and a transmission wavelength characteristic grid point reconstruction image is generated that is composed of pixel values ​​of the pixels that detect the incident light having the same characteristic values ​​of the transmission wavelength characteristics and that are located at the same positions in the row direction or column direction in the pixel array, The generated transmission wavelength characteristic grid point reconstruction image is developed. Image processing methods.

[0315] (51) A reconstruction unit that reconstructs a RAW image, which is an output of a sensor having a pixel array configured with a plurality of types of pixels having different light receiving sensitivities, by using intra-block positions, which are pixel positions within a pixel block configured with a predetermined number of the pixels including the plurality of types of pixels, to generate an intra-block position reconstructed image configured with pixel values ​​of the pixels whose intra-block positions are the same; an encoding unit that encodes the intra-block position reconstructed image; An image processing device comprising: (52) The encoding unit encodes the plurality of intra-block position reconstructed images in an order based on the light receiving sensitivities. (51) An image processing device according to (51). (53) The encoding unit encodes the intra-block position reconstructed image using prediction utilizing intra-image correlation or inter-image correlation. An image processing device according to (51) or (52). (54) A RAW image, which is an output of a sensor having a pixel array composed of a plurality of types of pixels having different light receiving sensitivities, is reconstructed by intra-block positions, which are pixel positions within a pixel block composed of a predetermined number of the pixels including the plurality of types of pixels, thereby generating an intra-block position reconstructed image composed of pixel values ​​of the pixels whose intra-block positions are the same; The generated intra-block position reconstruction image is encoded. Image processing methods.

[0316] (61) A decoding unit that generates an intra-block position reconstructed image by decoding encoded data, the intra-block position reconstructed image being composed of pixel values ​​of the pixels whose intra-block positions are identical to each other, in which a RAW image, which is an output of a sensor having a pixel array composed of a plurality of types of pixels having different light receiving sensitivities, is reconstructed by intra-block positions, which are pixel positions within a pixel block composed of a predetermined number of the pixels including the plurality of types of pixels; a development processing unit that develops the intra-block position reconstructed image; An image processing device comprising: (62) The present invention further includes an image processing unit that performs image processing on a developed image generated by the development processing unit after the block intra-position reconstruction image is developed. (61) An image processing device according to (61). (63) The image processing unit performs a process for reducing noise on the developed image. (62) An image processing device according to (62). (64) The image processing unit enlarges or reduces the developed image. An image processing device according to (62) or (63). (65) The present invention further includes a focusing processing unit that performs focusing processing based on a developed image generated by developing the reconstructed image within the block by the development processing unit. An image processing device according to any one of (61) to (64). (66) By decoding the encoded data, a RAW image, which is an output of a sensor having a pixel array composed of a plurality of types of pixels having different light receiving sensitivities, is reconstructed by intra-block positions, which are pixel positions within a pixel block composed of a predetermined number of the pixels including the plurality of types of pixels, to generate an intra-block position reconstructed image composed of pixel values ​​of the pixels whose intra-block positions are the same; The generated intra-block position reconstruction image is developed. Image processing methods. [Explanation of symbols]

[0317] 300 polarization sensor unit, 301 detection unit, 302 image processing unit, 311 polarization sensor, 312 reconstruction unit, 313 reconstruction image encoding unit, 314 transmission unit, 321 transmission wavelength characteristics reconstruction unit, 322 polarization angle characteristics reconstruction unit, 331 encoding unit, 351 reception unit, 352 reconstruction image decoding unit, 353 polarization angle characteristics reconstruction unit, 354 development processing unit, 355 image processing unit, 361 decoding unit, 500 multispectral image sensor unit, 501 detection unit, 502 image processing unit, 511 multispectral image sensor, 512 reconstruction unit, 513 reconstruction image encoding unit, 514 transmission unit, 531 encoding unit, 551 reception unit, 552 reconstruction image decoding unit, 553 development processing unit, 554 Image processing unit, 555 wavelength extension processing unit, 561 decoding unit, 700 image plane phase difference AF image sensor unit, 701 detection unit, 702 image processing unit, 711 image sensor, 712 reconstruction unit, 713 reconstruction image encoding unit, 714 transmission unit, 731 encoding unit, 751 reception unit, 752 reconstruction image decoding unit, 753 development processing unit, 754 image processing unit, 755 focusing processing unit, 761 decoding unit, 900 computer

Claims

1. a reconstruction unit that reconstructs a RAW image, which is an output of a sensor having a pixel array constituted by a plurality of types of pixels that detect incident light having optical characteristics that differ from one another, including at least polarization angle characteristics, by intra-block positions, which are pixel positions within a pixel block constituted by a predetermined number of the pixels having a plurality of types of light receiving sensitivity, to generate an intra-block position reconstructed image constituted by pixel values ​​of the pixels having the same intra-block positions, and reconstructs the intra-block position reconstructed image by the polarization angle characteristics to generate a polarization angle characteristic reconstructed image constituted by pixel values ​​of the pixels that detected the incident light having at least the same characteristic values ​​of the polarization angle characteristics; an encoding unit that encodes the polarization angle characteristic reconstructed image; An image processing device comprising:

2. The optical characteristics further include a transmission wavelength characteristic, which is a wavelength characteristic of the incident light. The image processing device according to claim 1 .

3. The encoding unit encodes the plurality of polarization angle characteristic reconstructed images in an order based on the characteristic values ​​of the transmission wavelength characteristics. The image processing device according to claim 2 .

4. The encoding unit encodes the polarization angle characteristic reconstructed image using prediction utilizing intra-image correlation or inter-image correlation. The image processing device according to claim 2 .

5. The encoding unit encodes the plurality of polarization angle characteristic reconstructed images in an order based on the light receiving sensitivity. The image processing device according to claim 1 .

6. a RAW image, which is the output of a sensor having a pixel array made up of a plurality of types of pixels that detect incident light having optical characteristics that differ from one another, including at least polarization angle characteristics, is reconstructed by intra-block positions, which are pixel positions within a pixel block made up of a predetermined number of the pixels having a plurality of types of light receiving sensitivity, to generate an intra-block position reconstructed image made up of pixel values ​​of the pixels having the same intra-block positions; and a polarization angle characteristic reconstructed image made up of pixel values ​​of the pixels that detected the incident light having at least the same characteristic values ​​of the polarization angle characteristics by reconstructing the intra-block position reconstructed image by the polarization angle characteristics; The generated polarization angle characteristic reconstruction image is encoded. Image processing methods.

7. a decoding unit that decodes the encoded data to generate a polarization angle characteristic reconstructed image in which a RAW image, which is an output of a sensor having a pixel array composed of a plurality of types of pixels that detect incident light having different optical characteristics including at least polarization angle characteristics, is reconstructed by intra-block positions that are pixel positions within a pixel block composed of a predetermined number of pixels having a plurality of types of light receiving sensitivity, and further reconstructed by the polarization angle characteristics, and the polarization angle characteristic reconstructed image is composed of pixel values ​​of the pixels that detected the incident light having the same characteristic values ​​as each other at least in the polarization angle characteristics; a development processing unit that develops the polarization angle characteristic reconstructed image; An image processing device comprising:

8. The optical characteristics further include a transmission wavelength characteristic, which is a wavelength characteristic of the incident light. The image processing device according to claim 7 .

9. a reconstruction unit that reconstructs the polarization angle characteristic reconstructed image obtained by decoding the encoded data by the decoding unit, using the polarization angle characteristic; The development processing unit develops the polarization angle characteristic reconstructed image reconstructed by the reconstruction unit. The image processing device according to claim 7 .

10. The polarization angle characteristic reconstruction image may further include an image processing unit that performs processing to reduce noise in a developed image generated by the development processing unit after the development processing. The image processing device according to claim 7 .

11. an image processing unit that enlarges or reduces a developed image generated by the development processing unit after the polarization angle characteristic reconstructed image is developed; The image processing device according to claim 7 .

12. a wavelength extension processing unit that performs wavelength extension processing on a developed image generated by developing the polarization angle characteristic reconstructed image by the development processing unit; The image processing device according to claim 7 .

13. a focusing processing unit that performs focusing processing based on a developed image generated by developing the polarization angle characteristic reconstructed image by the development processing unit; The image processing device according to claim 7 .

14. By decoding the encoded data, a RAW image, which is the output of a sensor having a pixel array composed of a plurality of types of pixels that detect incident light having different optical characteristics including at least polarization angle characteristics, is reconstructed by intra-block positions, which are pixel positions within a pixel block composed of a predetermined number of pixels having a plurality of types of light receiving sensitivity, and a polarization angle characteristic reconstructed image is generated, which is composed of pixel values ​​of the pixels that detected the incident light having at least the same characteristic values ​​of the polarization angle characteristics, reconstructed by the polarization angle characteristics; The generated polarization angle characteristic reconstructed image is developed. Image processing methods.

Images