Image processing method and image processing system

Abstract

This image processing method includes: detecting a plurality of optical signals corresponding to a plurality of pixels via an optical element having, for each of a plurality of light components, a plurality of light transmittances corresponding to the plurality of pixels (S101); acquiring a first compressed image signal obtained by summing the plurality of optical signals for each adjacent pixel group of two or more adjacent pixels from among the plurality of pixels (S102); and generating, from the first compressed image signal and according to first transmittance information related to the plurality of light transmittances, a first restored image signal group including a plurality of restored image signals corresponding to the plurality of light components (S103).

Description

Image processing method and image processing system

[0001] The present disclosure relates to an image processing method and the like.

[0002] Patent Literature 1 describes a compressed sensing method that performs binning. Patent Literature 2 describes calculating image data for multiple wavelength bands using a compressed sensing technique. Non-Patent Literature 1 describes compressed sensing across the spatial, spectral, and polarization domains. Non-Patent Literature 2 describes a multispectral polarization filter array. Non-Patent Literature 3 describes a multispectral polarization camera. Non-Patent Literatures 4 and 5 describe technologies related to hyperspectral imaging.

[0003] JP 2015-056903 A JP 2016-156801 A

[0004] Chen Fu et al. , “Compressive spectral polarization imaging by a pixelized polarizer and colored patterned detector”, Journal of the Optical Society of America A, 32, 11, 2178-2188 (2015) Kazuma Shinoda et al. , “Snapshot multispectral polarization imaging using a photonic crystal filter array”, Optics Express, 26, 12, 15948-15961 (2018) Kazuma Shinoda et al. , “Alignment-free filter array: Snapshot multispectral polarization imaging based on a Voronoi-like random photonic "crystal filter", Optics Express, 28, 26, 38867-38882 (2020) "World's first technology to capture hyperspectral images and videos by combining metalens and AI with a conventional digital camera - Transforming an 'ordinary camera' into a 'camera that can see the properties of objects' by combining optical technology and AI," [online], October 24, 2022, Nippon Telegraph and Telephone Corporation, [Retrieved November 16, 2023], Internet <URL: https: / / group.ntt / jp / newsrelease / 2022 / 10 / 24 / 221024a.html> Ahasan Ahamed et al., "Reconstruction-based spectroscopy using CMOS image sensors with random photon-trapping" “nanostructure per sensor”, Proc. SPIE 11971, High-Speed ​​Biomedical Imaging and Spectroscopy VII, 1197106, 2 March 2022

[0005] However, the sensitivity of the sensor may be low, and image processing to effectively increase the sensitivity of the sensor may cause parallax in the image.

[0006] Therefore, an image processing method and the like are provided that can effectively increase the sensitivity of the sensor while suppressing the parallax that occurs in the image.

[0007] An image processing method according to one aspect of the present disclosure includes detecting a plurality of optical signals corresponding to a plurality of pixels via an optical element having a plurality of optical transmittances corresponding to the plurality of pixels, obtaining a first compressed image signal obtained by summing the plurality of optical signals for each group of two or more adjacent pixels in the plurality of pixels, and generating a first group of restored image signals from the first compressed image signal in accordance with first transmittance information regarding the plurality of optical transmittances.

[0008] These comprehensive or specific aspects may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0009] An image processing method according to one aspect of the present disclosure makes it possible to increase the effective sensitivity of a sensor while suppressing parallax that occurs in an image.

[0010] 10 is a conceptual diagram showing binning processing in a first reference example. FIG. 11 is a conceptual diagram showing binning processing in a second reference example. FIG. 12 is a conceptual diagram showing binning processing in a third reference example. FIG. 13 is a conceptual diagram showing binning processing in a fourth reference example. FIG. 14 is a conceptual diagram showing binning processing in a fifth reference example. FIG. 15 is a conceptual diagram showing binning processing in a sixth reference example. FIG. 16 is a block diagram showing an example of a configuration of an image processing system in an embodiment. FIG. 17 is a conceptual diagram showing an example of a configuration of an imaging device in an embodiment. FIG. 18 is a conceptual diagram showing an example of a configuration of a filter array in an embodiment. FIG. 19 is a graph showing an example of a transmission spectrum of a filter in an embodiment. FIG. 19 is a graph showing an example of a transmission spectrum of another filter in an embodiment. FIG. 20 is a graph showing an example of a transmittance of a first wavelength band of a filter array in an embodiment. FIG. 21 is a graph showing an example of a transmittance of a second wavelength band of a filter array in an embodiment. FIG. 22 is a flowchart showing an example of an operation of an image processing system in an embodiment. FIG. 23 is a block diagram showing a first specific example of a configuration of an image processing system in an embodiment. FIG. 24 is a block diagram showing a first specific example of an operation of an image processing system in an embodiment. FIG. 25 is a block diagram showing a second specific example of an operation of an image processing system in an embodiment. FIG. 26 is an explanatory diagram showing binning processing of a compressed image signal in an embodiment. FIG. 27 is an explanatory diagram showing binning processing of mask data in an embodiment. FIG. 28 is a conceptual diagram showing binning processing in an embodiment. FIG. 1 is an image diagram showing a restored image in an embodiment. FIG. 2 is a block diagram showing a second specific example of the configuration of an image processing system in an embodiment. FIG. 3 is a block diagram showing a third specific example of the operation of an image processing system in an embodiment. FIG. 4 is a block diagram showing a fourth specific example of the operation of an image processing system in an embodiment. FIG. 5 is a conceptual diagram showing a method of generating a restored image signal corresponding to a polarization direction in an embodiment. FIG. 6 is a diagram showing a plurality of pixels arranged in a matrix included in an image sensor and pixel values ​​output by each of the plurality of pixels. Image I1 １ , ...Image I1 ｗ The circuit is shown in FIG. １ Image I2 １ , and convert it into image I1ｗ Image I2 ｗ FIG. 10 is a diagram showing that the above can be converted into

[0011] 1 is a conceptual diagram showing binning processing in a first reference example. For example, a camera detects component values ​​of an optical signal for each pixel. However, it is difficult to detect multiple component values ​​for each pixel. Therefore, for example, the camera is configured to detect multiple component values ​​for each pixel group made up of multiple pixels.

[0012] Specifically, as shown in the example on the left side of FIG. 1 , the camera detects the values ​​of four components a, b, c, and d for each pixel group consisting of four pixels. An image is represented by these values. In order to increase the effective sensitivity of the sensor, a binning process may be performed. In the binning process, multiple pixels are combined. That is, the values ​​of multiple pixels are added together. This increases the value, thereby increasing the effective sensitivity of the sensor.

[0013] In the example of FIG. 1 , multiple pixels corresponding to the same type of component are integrated. Specifically, four pixels are integrated with a one-pixel gap between them. Because there is a gap between the integrated pixels, the integration may result in parallax. Here, parallax refers to differences in the imaging area that arise when the spatial positions of each pixel are not identical, and this may result in visual differences when the image is output.

[0014] For example, there is a possibility that an edge will be introduced at an interval of one pixel. When four pixels are combined with this interval, the edge information may be lost. Furthermore, there is a possibility that a pixel different from any of the four combined pixels will be generated due to the one-pixel interval. This may result in false colors. Therefore, there is a possibility that visual differences will occur due to the spatial separation of the combined pixels, i.e., due to parallax.

[0015] FIG. 2 is a conceptual diagram illustrating the binning process in the second reference example. The example in FIG. 2 differs from the example in FIG. 1 in that adjacent pixels are combined without leaving an interval of one pixel. In this case, the values ​​of the four components a, b, c, and d are combined. As a result, the values ​​of each component are lost. It is difficult to derive an image similar to the image composed of the values ​​before combination from such combined values.

[0016] For example, if a pixel in row x and column y of a plurality of pixels arranged in m rows and n columns is expressed as (x, y), the pixels adjacent to (x, y) are expressed as (x-1, y-1), (x, y-1), (x+1, y-1), (x-1, y), (x+1, y), (x-1, y+1), (x, y+1), or (x+1, y+1), where x is a natural number equal to or less than m, and y is a natural number equal to or less than n.

[0017] 3 is a conceptual diagram showing the binning process in the third reference example. In this example, an RGB camera detects values ​​of three wavelength bands corresponding to R (red), G (green), and B (blue) for each pixel group consisting of four pixels. Specifically, of the four pixels, one pixel corresponds to R (red), two pixels correspond to G (green), and one pixel corresponds to B (blue).

[0018] In this case, too, in the binning process, multiple pixels are combined without mixing R (red), G (green), and B (blue), so four pixels are combined with a one-pixel interval between them, which results in parallax, as in the example of FIG.

[0019] 4 is a conceptual diagram illustrating binning processing in a fourth reference example. The example in FIG. 4 differs from the example in FIG. 3 in that adjacent pixels are combined without leaving an interval of one pixel. In this case, the values ​​of R (red), G (green), and B (blue) are combined. As a result, the values ​​of R (red), G (green), and B (blue) are lost. It is difficult to derive an image similar to the image composed of the values ​​before combination from such combined values.

[0020] That is, in a monochrome camera, it is possible to increase sensitivity while suppressing parallax by binning adjacent pixels, but in an RGB camera, it is not easy to increase sensitivity while suppressing parallax by binning.

[0021] 5 is a conceptual diagram showing the binning process in the fifth reference example. In this example, the polarization camera detects four polarization direction values ​​for each pixel group consisting of four pixels. Specifically, of the four pixels, one pixel corresponds to a polarization direction of 0 degrees, one pixel corresponds to a polarization direction of 45 degrees, one pixel corresponds to a polarization direction of 90 degrees, and one pixel corresponds to a polarization direction of 135 degrees.

[0022] In this case, too, in the binning process, a plurality of pixels are combined without mixing the 0-degree polarization direction, the 45-degree polarization direction, the 90-degree polarization direction, and the 135-degree polarization direction, so four pixels are combined with an interval of one pixel. Therefore, parallax occurs, as in the example of FIG.

[0023] 6 is a conceptual diagram illustrating the binning process in the sixth reference example. The example of FIG. 6 differs from the example of FIG. 5 in that adjacent pixels are integrated without leaving an interval of one pixel. In this case, the values ​​of the 0-degree polarization direction, the 45-degree polarization direction, the 90-degree polarization direction, and the 135-degree polarization direction are integrated. As a result, the values ​​of the 0-degree polarization direction, the 45-degree polarization direction, the 90-degree polarization direction, and the 135-degree polarization direction are lost. It is difficult to derive an image similar to the image composed of the values ​​before integration from such integrated values.

[0024] In any case, it is not easy to increase sensitivity while suppressing parallax by binning processing in a camera that detects multiple component values.

[0025] Therefore, the image processing method of Example 1 includes detecting a plurality of optical signals corresponding to a plurality of pixels via an optical element having a plurality of optical transmittances for each of a plurality of optical components corresponding to the plurality of pixels, obtaining a first compressed image signal obtained by summing the plurality of optical signals for each adjacent pixel group of two or more adjacent pixels in the plurality of pixels, and generating a first restored image signal group having a plurality of restored image signals corresponding to the plurality of optical components from the first compressed image signal in accordance with first transmittance information regarding the plurality of optical transmittances.

[0026] This makes it possible to generate restored image signals for each component from the first compressed image signals summed for each group of adjacent pixels in accordance with the first transmittance information relating to the multiple light transmittances of each component. That is, it becomes possible to sum adjacent pixels and separate restored image signals for each component. This makes it possible to increase the effective sensitivity of the sensor while suppressing parallax occurring in the image.

[0027] The image processing method of Example 2 may be the image processing method of Example 1, in which the plurality of light components are a plurality of wavelength bands.

[0028] This makes it possible to separate the restored image signals of each wavelength band with high accuracy, that is, it is possible to increase sensitivity and maintain wavelength resolution while suppressing parallax.

[0029] Furthermore, the image processing method of Example 3 may be the image processing method of Example 1 or 2, wherein the first transmittance information is obtained by integrating, for each of the plurality of light components, the plurality of light transmittances of second transmittance information having as information the plurality of light transmittances corresponding to the plurality of pixels, for each group of adjacent pixels of the two or more adjacent pixels.

[0030] This makes it possible to accurately generate restored image signals for each component from a first compressed image signal in which multiple optical signals are summed for each adjacent pixel group, according to first transmittance information in which multiple optical transmittances are integrated for each adjacent pixel group.

[0031] Furthermore, the image processing method of Example 4 may be any of the image processing methods of Examples 1 to 3, in which an optical signal corresponding to a first pixel of the two or more adjacent pixels is an optical signal that has passed through a first region of the optical element, an optical signal corresponding to a second pixel of the two or more adjacent pixels is an optical signal that has passed through a second region of the optical element, and the optical transmission spectrum of the first region for the multiple light components is different from the optical transmission spectrum of the second region for the multiple light components.

[0032] This makes it possible to accurately generate restored image signals of each component from the first compressed image signal in accordance with a plurality of optical transmission spectra that differ depending on the pixel.

[0033] Furthermore, the image processing method of Example 5 may be any of the image processing methods of Examples 1 to 4, wherein the optical element has a plurality of light transmission spectra corresponding to the plurality of pixels, each of the plurality of light transmission spectra being a light transmission spectrum for the plurality of light components, and the plurality of light transmission spectra having spatial randomness with respect to the plurality of pixels.

[0034] This makes it possible to accurately generate restored image signals of each component from the first compressed image signal in accordance with a plurality of spatially random optical transmission spectra.

[0035] In addition, the image processing method of Example 6 may be any of the image processing methods of Examples 1 to 5, further including acquiring a second compressed image signal having the plurality of optical signals, wherein the first compressed image signal is acquired by adding up the plurality of optical signals of the second compressed image signal for each group of adjacent pixels of the two or more adjacent pixels.

[0036] This makes it possible to obtain the first compressed image signal by summing the optical signals of the second compressed image signal for each group of adjacent pixels, thereby enabling flexible control of the summation process.

[0037] Furthermore, the image processing method of Example 7 may be any of the image processing methods of Examples 1 to 6, further including switching between a first mode in which the first group of restored image signals is generated from the first compressed image signal and a second mode in which a second group of restored image signals is generated from a second compressed image signal having the plurality of optical signals.

[0038] This makes it possible to adaptively switch between a first mode in which restoration is performed from a first compressed image signal in which adjacent pixel groups are integrated, and a second mode in which restoration is performed from a second compressed image signal in which adjacent pixel groups are not integrated.

[0039] Furthermore, the image processing method of Example 8 may be the image processing method of Example 7, in which the switching between the first mode and the second mode is performed based on an operation performed by a user.

[0040] This makes it possible to switch between a first mode in which restoration is performed from a first compressed image signal in which adjacent pixel groups are integrated, and a second mode in which restoration is performed from a second compressed image signal in which adjacent pixel groups are not integrated, in accordance with the user's intentions.

[0041] Furthermore, the image processing method of Example 9 may be the image processing method of Example 7 or 8, in which the number of pixel signals in each restored image signal of the first restored image signal group is smaller than the number of pixel signals in each restored image signal of the second restored image signal group.

[0042] This makes it possible to reduce the amount of calculation in the first mode, in which restoration is performed from a first compressed image signal in which adjacent pixel groups are integrated, compared to the second mode, in which restoration is performed from a second compressed image signal in which adjacent pixel groups are not integrated.

[0043] Furthermore, the image processing method of Example 10 may be any of the image processing methods of Examples 1 to 9, in which the number of pixel signals in each restored image signal of the first restored image signal group is smaller than the number of the plurality of pixels.

[0044] This makes it possible to suppress the amount of calculation and efficiently generate restored image signals of each component from the first compressed image signal.

[0045] Furthermore, the image processing method of Example 11 may be any of the image processing methods of Examples 1 to 10, in which the number of pixel signals in each restored image signal of the first restored image signal group is the same as the number of pixel signals in the first compressed image signal.

[0046] This makes it possible to generate a restored image signal having the same resolution as the first compressed image signal, i.e., it is possible to generate a restored image signal while maintaining the resolution.

[0047] Furthermore, the image processing method of Example 12 may be any of the image processing methods of Examples 1 to 11, in which the first transmittance information has, as information, a plurality of light transmittances for each of the plurality of light components that are smaller than the plurality of light transmittances corresponding to the plurality of pixels.

[0048] This makes it possible to suppress the amount of calculation and efficiently generate restored image signals of each component from the first compressed image signal.

[0049] The image processing method of Example 13 may be the image processing method of Example 1, in which the plurality of light components are a plurality of polarization directions.

[0050] This makes it possible to separate the restored image signals for each polarization direction with high accuracy, that is, it is possible to increase sensitivity and maintain the angular resolution of polarization while suppressing parallax.

[0051] Furthermore, the image processing method of Example 14 may be any of the image processing methods of Examples 1 to 13, in which the first compressed image signal is obtained by adding up the plurality of optical signals for each group of adjacent pixels of the two or more adjacent pixels and performing an averaging process.

[0052] This makes it possible to improve the signal-to-noise ratio, suppress parallax occurring in the image, and increase the effective sensitivity of the image sensor.

[0053] Moreover, the image processing method of Example 15 includes detecting a plurality of optical signals corresponding to a plurality of pixels via an optical element having a plurality of optical transmission spectra corresponding to the plurality of pixels, obtaining a first compressed image signal obtained by summing the plurality of optical signals for each adjacent pixel group of two or more adjacent pixels in the plurality of pixels, and generating a first restored image signal group having a plurality of restored image signals corresponding to a plurality of wavelength bands from the first compressed image signal according to information on the plurality of optical transmission spectra.

[0054] This makes it possible to generate restored image signals for each wavelength band from the first compressed image signals summed for each group of adjacent pixels in accordance with information about the multiple optical transmission spectra. That is, it is possible to sum adjacent pixels and separate restored image signals for each wavelength band. This makes it possible to increase the effective sensitivity of the sensor while suppressing parallax that occurs in the image.

[0055] Moreover, the image processing system of Example 16 includes an image sensor that detects a plurality of optical signals corresponding to a plurality of pixels via an optical element having a plurality of optical transmittances for each of a plurality of light components corresponding to the plurality of pixels, and a processing circuit that acquires a first compressed image signal obtained by summing the plurality of optical signals for each group of adjacent pixels of two or more adjacent pixels in the plurality of pixels, and the processing circuit further generates a first restored image signal group having a plurality of restored image signals corresponding to the plurality of light components from the first compressed image signal in accordance with first transmittance information regarding the plurality of optical transmittances.

[0056] This makes it possible to generate restored image signals for each component from the first compressed image signals summed for each group of adjacent pixels in accordance with the first transmittance information relating to the multiple light transmittances of each component. That is, it becomes possible to sum adjacent pixels and separate restored image signals for each component. This makes it possible to increase the effective sensitivity of the sensor while suppressing parallax occurring in the image.

[0057] Moreover, the image processing system of Example 17 includes an image sensor that detects a plurality of optical signals corresponding to a plurality of pixels via an optical element having a plurality of optical transmission spectra corresponding to the plurality of pixels, and a processing circuit that acquires a first compressed image signal obtained by summing the plurality of optical signals for each group of adjacent pixels of two or more adjacent pixels in the plurality of pixels, and the processing circuit further generates a first restored image signal group having a plurality of restored image signals corresponding to a plurality of wavelength bands from the first compressed image signal according to information regarding the plurality of optical transmission spectra.

[0058] This makes it possible to generate restored image signals for each wavelength band from the first compressed image signals summed for each group of adjacent pixels in accordance with information about the multiple optical transmission spectra. That is, it is possible to sum adjacent pixels and separate restored image signals for each wavelength band. This makes it possible to increase the effective sensitivity of the sensor while suppressing parallax that occurs in the image.

[0059] Furthermore, these comprehensive or specific aspects may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0060] Hereinafter, embodiments will be described with reference to the drawings. Note that the embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, the arrangement and connection of the components, steps, the order of steps, and the like shown in the following embodiments are merely examples and are not intended to limit the scope of the claims.

[0061] In the present disclosure, a pixel is defined as an area of ​​an image. Specifically, a pixel is defined as, for example, the smallest unit of an area of ​​an image. Detecting a light signal at a pixel means detecting a light signal corresponding to the pixel, and means detecting the light signal at a light receiving element corresponding to the pixel. Furthermore, an image signal is a signal for representing an image, and an image can be represented by the image signal. Furthermore, the image signal can be composed of multiple pixel signals. The pixel signal may correspond to, for example, a light signal detected for a pixel.

[0062] Fig. 7 is a block diagram showing an example of the configuration of an image processing system according to an embodiment. As shown in Fig. 7, the image processing system 100 includes an image sensor 111 and a processing circuit 121. The image processing system 100 may further include a display device 130. The image processing system 100 may also include an imaging device 110 and an image processing device 120. The imaging device 110 may then include the image sensor 111. The image processing device 120 may then include the processing circuit 121.

[0063] The image sensor 111 detects an optical signal for each pixel and generates an image signal composed of a plurality of optical signals corresponding to the plurality of pixels, thereby acquiring an image signal. Here, the image sensor 111 acquires a compressed image signal. Specifically, for example, the image sensor 111 acquires a compressed image signal in which information of four or more wavelength bands is superimposed for each pixel based on a filter array described below. Note that a wavelength band may be simply referred to as a band.

[0064] The image sensor 111 may be a monochrome photodetector having a plurality of photodetection elements arranged in a matrix. More specifically, the image sensor 111 may be a charge-coupled device (CCD) image sensor, a complementary metal oxide semiconductor (CMOS) image sensor, an infrared array image sensor, a terahertz array image sensor, or a millimeter wave array image sensor.

[0065] Alternatively, the image sensor 111 may be a color-type photodetector. The wavelength range detectable by the image sensor 111 is not limited, and may be, for example, visible light, ultraviolet light, infrared light, terahertz waves, or any combination thereof.

[0066] The processing circuitry 121 is a circuit that performs information processing. For example, the processing circuitry 121 performs a restoration operation on the compressed image signal to generate a plurality of restored image signals corresponding to a plurality of wavelength bands. Here, the plurality of restored images are a plurality of spectral images, and the plurality of restored image signals are a plurality of spectral image signals.

[0067] The multiple restored images represented by the multiple restored image signals may constitute a hyperspectral image represented by information of four or more wavelength bands. For example, the hyperspectral image is composed of four or more spectral images corresponding to four or more wavelength bands. The restoration calculation may be the same as the restoration calculation described in Patent Document 2. Specifically, four or more spectral images may be generated as the hyperspectral image based on the following equation (1):

[0068]

[0069] Here, g is data representing a compressed image and is represented, for example, by a one-dimensional array (i.e., a vector). If the compressed image is an image of n×m pixels, data g is represented by a one-dimensional array having n×m elements. f is data representing w spectral images corresponding one-to-one to w wavelength bands and is represented, for example, by a one-dimensional array. When w spectral images constitute a hyperspectral image, w is an integer greater than or equal to 4.

[0070] f １ is the wavelength band W １ Spectral image data corresponding to f ２ is the wavelength band W ２ Spectral image data corresponding to f ｗ is the wavelength band W ｗ The spectral image data corresponds to the

[0071] Data f １ , f ２ , ..., fｗ Each of the spectral images is represented by, for example, a one-dimensional array. If each spectral image is an image of n×m pixels, the data f １ , f ２ , ..., f ｗ are represented by a one-dimensional array having n×m elements, and data f is represented by a one-dimensional array having n×m×w elements. H is a matrix with n×m rows and n×m×w columns, called the system matrix, and corresponds to the mask data.

[0072] The matrix H is the wavelength band W of the filter array described below. １ Transmission spectrum of wavelength band W ２ Transmission spectrum of wavelength band W ｗ The transmittance may be determined based on the transmittance spectrum of the light emitting element.

[0073] The data f that satisfies equation (1) can be estimated using a compressed sensing technique, specifically, by equation (2).

[0074]

[0075] Equation (2) expresses finding data f that minimizes the sum of the first and second terms in the parentheses. Data f can be calculated as final calculation result data by converging the calculation result data through recursive iterative calculation.

[0076] The first term in the parentheses in equation (2) represents the sum of squares of the difference between data g and data Hf obtained by transforming data f in the estimation process using matrix H, and is a so-called residual term. Although the sum of squares is used here, the sum of absolute values, the square root of the sum of squares, or the like may be used instead of the sum of squares.

[0077] The sum of squares of the difference between data Hf and data g is (g １ -r １ ) × (g １ -r １ ) + ... + (g ｎ×ｍ -r ｎ×ｍ ) × (g ｎ×ｍ -r ｎ×ｍ ) where g １ , ..., g ｎ×ｍ is g = (g １ ...gｎ×ｍ ) Ｔ is an element of data g expressed as １ , ..., r ｎ×ｍ is Hf = (r １ ...r ｎ×ｍ ) Ｔ The elements of data Hf are expressed as follows:

[0078] The second term in the parentheses in Equation (2) is a regularization term, sometimes called a stabilization term. Φ(f) represents a constraint on the regularization of f and is a function that reflects the sparse information of the data f. This function has the effect of smoothing or stabilizing the data f. Φ(f) can be expressed, for example, by a discrete cosine transform (DCT), a wavelet transform, a Fourier transform, a total variation (TV), or any combination thereof.

[0079] τ is a weighting coefficient for the regularization term, and corresponds to the influence of regularization in the reconstruction calculation. The larger the value of τ, the greater the influence of regularization, and the stronger the convergence of the solution in the iterative calculation. Conversely, the smaller the value of τ, the less the influence of regularization, and the weaker the convergence of the solution in the iterative calculation.

[0080] The display device 130 is a display device for displaying information. A compressed image, a monochrome image, an RGB image, a hyperspectral image, or the like is displayed on the display device 130 by the processing circuit 121. The display device 130 may be, for example, a liquid crystal display (LCD) or an organic light-emitting diode (OLED) display.

[0081] A graphical user interface (GUI) may be displayed on the display device 130. The display device 130 may also be a touch panel. A user may input information to the display device 130. The processing circuitry 121 may acquire information from the user via the display device 130. That is, the display device 130 may be an input / output device. Alternatively, the processing circuitry 121 may acquire information from the user via an input device different from the display device 130.

[0082] 7 shows an example of the configuration of the image processing system 100, but the configuration of the image processing system 100 is not limited to the example of the configuration in FIG. 7. For example, multiple devices may be integrated into one device, or one device may be distributed among multiple devices. Furthermore, multiple devices that are distributed may be able to communicate with each other via wired or wireless communication.

[0083] The processing circuit 121 corresponds to an acquirer that acquires compressed image signals from the image sensor 111, a generator that generates a group of restored image signals, and a display controller that displays the group of restored image signals as a group of restored images on the display device 130. The image processing device 120, rather than the processing circuit 121, may include some or all of these components.

[0084] Fig. 8 is a conceptual diagram showing an example of the configuration of the imaging device 110 shown in Fig. 7. The imaging device 110 may have a configuration similar to that of the imaging device disclosed in Patent Document 2. For example, the imaging device 110 includes an image sensor 111, a filter array 112, and an optical system 113.

[0085] The filter array 112 is disposed on the optical path of light incident from the subject, and is disposed between the optical system 113 and the image sensor 111. The filter array 112 functions as the encoding element described in Patent Document 2. The filter array 112 may be integrated with the image sensor 111.

[0086] The arrangement of the filter array 112 is not limited to the arrangement shown in Fig. 8. For example, the filter array 112 may be arranged between the optical system 113 and the image sensor 111, but away from the image sensor 111. Alternatively, for example, the filter array 112 may be arranged between the subject and the optical system 113. Alternatively, for example, the filter array 112 may be arranged within the optical system 113.

[0087] The optical system 113 is disposed on the optical path of light incident from the subject, and is disposed between the subject and the filter array 112. The optical system 113 includes at least one lens, and can form an image of the subject on the imaging surface of the image sensor 111 via the filter array 112.

[0088] The configuration and arrangement of the optical system 113 are not limited to those shown in Fig. 8. For example, the optical system 113 may be arranged between the filter array 112 and the image sensor 111. Furthermore, for example, the optical system 113 may include a plurality of lenses arranged on the optical path. In this case, the filter array 112 may be arranged between adjacent lenses of the plurality of lenses.

[0089] The filter array 112 and the optical system 113 are examples of optical elements. The image sensor 111 may detect the optical signal through an optical element different from the filter array 112 and the optical system 113. The optical element may be an object expressed as an optical member, an optical mechanism, an optical device, or the like.

[0090] 9 is a conceptual diagram showing an example of the configuration of the filter array 112 shown in FIG. 8. The filter array 112 is made up of a plurality of filters F １１ , ..., F ｎｍ In the example shown in FIG. 9, the filter array 112 includes 48 filters F １１ , ..., F ｎｍ The filter F １１ is 48 filters F １１ , ..., F ｎｍ The filter F is located at the top left of the ｎｍ is 48 filters F １１ , ..., F ｎｍ This is the filter located at the bottom right of the list.

[0091] The filter F included in the filter array 112 １１ , ..., F ｎｍ The number of filters F included in the filter array 112 is not limited to 48. １１ , ..., F ｎｍ The number may be approximately the same as the number of pixels of the image sensor 111, and may be determined depending on the application within the range of, for example, several tens to several tens of millions.

[0092] For example, the filter array 112 may include n×m filters F corresponding to n×m pixels. １１, ..., F ｎｍ Waveband W １ , ..., W ｗ In this case, the filter F １１ , ..., F ｎｍ are the transmission spectrum S １１ , ..., S ｎｍ The transmission spectrum S １１ , ..., S ｎｍ Alternatively, the transmission spectrum S １１ , ..., S ｎｍ may be the same. Here, the transmission spectrum may refer to the light transmittance spectrum.

[0093] n×m filters F １１ , ..., F ｎｍ is w wavelength bands W １ , ..., W ｗ For n×m×w transmittances S １１Ｗ１ , ..., S ｎｍＷｗ The transmittance S １１Ｗ１ , ..., S ｎｍＷｗ Alternatively, the transmittance S １１Ｗ１ , ..., S ｎｍＷｗ Here, transmittance may refer to light transmittance.

[0094] Wavelength band W α The transmittance of the filter β in the above formula may be expressed by the following formula (3):

[0095]

[0096] where Wαmin is the wavelength band W α is the minimum wavelength value of the wavelength band W α is the maximum wavelength value of h(λ), and h(λ) is a function indicating the transmission spectrum, where λ is the wavelength.

[0097] In addition, the wavelength band W α The transmittance of the filter β in the wavelength band W is not limited to the formula (3). αThe transmittance of the filter β in the wavelength band W may be the transmittance obtained by dividing the formula (3) by (Wαmax-Wαmin). α The transmittance of the filter β in the wavelength band W α The wavelength λ that represents α０ Transmittance h (λ α０ ) may also be used.

[0098] Here, the wavelength λ α０ is Wαmin≦λ α０ For example, the wavelength λ α０ is the wavelength band W α The center wavelength may be ((Wαmax−Wαmin) / 2).

[0099] FIG. 10 shows the filter F shown in FIG. １１ 11 is an example of the transmission spectrum of the filter F shown in FIG. ｎｍ 12 is an example of the transmission spectrum of the wavelength band W of the filter array 112 shown in FIG. １ 13 is a diagram showing an example of the transmittance of the wavelength band W of the filter array 112 shown in FIG. ２ 12 and 13, the shading of each region represents the transmittance of the filter, with lighter regions representing higher transmittance and darker regions representing lower transmittance.

[0100] A plurality of filters F included in the filter array 112 １１ , ..., F ｎｍ The wavelength dependence of the transmittance of each filter is different. １１ In the wavelength band W １ The transmittance of the wavelength band W ２ On the other hand, the transmittance of the filter F ｎｍ In the wavelength band W １ The transmittance of the wavelength band W ２ The transmittance of the filter F is approximately the same as that of the filter F. １１ The wavelength dependence of the transmittance of the filter F ｎｍ The wavelength dependence of the transmittance is different from that of the

[0101] Here, w wavelength bands W １ , ..., W ｗ Two of the wavelength bands W １ and W ２ The transmittance of the w wavelength bands W １ , ..., W ｗ The transmittance of other wavelength bands will not be shown or explained.

[0102] For example, the mask data may include w wavelength bands W １ , W ２ , ..., W ｗ Specifically, the mask data is expressed as w matrices corresponding to w wavelength bands W １ , W ２ , ..., W ｗ is expressed by a matrix of transmittances (transmittance matrix) with n rows and m columns corresponding to pixels with n rows and m columns. Then, by changing the representation format from the mask data expressed by w matrices each having n rows and m columns, a single matrix H with n×m rows and n×m×w columns is obtained.

[0103] Also, here, for example, the image sensor 111 acquires the compressed image through a plurality of light-receiving regions having a plurality of transmission spectra. Specifically, the image sensor 111 acquires the compressed image through the filter array 112. The plurality of light-receiving regions may correspond to a plurality of filters included in the filter array 112, respectively.

[0104] Alternatively, the metalens described in Non-Patent Document 4 may be used instead of the filter array 112. In this case, the wavelength transmittance varies depending on the location. The light receiving region corresponds to such a location. Alternatively, the CMOS image sensor described in Non-Patent Document 5 may be used instead of the filter array 112. In this case, the sensing region is processed so that a predetermined transmittance is obtained for each pixel. The light receiving region corresponds to such a sensing region.

[0105] The light receiving region may be expressed as an optical element, a modulation element, an encoding element, an optical processing region, a masking region, etc. Furthermore, for example, the optical element, the modulation element, or the encoding element may correspond to a plurality of light receiving regions.

[0106] Fig. 14 is a flowchart showing an example of the operation of the image processing system 100 shown in Fig. 7. In this example, the image sensor 111 of the imaging device 110 detects a plurality of optical signals corresponding to a plurality of pixels via an optical element (S101). Here, the optical element has a plurality of optical transmittances for a plurality of light components corresponding to the plurality of pixels.

[0107] The processing circuit 121 of the image processing device 120 then acquires a first compressed image signal (S102). Here, in the first compressed image signal, a plurality of optical signals detected by the image sensor 111 are summed for each adjacent pixel group. Each adjacent pixel group is made up of two or more adjacent pixels among the plurality of pixels.

[0108] Then, the processing circuit 121 generates a first restored image signal group having a plurality of restored image signals corresponding to a plurality of light components from the first compressed image signal in accordance with first transmittance information regarding a plurality of light transmittances corresponding to a plurality of pixels (S103).

[0109] This makes it possible to generate restored image signals for each component from the first compressed image signals summed for each group of adjacent pixels in accordance with the first transmittance information relating to the multiple light transmittances of each component. That is, it becomes possible to sum adjacent pixels and separate restored image signals for each component. This makes it possible to increase the effective sensitivity of the sensor while suppressing parallax occurring in the image.

[0110] Here, having a plurality of light transmittances for a plurality of light components corresponding to a plurality of pixels means having a plurality of light transmittances that differ depending on the light components. In other words, the optical element has, for example, a plurality of light transmittances for a first light component corresponding to a plurality of pixels, and further has another plurality of light transmittances for a second light component corresponding to the same plurality of pixels. If the plurality of light transmittances corresponding to a plurality of pixels are expressed as a light transmittance group, the optical element has a plurality of light transmittance groups corresponding to a plurality of light components.

[0111] For example, the multiple light components may be multiple wavelength bands, which allows for high-precision separation of restored image signals for each wavelength band. That is, it is possible to increase sensitivity and maintain wavelength resolution while suppressing parallax.

[0112] Furthermore, for example, the first transmittance information may be obtained by integrating, for each of a plurality of light components, a plurality of light transmittances of second transmittance information having, as information, a plurality of light transmittances corresponding to a plurality of pixels for each of two or more adjacent pixel groups. This makes it possible to accurately generate a restored image signal for each component from a first compressed image signal in which a plurality of light signals are summed for each adjacent pixel group, in accordance with the first transmittance information in which the plurality of light transmittances are integrated for each adjacent pixel group.

[0113] For example, an optical signal corresponding to a first pixel among two or more adjacent pixels may be an optical signal that has passed through a first region of an optical element, and an optical signal corresponding to a second pixel among two or more adjacent pixels may be an optical signal that has passed through a second region of an optical element, and the optical transmission spectrum of the first region for the multiple light components may be different from the optical transmission spectrum of the second region for the multiple light components.

[0114] This makes it possible to accurately generate restored image signals of each component from the first compressed image signal in accordance with a plurality of optical transmission spectra that differ depending on the pixel.

[0115] Furthermore, for example, the optical element may have a plurality of optical transmission spectra corresponding to a plurality of pixels. Here, each of the plurality of optical transmission spectra is an optical transmission spectrum for a plurality of optical components. The plurality of optical transmission spectra may have spatial randomness for the plurality of pixels. This makes it possible to accurately generate restored image signals for each component from the first compressed image signal in accordance with the plurality of spatially random optical transmission spectra.

[0116] Here, the randomness is not limited to complete randomness. For example, a plurality of light transmission spectra having spatial randomness for a plurality of pixels may contain locally identical patterns. Alternatively, the randomness may be pseudo-randomness.

[0117] For example, the optical transmission spectra for the pixels of an RGB filter for generating an RGB image signal have periodic characteristics, whereas the optical transmission spectra for the pixels of the optical element may have non-periodic characteristics.

[0118] Furthermore, for example, the processing circuitry 121 may acquire a second compressed image signal having a plurality of optical signals. The first compressed image signal may be acquired by summing the plurality of optical signals of the second compressed image signal for each adjacent pixel group of two or more adjacent pixels. This makes it possible to acquire the first compressed image signal using an operation that sums the plurality of optical signals of the second compressed image signal for each adjacent pixel group. This allows for flexible control of the summation process.

[0119] Furthermore, for example, the processing circuitry 121 may switch between a first mode in which a first group of restored image signals is generated from a first compressed image signal and a second mode in which a second group of restored image signals is generated from a second compressed image signal having a plurality of optical signals, thereby enabling adaptive switching between the first mode in which restoration is performed from a first compressed image signal in which adjacent pixel groups are integrated, and the second mode in which restoration is performed from a second compressed image signal in which adjacent pixel groups are not integrated.

[0120] Furthermore, for example, the processing circuitry 121 may switch between the first mode and the second mode based on an operation performed by a user, thereby enabling switching, in accordance with the user's intention, between the first mode in which restoration is performed from a first compressed image signal in which adjacent pixel groups are integrated, and the second mode in which restoration is performed from a second compressed image signal in which adjacent pixel groups are not integrated.

[0121] Furthermore, for example, the number of pixel signals in each restored image signal of the first restored image signal group may be smaller than the number of pixel signals in each restored image signal of the second restored image signal group, thereby making it possible to reduce the amount of calculation in the first mode in which restoration is performed from the first compressed image signal in which adjacent pixel groups are integrated, compared to the second mode in which restoration is performed from the second compressed image signal in which adjacent pixel groups are not integrated.

[0122] Furthermore, for example, the number of pixel signals in each restored image signal in the first restored image signal group may be less than the number of pixels, thereby reducing the amount of calculation and enabling the restored image signals of each component to be efficiently generated from the first compressed image signal.

[0123] Furthermore, for example, the number of pixel signals in each restored image signal of the first restored image signal group may be the same as the number of pixel signals in the first compressed image signal. This makes it possible to generate restored image signals having the same resolution as the resolution of the first compressed image signal. In other words, it is possible to generate restored image signals while maintaining the resolution.

[0124] Furthermore, for example, the first transmittance information may include, for each of the plurality of light components, a plurality of light transmittances that are smaller than the plurality of light transmittances corresponding to the plurality of pixels, thereby reducing the amount of calculation and enabling efficient generation of restored image signals for each component from the first compressed image signal.

[0125] Furthermore, for example, the multiple light components may have multiple polarization directions. This allows for highly accurate separation of restored image signals for each polarization direction. That is, it is possible to increase sensitivity and maintain angular resolution of polarization while suppressing parallax.

[0126] Furthermore, for example, the first compressed image signal may be obtained by summing and averaging a plurality of optical signals for each group of two or more adjacent pixels, thereby improving the signal-to-noise ratio and increasing the effective sensitivity of the image sensor while suppressing parallax that occurs in the image.

[0127] Here, the SN ratio represents the ratio between the target signal and noise, and the higher the SN ratio, the less noise there is and the better the observation results. For example, the signal may be the true pixel value of each pixel in the acquired image, and the noise may be the deviation from the true pixel value. Although it is difficult to directly observe the true pixel value, for example, by continuously capturing multiple images of a stationary subject and calculating the time average of the pixel value of each pixel, it is possible to calculate the true pixel value and the deviation from the true value in each image.

[0128] Furthermore, for each light component, the plurality of light transmittances may or may not correspond one-to-one to the plurality of pixels. For example, two or more light transmittances may correspond to one pixel, or two or more pixels may correspond to one light transmittance.

[0129] Furthermore, for example, the plurality of light transmittances may be non-uniform. Such non-uniform light transmittances are effective for generating a restored image signal. Here, the plurality of light transmittances may be considered non-uniform when they include at least two light transmittances that are different from each other. Furthermore, for example, the plurality of light transmittances may have randomness.

[0130] Furthermore, for example, the transmittance information may correspond to mask data. That is, the first transmittance information may correspond to the first mask data. Furthermore, the second transmittance information may correspond to the second mask data. Alternatively, the transmittance information may be expressed as light transmittance information or coded information. That is, the first transmittance information may be expressed as first light transmittance information or first coded information, and the second transmittance information may be expressed as second light transmittance information or second coded information.

[0131] In another example, the image sensor 111 of the imaging device 110 may detect a plurality of optical signals corresponding to the plurality of pixels via an optical element having a plurality of optical transmission spectra corresponding to the plurality of pixels. Then, the processing circuit 121 of the image processing device 120 may acquire a first compressed image signal. Here, in the first compressed image signal, the plurality of optical signals detected by the image sensor 111 are summed for each adjacent pixel group. Each adjacent pixel group is composed of two or more adjacent pixels among the plurality of pixels.

[0132] Then, the processing circuit 121 of the image processing device 120 may generate a first restored image signal group having a plurality of restored image signals corresponding to a plurality of wavelength bands from the first compressed image signal according to information regarding a plurality of optical transmission spectra.

[0133] This makes it possible to generate restored image signals for each wavelength band from the first compressed image signals summed for each group of adjacent pixels in accordance with information about the multiple optical transmission spectra. That is, it is possible to sum adjacent pixels and separate restored image signals for each wavelength band. This makes it possible to increase the effective sensitivity of the sensor while suppressing parallax that occurs in the image.

[0134] Fig. 15 is a block diagram showing a first specific example of the configuration of the image processing system 100 shown in Fig. 7. In the example of Fig. 15, the image processing system 100 includes an image sensor 111, an image processing device 120, and a display device 130. The image processing device 120 includes processing circuits 121a and 121b, and a memory 122.

[0135] The image sensor 111, image processing device 120, and display device 130 in Fig. 15 correspond to the image sensor 111, image processing device 120, and display device 130 in Fig. 7. Furthermore, the processing circuits 121a and 121b in Fig. 15 correspond to the processing circuit 121 in Fig. 7. The processing circuits 121a and 121b may be integrated into one processing circuit.

[0136] The memory 122 is a storage element for storing information. The memory 122 may be a volatile memory or a non-volatile memory. In this example, the memory 122 stores mask data before binning, that is, mask data that has not been subjected to binning.

[0137] For example, the image sensor 111 detects a plurality of optical signals corresponding to a plurality of pixels via optical elements corresponding to the optical system 113 and the filter array 112 shown in Fig. 8. The plurality of optical signals detected by the image sensor 111 constitute a compressed image signal in which information of a plurality of wavelength bands is superimposed. In this example, the compressed image signal constituted by the plurality of optical signals detected by the image sensor 111 is a compressed image signal before binning processing, that is, a compressed image signal that has not been subjected to binning processing.

[0138] The processing circuit 121a acquires a compressed image signal before binning processing from the image sensor 111. That is, the compressed image signal before binning processing is transmitted from the image sensor 111 to the processing circuit 121a. The processing circuit 121a also acquires mask data before binning processing from the memory 122. That is, the mask data before binning processing is transmitted from the memory 122 to the processing circuit 121a.

[0139] The processing circuit 121a then performs binning on the compressed image signal and the mask data, i.e., the processing circuit 121a integrates the compressed image signal for each group of adjacent pixels, and integrates the mask data for each group of adjacent pixels.

[0140] For example, if the number of pixels in the compressed image signal before binning is A (A is a natural number), the number of pixels in the compressed image signal after binning may be C (C is a natural number smaller than A). If the number of pixels in the mask data before binning is B (B is a natural number), the number of pixels in the mask data after binning, i.e., the mask data on which binning has been performed, may be C (C is a natural number smaller than B).

[0141] Here, the number of pixels in the image signal is the number of pixels in the image represented by the image signal and corresponds to the number of pixel signals constituting the image signal. That is, the number of pixels in the compressed image signal is the number of pixels in the compressed image represented by the compressed image signal and corresponds to the number of pixel signals constituting the compressed image signal. Also, the number of pixels in the mask data corresponds to the number of transmittances that the mask data has for multiple pixels for each wavelength band.

[0142] The number of pixels B of the mask data before binning may be different from or the same as the number of pixels A of the compressed image signal before binning. Then, binning may be performed so that the number of pixels C of the mask data after binning is the same as the number of pixels C of the compressed image signal after binning, i.e., the compressed image signal on which binning has been performed.

[0143] The binning process may be performed on a portion of the compressed image. Alternatively, the binning process may be performed on a portion of the mask data. For example, the compressed image may have 4K resolution (3840 x 1920) and the mask data may have Full-HD resolution (1920 x 1080). Then, a 2 x 2 pixel binning process may be performed on a region near the center. Then, the binning process may generate a compressed image of 640 x 360 pixels and mask data of the same size, 640 x 360 pixels.

[0144] The processing circuit 121b generates a plurality of restored image signals corresponding to a plurality of wavelength bands from the binned compressed image signal in accordance with the binned mask data.

[0145] FIG. 16 is a block diagram showing a first specific example of the operation of the image processing system 100 shown in FIG.

[0146] In this example, the processing circuitry 121a acquires mask data before binning (S201). The processing circuitry 121a also acquires compressed image signals before binning (S202). These processes may be performed in the reverse order or in parallel.

[0147] Then, the processing circuitry 121a determines whether or not to perform binning processing (S203). For example, a mode in which binning processing is performed and a mode in which binning processing is not performed may be switched by an operation performed by a user. Specifically, the processing circuitry 121a may obtain the mode selected by the user via an input device (not shown) or the like. Then, the processing circuitry 121a may determine whether or not to perform binning processing according to the selected mode.

[0148] The processing circuit 121a may also determine whether to perform binning processing based on the brightness of the photographed subject. Specifically, the processing circuit 121a may determine whether to perform binning processing based on the brightness of the subject indicated by the compressed image signal, that is, the luminance value of the pixel. For example, if the photographed subject is dark, the processing circuit 121a determines to perform binning processing.

[0149] More specifically, the processing circuitry 121a may determine to perform binning when the average luminance value of the entire compressed image represented by the compressed image signal is equal to or less than a threshold value. Alternatively, the processing circuitry 121a may determine to perform binning when the average luminance values ​​of a plurality of pixels included in the top P% (P is a constant) in order of brightness among a plurality of pixels of the compressed image represented by the compressed image signal are equal to or less than a threshold value.

[0150] The processing circuitry 121a may also determine whether to perform binning processing based on the number of pixels. For example, when the number of pixels in the compressed image signal, i.e., the number of pixel signals included in the compressed image signal, is equal to or greater than a threshold, the processing circuitry 121a may determine to perform binning processing in order to suppress processing delay.

[0151] If the processing circuit 121a determines that binning processing should be performed (Yes in S203), it performs binning processing on the mask data and the compressed image signal (S204). In this case, the processing circuit 121a performs binning processing so that the number of pixels in the mask data after binning processing matches the number of pixels in the compressed image signal after binning processing. If the processing circuit 121a determines that binning processing should not be performed (No in S203), it skips the binning processing.

[0152] The processing circuitry 121b then restores the image signal (S205). For example, the processing circuitry 121b generates a plurality of restored image signals corresponding to a plurality of wavelength bands. At this time, the image signal may be restored based on the mask data and compressed image signal after the binning process, or may be restored based on the mask data and compressed image signal before the binning process.

[0153] FIG. 17 is a block diagram showing a second specific example of the operation of the image processing system 100 shown in FIG.

[0154] In this example, the processing circuitry 121a determines whether or not to perform binning processing (S301). For example, a user may perform an operation to switch between a mode in which binning processing is performed and a mode in which binning processing is not performed. Specifically, the processing circuitry 121a may obtain a mode selected by the user via an input device (not shown) or the like. Then, the processing circuitry 121a may determine whether or not to perform binning processing according to the selected mode.

[0155] When the processing circuitry 121a determines that binning processing is to be performed (Yes in S301), it acquires mask data (S302) and performs binning processing on the mask data (S303).The processing circuitry 121a also acquires a compressed image signal (S304) and performs binning processing on the compressed image signal (S305).

[0156] When the processing circuitry 121a determines that binning processing is not to be performed (No in S301), it acquires mask data (S306) and acquires a compressed image signal (S307). In this case, binning processing is not performed.

[0157] The processing circuitry 121b then restores the image signal (S308). For example, the processing circuitry 121b generates a plurality of restored image signals corresponding to a plurality of wavelength bands. At this time, the image signal may be restored based on the mask data and compressed image signal after the binning process, or may be restored based on the mask data and compressed image signal before the binning process.

[0158] 18 is an explanatory diagram showing the binning process of a compressed image signal according to an embodiment of the present invention, in which compressed images are shown according to various implementation levels of the binning process.

[0159] The top row of Fig. 18 shows a compressed image when no binning is performed, the middle row of Fig. 18 shows a compressed image when 2x2 pixels are combined by binning, and the bottom row of Fig. 18 shows a compressed image when 4x4 pixels are combined by binning.

[0160] The range of adjacent pixels that are integrated increases in the order of no binning, 2x2 pixel binning, and 4x4 pixel binning, and accordingly the resolution of the compressed image decreases in the order of 256x256 pixels, 128x128 pixels, and 64x64 pixels, resulting in a blurred compressed image.

[0161] FIG. 19 is an explanatory diagram showing the binning process of mask data in an embodiment. The binning process is also performed on the mask data. For example, without binning, the four transmittances corresponding to the four upper left pixels are 0.63, 0.37, 0.75, and 0.52. By binning the 2×2 pixels, these four transmittances are combined into a single transmittance. For example, these four transmittances are combined into a single transmittance of 0.57, which is the average of these four transmittances. This combined transmittance can then be used in the restoration process.

[0162] In the result of the 2x2 pixel binning process, the four transmittances corresponding to the four upper left pixels are 0.57, 0.57, 0.47, and 0.56. By the 4x4 pixel binning process, these four transmittances are integrated into a single transmittance. For example, these four transmittances are integrated into a single transmittance of 0.54, which is the average of these four transmittances. This integrated transmittance can then be used in the restoration process.

[0163] 20 is a conceptual diagram illustrating binning processing in an embodiment. For example, when compressed sensing is used, multiple component values ​​of an optical signal are combined according to multiple transmittances for the multiple components and detected as a combined component value for each pixel. These combined component values ​​can then be separated into multiple component values ​​using mask data indicating multiple transmittances for the multiple components.

[0164] 20, even if multiple composite component values ​​corresponding to multiple pixels are integrated for each adjacent pixel group, the integrated composite component value can be separated into multiple component values ​​using mask data integrated for each adjacent pixel group. For example, it is possible to generate a restored image signal for each wavelength band having the same number of pixels as the number of pixels in the compressed image signal after binning processing. Therefore, it is possible to integrate adjacent pixel groups. Therefore, it is possible to perform binning processing while suppressing parallax.

[0165] Fig. 21 is a conceptual diagram showing a restored image in an embodiment. The left side of Fig. 21 shows a hyperspectral image corresponding to the restored image when binning processing is not performed. The right side of Fig. 21 shows a hyperspectral image corresponding to the restored image when binning processing is performed. Specifically, in both cases, a composite image of 20 spectral images constituting a hyperspectral image having 20 wavelength bands is shown as the hyperspectral image corresponding to the restored image.

[0166] In this example, the 2x2 pixel binning process adds up four optical signals corresponding to four pixels, resulting in a four times brighter image. The resolution is also halved in both the vertical and horizontal directions, meaning the number of pixels is reduced to one-quarter.

[0167] Here, multiple optical signals are added together in the binning process. An averaging process may be further performed in the binning process. That is, the signal value obtained by adding together the multiple optical signals may be divided by the number of the multiple optical signals to output an average of the multiple optical signals. This also improves the signal-to-noise ratio and the effective sensitivity of the sensor.

[0168] Fig. 22 is a block diagram showing a second specific example of the configuration of the image processing system 100 shown in Fig. 7. In the example of Fig. 22, compared to the example of Fig. 15, the image processing device 120 includes a processing circuit 121c instead of the processing circuits 121a and 121b. In addition, the memory 122 stores mask data after binning processing.

[0169] For example, the image sensor 111 detects a plurality of optical signals corresponding to a plurality of pixels via optical elements corresponding to the optical system 113 and the filter array 112 shown in FIG. 8. The plurality of optical signals detected by the image sensor 111 constitute a compressed image signal in which information of a plurality of wavelength bands is superimposed. In this example, the image sensor 111 also performs binning processing on the compressed image signal. That is, the image sensor 111 adds up a plurality of optical signals corresponding to a plurality of pixels for each group of adjacent pixels.

[0170] The processing circuit 121c acquires the compressed image signal after binning processing from the image sensor 111. That is, the compressed image signal after binning processing is transmitted from the image sensor 111 to the processing circuit 121c. The processing circuit 121c also acquires mask data after binning processing from the memory 122. That is, the mask data after binning processing is transmitted from the memory 122 to the processing circuit 121c.

[0171] For example, the number of pixels of the mask data after binning is C, and the number of pixels of the compressed image signal after binning is C. That is, the processing circuit 121c acquires mask data and compressed image signals corresponding to the same number of pixels.

[0172] The processing circuit 121c generates a plurality of restored image signals corresponding to a plurality of wavelength bands from the binned compressed image signal in accordance with the binned mask data.

[0173] FIG. 23 is a block diagram showing a specific example of the operation of the image processing system 100 shown in FIG.

[0174] In this example, the processing circuit 121c acquires mask data after binning (S401). The processing circuit 121c also acquires compressed image signals after binning (S402). These processes may be performed in reverse order or in parallel.

[0175] Then, the processing circuitry 121c restores the image signal based on the mask data after the binning process and the compressed image signal (S403). For example, the processing circuitry 121c generates a plurality of restored image signals corresponding to a plurality of wavelength bands.

[0176] Fig. 24 is a block diagram showing a third specific example of the configuration of the image processing system 100 shown in Fig. 7. In the example of Fig. 24, compared to the example of Fig. 15, the image processing device 120 includes processing circuits 121d and 121e instead of the processing circuits 121a and 121b. The processing circuits 121d and 121e may be integrated into a single processing circuit. In addition, the memory 122 stores mask data before binning processing and mask data after binning processing.

[0177] For example, the image sensor 111 detects a plurality of optical signals corresponding to a plurality of pixels via optical elements corresponding to the optical system 113 and the filter array 112 shown in Fig. 8. The plurality of optical signals detected by the image sensor 111 constitute a compressed image signal in which information of a plurality of wavelength bands is superimposed. In this example, the compressed image signal constituted by the plurality of optical signals detected by the image sensor 111 is a compressed image signal before binning processing, that is, a compressed image signal that has not been subjected to binning processing.

[0178] The processing circuit 121d acquires the compressed image signal before binning processing from the image sensor 111. That is, the compressed image signal before binning processing is transmitted from the image sensor 111 to the processing circuit 121d. The processing circuit 121d also acquires mask data before binning processing from the memory 122. That is, the mask data before binning processing is transmitted from the memory 122 to the processing circuit 121d.

[0179] The processing circuit 121d then performs binning processing on the compressed image signal. That is, the processing circuit 121d integrates the compressed image signal for each group of adjacent pixels. The processing circuit 121d also acquires mask data after the binning processing from the memory 122. That is, the mask data after the binning processing is transmitted from the memory 122 to the processing circuit 121d.

[0180] For example, the number of pixels of the mask data after binning is C, and the number of pixels of the compressed image signal after binning is C. That is, the processing circuit 121d acquires mask data and compressed image signals corresponding to the same number of pixels.

[0181] The processing circuit 121e generates a plurality of restored image signals corresponding to a plurality of wavelength bands from the binned compressed image signal in accordance with the binned mask data.

[0182] Alternatively, the processing circuit 121d may not perform binning on the compressed image signal. In this case, the processing circuit 121d acquires mask data before binning from the memory 122. That is, the mask data before binning is transmitted from the memory 122 to the processing circuit 121d. In this case, the number of pixels B of the mask data before binning is the same as the number of pixels A of the compressed image signal before binning.

[0183] The processing circuitry 121e may then generate a plurality of restored image signals corresponding to a plurality of wavelength bands from the compressed image signal before binning in accordance with the mask data before binning.

[0184] FIG. 25 is a block diagram showing a specific example of the operation of the image processing system 100 shown in FIG.

[0185] In this example, the processing circuit 121d acquires a compressed image signal before binning processing (S501), and then determines whether or not to perform binning processing (S502), similar to the determination processing (S203) in FIG.

[0186] If the processing circuit 121d determines that binning processing is to be performed (Yes in S502), it performs binning processing on the compressed image signal (S503). The processing circuit 121d acquires mask data after binning processing (S504). If the processing circuit 121d determines that binning processing is not to be performed (No in S502), it acquires mask data before binning processing (S506).

[0187] The processing circuitry 121e then restores the image signal (S507). For example, the processing circuitry 121e generates a plurality of restored image signals corresponding to a plurality of wavelength bands. At this time, the image signal may be restored based on the mask data and compressed image signal after the binning process, or may be restored based on the mask data and compressed image signal before the binning process.

[0188] This makes it possible to switch whether or not to perform binning while suppressing the processing load associated with the binning of mask data. Multiple sets of mask data may be stored in the memory 122 based on multiple implementation levels of the binning. Then, one set of mask data may be selected and acquired from the memory 122 based on the implementation level of the binning.

[0189] The present disclosure is not limited to the restoration of hyperspectral images, but is also effective for the restoration of images expressed in three wavelength bands, such as RGB images, and for the restoration of images expressed in two wavelength bands. Furthermore, in the above-described examples, a compressed image signal in which information on multiple wavelength bands is superimposed is basically used. Alternatively, a compressed image signal in which information on multiple polarization directions is superimposed may be used.

[0190] For example, the n×m filters F １１ , ..., F ｎｍmay have n×m×w transmittances for w polarization directions instead of w wavelength bands. The image sensor 111 may obtain a compressed image signal on which information of the w polarization directions is superimposed by detecting a plurality of optical signals corresponding to a plurality of pixels via the filter array 112. The processing circuitry 121 may generate w restored image signals corresponding to the w polarization directions from the compressed image signal by the restoration calculation described above.

[0191] Examples of application of polarizing filters are shown in Non-Patent Documents 1 to 3. Such polarizing filters may be applied to the present disclosure.

[0192] 26 is a conceptual diagram showing a method for generating a restored image signal corresponding to a polarization direction in an embodiment. The restored image signal is generated using mask data having a plurality of transmittances corresponding to a plurality of pixels for each of a plurality of polarization directions instead of a plurality of wavelength bands.

[0193] Mask data can be obtained by measuring the transmittance for each pixel and each polarization direction. For example, multiple transmittances corresponding to multiple pixels may be measured while shifting the polarization direction at a certain angle from a single reference direction to a specific direction. Specifically, a rotatable polarizing filter may be attached to a camera lens, and a standard white board may be photographed while rotating the polarizing filter, thereby measuring multiple transmittances corresponding to multiple pixels for each polarization direction.

[0194] The signal values ​​measured by photographing a standard white board for each polarization direction may be used directly as mask data corresponding to the transmittance, or the mask data value may be derived by dividing the signal value measured with a polarizing filter by the signal value measured without a polarizing filter.

[0195] The mask data having multiple transmittances corresponding to multiple pixels for each of the multiple polarization directions is reflected in the matrix H and used in the above-described equation (2). This allows multiple restored image signals corresponding to the multiple polarization directions to be calculated from the multiple compressed image signals. For example, multiple restored images obtained from the multiple polarization directions can be used for surface inspection of a highly reflective component.

[0196] 7 may perform binning processing in the restoration of a plurality of restored image signals corresponding to a plurality of polarization directions, similar to the case of a plurality of wavelength bands, thereby enabling the image processing system 100 to increase the effective sensitivity of the sensor while suppressing parallax occurring in the image.

[0197] As described above, the image processing system 100 of the present disclosure can increase the effective sensitivity of the sensor while suppressing the parallax that occurs in the images when restoring a plurality of images corresponding to a plurality of components.

[0198] In this regard, Patent Literature 1 describes a compressed sensing method that performs binning. In this method, instead of reading out the luminance level of each pixel, a compressed image is acquired by measuring a weighted sum of luminance levels for each constituent unit having multiple pixels. That is, a compressed image with a reduced number of pixels is generated by binning. Then, an image corresponding to the original image before binning is restored from the compressed image based on the weighting coefficients and sparsity.

[0199] On the other hand, Patent Document 1 does not take into account multiple components of light. For example, when multiple values ​​of multiple components are detected at multiple adjacent pixels and the multiple values ​​detected at the multiple adjacent pixels are added together, as shown in Figure 2, there is no sparsity between the multiple components, making it difficult to derive the original image from the compressed image. Furthermore, when multiple values ​​detected at multiple non-adjacent pixels are added together, as shown in Figure 1, parallax occurs.

[0200] Therefore, even with the disclosure of Patent Document 1, it is not easy to increase sensitivity while suppressing parallax when restoring multiple images corresponding to multiple components. Furthermore, Patent Document 1 assumes that the number of pixels in the restored image is the same as the number of pixels in the original image before binning, and is greater than the number of pixels in the compressed image. Therefore, it is assumed that the brightness of the restored image does not improve, and the effective sensitivity of the sensor does not improve.

[0201] In contrast, the image processing system 100 of the present disclosure detects a signal obtained by combining multiple components for each pixel. Then, it is possible to separate multiple signals corresponding to the multiple components from the detected signal according to the transmittance of each component. Furthermore, by performing binning processing of multiple adjacent pixels on both the compressed image signal and the mask data, it is possible to increase sensitivity while suppressing parallax and generate multiple restored image signals corresponding to the multiple components.

[0202] The image processing system 100 of the present disclosure may acquire the compressed image signal and mask data after binning, and generate a plurality of restored image signals corresponding to a plurality of components using the compressed image signal and mask data after binning. This configuration also makes it possible to suppress parallax, increase sensitivity, and generate a plurality of restored image signals corresponding to a plurality of components.

[0203] Furthermore, in the image processing system 100 of the present disclosure, the number of pixels in the restored image signal is the same as the number of pixels in the compressed image signal after binning processing, but is smaller than the number of pixels in the compressed image signal before binning processing (in other words, the number of pixels at the time of detecting the optical signal). This makes it possible to reduce the amount of calculation required for the restoration operation, and to generate a restored image with a small data size.

[0204] Although the image processing system and the like have been described according to the embodiments, the aspects of the image processing system and the like are not limited to the embodiments. Modifications conceivable by those skilled in the art may be made to the embodiments, and multiple components in the embodiments may be combined in any manner.

[0205] For example, a process performed by a specific component in an embodiment may be performed by another component instead of the specific component. Furthermore, the order of multiple processes may be changed, or multiple processes may be performed in parallel. Furthermore, ordinal numbers such as "first" and "second" used in the description may be changed, removed, or newly assigned as appropriate. These ordinal numbers do not necessarily correspond to a meaningful order, and may be used to identify elements.

[0206] Also, for example, a phrase "at least one of a first element, a second element, and a third element" corresponds to the first element, the second element, the third element, or any combination thereof.

[0207] Furthermore, a method including steps performed by each component of an image processing system or the like may be executed by any system or device. In other words, the method may be executed by the image processing system or the like described above, or may be executed by another system or device.

[0208] For example, a part or all of the method may be executed by a computer including a processor, a memory, an input / output circuit, etc. In this case, the method may be executed by the computer executing a program for causing the computer to execute the method.

[0209] For example, the above program causes a computer to execute an image processing method including: detecting a plurality of optical signals corresponding to a plurality of pixels via an optical element having a plurality of optical transmittances for each of a plurality of light components corresponding to the plurality of pixels; acquiring a first compressed image signal obtained by adding up the plurality of optical signals for each adjacent pixel group of two or more adjacent pixels in the plurality of pixels; and generating a first restored image signal group having a plurality of restored image signals corresponding to the plurality of light components from the first compressed image signal in accordance with first transmittance information regarding the plurality of optical transmittances.

[0210] Furthermore, for example, the above program may cause a computer to execute an image processing method including: detecting a plurality of optical signals corresponding to a plurality of pixels via an optical element having a plurality of optical transmission spectra corresponding to the plurality of pixels; acquiring a first compressed image signal obtained by adding up the plurality of optical signals for each adjacent pixel group of two or more adjacent pixels in the plurality of pixels; and generating, from the first compressed image signal, a first restored image signal group having a plurality of restored image signals corresponding to a plurality of wavelength bands in accordance with information on the plurality of optical transmission spectra.

[0211] The above program may also be recorded on a non-transitory computer-readable recording medium such as a CD-ROM.

[0212] Furthermore, each component of the image processing system may be configured with dedicated hardware, general-purpose hardware that executes the above-mentioned programs, or a combination of these. The general-purpose hardware may be configured with a memory in which the programs are recorded and a general-purpose processor that reads and executes the programs from the memory. Here, the memory may be a semiconductor memory or a hard disk, and the general-purpose processor may be a CPU.

[0213] Furthermore, the dedicated hardware may be configured with a memory and a dedicated processor, etc. For example, the dedicated processor may refer to the memory and execute the above-described method.

[0214] Furthermore, each component of the image processing system or the like may be an electric circuit. These electric circuits may form a single electric circuit as a whole, or may be separate electric circuits. These electric circuits may correspond to dedicated hardware, or may correspond to general-purpose hardware that executes the above-mentioned programs or the like.

[0215] (Others) Modifications of the embodiment may be as follows.

[0216] An apparatus, the apparatus including a circuit, the circuit receives first information, the circuit generates one or more images based on the first information and second information, the circuit outputs the one or more images, an image sensor including a plurality of pixels is capable of outputting the first information after receiving first light from a filter array including four or more filters having different transmission spectra in a wavelength range, the first information being a plurality of pixel values ​​output by the plurality of pixels, and the circuit generates an image I1 corresponding to a wavelength band W1 included in the wavelength range based on the first information and third information. １, ... an image I1 corresponding to a wavelength band Ww included in the wavelength range ｗ the third information is a matrix H=[H1...Hw] having (nxm) rows and (nxmxw) columns, each of H1, ..., Hw is a submatrix of H, each of H1, ..., Hw is a diagonal matrix having (nxm) rows and (nxm) columns, and the circuit generates an image I2 corresponding to a wavelength band W1 included in the wavelength range based on the first information and the second information. １ , ...Image I2 corresponding to the wavelength band Ww included in the wavelength range ｗ and the image I2 １ , ... the image I2 ｗ includes the one or more images, and the image I1 １ The P pixels included in the image I2 １ , and corresponds to one pixel included in the image I1 ｗ The P pixels included in the image I2 ｗ the second information corresponds to one pixel included in the matrix Hr=[Hr1...Hrw] having ((n×m) / d) rows and ((n×m×w) / d) columns, the matrix Hr1 is a diagonal matrix having ((n×m) / d) rows and ((n×m) / d) columns generated based on the diagonal matrix H1, ..., the matrix Hrw is a diagonal matrix having ((n×m) / d) rows and ((n×m) / d) columns generated based on the diagonal matrix Hw.

[0217] The d may be 2.

[0218] The d may be 4.

[0219] (Description of a Modification of the Above-mentioned Embodiment) The device may be the image processing device 120 .

[0220] The circuitry may be processing circuitry 121 .

[0221] The filter array may be filter array 112 .

[0222] The image sensor may be the image sensor 111 .

[0223] FIG. 27 is a diagram showing a plurality of pixels arranged in a matrix included in an image sensor and pixel values ​​output by each of the plurality of pixels.

[0224] FIG. 27(a) is a diagram showing pixels p11, . . . , pnm arranged in a matrix form included in the image sensor.

[0225] FIG. 27(b) shows pixel p 11 is the pixel value g 11 , ..., pixel p nm is the pixel value g nm FIG.

[0226] The first information is g = (g 11 …g nm ) T is.

[0227] The third information may be a matrix H.

[0228] The matrix H is H=[H1...Hw], and each of H1,...,Hw may be a diagonal matrix containing (nxm) diagonal elements.

[0229]

[0230] may be.

[0231] The circuit is expressed by (Equation 1), that is, g=(g 11 …g nm ) T = [H1…Hw] Based on f1, f1=(f11…f1w) T This generation method was explained in the description of (Equation 1) and (Equation 2). 11 …f11 nm ) T , ..., f1w = (f1w 11 …f1w nm ) T is.

[0232] FIG. 28 shows the image I1 １ , ...Image I1 ｗ FIG.

[0233] Image I1 １ is the pixel value f1１１ Pixel p11 having １１ , ..., pixel value f1 ｎｍ Pixel p11 having ｎｍ Including,..., Image I1 ｗ is the pixel value f1w １１ Pixel p1w having １１ , ..., pixel value f1w ｎｍ Pixel p1w having ｎｍ１ Includes.

[0234] The circuit is image I2 １ , ..., Image I2 ｗ The number of pixels to be included in each of the above may be specified, i.e., pixel count information specifying (n / d) x (m / d), where d may be an integer greater than or equal to 2. The following example illustrates processing with d = 2.

[0235] FIG. 29 shows that the circuit １ Image I2 １ , and convert it into image I1 ｗ Image I2 ｗ FIG. 10 is a diagram showing that the above can be converted into

[0236] The image I １ , ..., the image I1 ｗ Each of the rows contains n×m pixels.

[0237] Image I2 １ , ..., the image I2 ｗ Each of the image planes comprises (n / d)×(m / d) pixels, where d=2. The circuitry may receive information specifying d from a user via an input device.

[0238] In FIG. 29, four pixels are converted into one pixel.

[0239] Image I1 １ Pixel p1 included in １１ , pixel p11 １２ , pixel p11 ２１ , pixel p11 ２２ is image I2 １ Pixel p21 included in １１ , and the image I1 １Pixel p11(n-1)(m-1) and pixel p11 (n-1)m , pixel p11 n(m-1) , pixel p11 nm Image I2 １ is converted to pixel p21(n / 2)(m / 2) included in

[0240] ... Image I1 ｗ Pixel p1w included in １１ , pixel p1w １２ , p1w ２１ , pixel p1w ２２ is image I2 ｗ Pixel p21 included in １１ , and image I1 ｗ Pixel p11(n-1)(m-1) included in (n-1)m , pixel p11 n(m-1) , pixel p11 nm is image I2 ｗ is converted to pixel p21(n / 2)(m / 2) included in

[0241] The second information may be a matrix Hr=[Hr1...Hrw]. The circuit may generate the second information based on the third information. The second information Hr=[Hr1...Hrw] generated from the third information H=[H1...Hw] is shown below as an example. Each of Hr1,...,Hrw is a diagonal matrix.

[0242] The diagonal matrix H1 is １１ Component h1(11) corresponding to pixel p1 １２ Component h1(22), ..., pixel p11 ２１ Component h1((m+1)(m+1)) corresponding to pixel p1 ２２ Component h1((m+2)(m+2)) corresponding to pixel p11(n-1)(m-1), ... Component h1(((n-2)×m+(m-1))((n-2)×m+(m-1))) corresponding to pixel p11 (n-1)m Component h1(((n-2)×m+m)((n-2)×m+m)), ..., corresponding to pixel p11 n(m-1) Component h1(((n-1)×m+(m-1))((n-1)×m+(m-1))) corresponding to pixel p11 nmIt contains a component h1(((n-1)×m+m)((n-1)×m+m))= h1((n×m)(n×m)) corresponding to

[0243] The diagonal matrix Hr1 is １１ a component hr(11)=(h1(11)+h1(22)+h1((m+1)(m+1))+h1((m+2)(m+2))) / 4 corresponding to pixel p21(n / 2)(m / 2), ... a component hr(((n / 2)×(m / 2))((n / 2)×(m / 2))=(h1(((n-2)×m+(m-1))((n-2)×m+(m-1)))+h1(((n-2)×m+m)((n-2)×m+m))+h1(((n-1)×m+(m-1))((n-1)×m+(m-1)))+h1((n×m)(n×m))) / 4 corresponding to pixel p21(n / 2)(m / 2).

[0244] That is, the ((n / d)×(m / d)) diagonal elements contained in the diagonal matrix Hr1 can be expressed using the (n×m) diagonal elements contained in the diagonal matrix H1.

[0245] Similarly, the ((n / d) × (m / d)) diagonal elements contained in diagonal matrix Hr2 can be expressed using the (n × m) diagonal elements contained in diagonal matrix H2, ..., the ((n / d) × (m / d)) diagonal elements contained in diagonal matrix Hrw can be expressed using the (n × m) diagonal elements contained in diagonal matrix Hw.

[0246] The circuit receives the information specifying d and the first information g=(g 11 …g nm ) T Based on this, the fourth information g1 = (g1 11 …g1(n / d)(m / d)) T In the following description, d=2 is used.

[0247] Pixel p included in the image sensor １１ The pixel value g output from １１ , pixel p included in the image sensor １２ The pixel value g output from １２ , pixel p included in the image sensor ２１ The pixel value g output from ２１ , pixel p included in the image sensor２２ The pixel value g output from ２２ Based on the pixel value, pixel value g1 １１ , a pixel value g(n-1)(m-1) output from a pixel p(n-1)(m-1) included in the image sensor, and a pixel value g(n-1)(m-1) output from a pixel p(n-1)(m-1) included in the image sensor are determined. （ｎ－１）ｍ The pixel value g output from （ｎ－１）ｍ , pixel p included in the image sensor ｎ（ｍ－１） The pixel value g output from ｎ（ｍ－１） , pixel p included in the image sensor ｎｍ The pixel value g output from ｎｍ The pixel value g1(n / d)(m / d) is determined based on

[0248] Pixel value g1 １１ = (g １１ +g １２ +g ２１ +g ２２ ) / 4, ..., pixel value g1(n / d)(m / d) = (g(n-1)(m-1) + g （ｎ－１）ｍ +g ｎ（ｍ－１） +g ｎｍ ) / 4 may also be used.

[0249] The circuit is expressed by (Equation 1), that is, g1=(g1 11 …g1 nm ) T = [Hr1...Hrw]f2, f2 = (f21...f2w) T This generation method was explained in the description of (Equation 1) and (Equation 2). 11 …f21(n / d)(m / d)) T , ..., f2w = (f2w 11 …f2w(n / d)(m / d)) T is.

[0250] f21 １１ is pixel p21 １１ f21(n / d)(m / d) is the pixel value of pixel p21(n / d)(m / d), ..., f2w １１ is pixel p2w １１ f2w(n / d)(m / d) is the pixel value of pixel p21w(n / d)(m / d)11.

[0251] The present disclosure is applicable to an image processing method for restoring an image, and can be used in image processing systems, imaging systems, camera systems, analysis systems, recognition systems, and the like.

[0252] 100 Image processing system 110 Imaging device 111 Image sensor 112 Filter array 113 Optical system 120 Image processing device 121, 121a, 121b, 121c, 121d, 121e Processing circuit 122 Memory 130 Display device

Claims

1. An image processing method comprising: detecting a plurality of optical signals corresponding to a plurality of pixels via an optical element having a plurality of optical transmittances for each of a plurality of optical components corresponding to the plurality of pixels; obtaining a first compressed image signal obtained by adding up the plurality of optical signals for each group of two or more adjacent pixels in the plurality of pixels; and generating a first restored image signal group having a plurality of restored image signals corresponding to the plurality of optical components from the first compressed image signal in accordance with first transmittance information relating to the plurality of optical transmittances.

2. The image processing method according to claim 1, wherein the plurality of light components are a plurality of wavelength bands.

3. The image processing method according to claim 1 or 2, wherein the first transmittance information is obtained by integrating the multiple light transmittances of second transmittance information having information on the multiple light transmittances corresponding to the multiple pixels for each of the multiple light components for each group of adjacent pixels of the two or more adjacent pixels.

4. The image processing method according to claim 1 or 2, wherein an optical signal corresponding to a first pixel of the two or more adjacent pixels is an optical signal that has passed through a first region of the optical element, and an optical signal corresponding to a second pixel of the two or more adjacent pixels is an optical signal that has passed through a second region of the optical element, and the optical transmission spectrum of the first region for the multiple light components is different from the optical transmission spectrum of the second region for the multiple light components.

5. The image processing method according to claim 1 or 2, wherein the optical element has a plurality of light transmission spectra corresponding to the plurality of pixels, each of the plurality of light transmission spectra being a light transmission spectrum for the plurality of light components, and the plurality of light transmission spectra having spatial randomness with respect to the plurality of pixels.

6. The image processing method according to claim 1 or 2, further comprising obtaining a second compressed image signal having the plurality of optical signals, wherein the first compressed image signal is obtained by adding up the plurality of optical signals of the second compressed image signal for each adjacent pixel group of the two or more adjacent pixels.

7. An image processing method according to claim 1 or 2, further comprising switching between a first mode in which the first group of restored image signals is generated from the first compressed image signal, and a second mode in which a second group of restored image signals is generated from a second compressed image signal having the plurality of optical signals.

8. The image processing method according to claim 7, wherein switching between the first mode and the second mode is performed based on an operation performed by a user.

9. The image processing method according to claim 7, wherein the number of pixel signals in each restored image signal of said first restored image signal group is smaller than the number of pixel signals in each restored image signal of said second restored image signal group.

10. The image processing method according to claim 1 or 2, wherein the number of pixel signals in each of the first restored image signal group is smaller than the number of the plurality of pixels.

11. The image processing method according to claim 1 or 2, wherein the number of pixel signals in each of said first restored image signal group is the same as the number of pixel signals in said first compressed image signal.

12. The image processing method according to claim 1 or 2, wherein the first transmittance information has, for each of the plurality of light components, a plurality of light transmittances that are smaller than the plurality of light transmittances corresponding to the plurality of pixels.

13. The image processing method according to claim 1, wherein the plurality of light components are a plurality of polarization directions.

14. The image processing method according to claim 1 or 2, wherein the first compressed image signal is obtained by adding up the multiple optical signals for each adjacent pixel group of the two or more adjacent pixels and performing an averaging process.

15. An image processing method comprising: detecting a plurality of optical signals corresponding to a plurality of pixels via an optical element having a plurality of optical transmission spectra corresponding to the plurality of pixels; obtaining a first compressed image signal obtained by adding up the plurality of optical signals for each group of adjacent pixels of two or more adjacent pixels in the plurality of pixels; and generating from the first compressed image signal a first restored image signal group having a plurality of restored image signals corresponding to a plurality of wavelength bands in accordance with information on the plurality of optical transmission spectra.

16. An image processing system comprising: an image sensor that detects a plurality of optical signals corresponding to a plurality of pixels via an optical element having a plurality of optical transmittances for each of a plurality of optical components corresponding to the plurality of pixels; and a processing circuit that acquires a first compressed image signal obtained by summing the plurality of optical signals for each adjacent pixel group of two or more adjacent pixels in the plurality of pixels, wherein the processing circuit further generates a first restored image signal group having a plurality of restored image signals corresponding to the plurality of optical components from the first compressed image signal in accordance with first transmittance information regarding the plurality of optical transmittances.

Images