Image processing method and image processing system

Abstract

This image processing method includes: acquiring a compressed image (S101); switching between a view mode and a save mode (S102); displaying, in the view mode and on a display device, a first image that is either the compressed image or a display processing image that is generated on the basis of the compressed image and is expressed with information on three or fewer wavelength bands (S104); deleting the first image after the first image is displayed (S105); generating, in the save mode and on the basis of the compressed image, a second image expressed with information on four or more wavelength bands (S106); and saving the second image in a recording medium (S107).

Description

Image processing method and image processing system

[0001] The present disclosure relates to an image processing method and the like.

[0002] It has been proposed to apply compressed sensing to a hyperspectral camera, that is, to restore a hyperspectral image from a compressed image. Patent documents 1 to 3 and non-patent documents 1 and 2 relate to such technologies.

[0003] International Publication No. 2022 / 202236 International Publication No. 2021 / 192891 International Publication No. 2020 / 080045

[0004] "World's first technology has been established to capture hyperspectral images and videos by combining a metalens and AI with a regular digital camera - Combining optical technology and AI to transform a regular camera into a camera that can see the properties of objects," [online], October 24, 2022, Nippon Telegraph and Telephone Corporation, [Retrieved November 16, 2023], Internet <URL: https: / / group.ntt / jp / newsrelease / 2022 / 10 / 24 / 221024a.html> Ahasan Ahamed et.al., "Reconstruction-based spectroscopy using CMOS image sensors with random photon-trapping" “nanostructure per sensor”, Proc. SPIE 11971, High-Speed ​​Biomedical Imaging and Spectroscopy VII, 1197106, 2 March 2022

[0005] However, restoring an image represented by information from four or more wavelength bands, such as a hyperspectral image, from a compressed image requires a huge amount of calculation, resulting in a large processing load and large power and energy consumption.

[0006] Therefore, an image processing method and the like that can reduce the processing load related to image restoration is provided, thereby making it possible to reduce the amount of calculations performed by one or more computers that execute the image processing method, thereby improving the functionality of the one or more computers that execute the image processing method.

[0007] An image processing method according to one aspect of the present disclosure includes acquiring a compressed image, switching between a viewing mode and a storage mode, displaying, in the viewing mode, a first image on a display device, which is one of the compressed image and a display processing image generated based on the compressed image and expressed by information of three or less wavelength bands, and deleting the first image after displaying the first image, and generating, in the storage mode, a second image expressed by information of four or more wavelength bands based on the compressed image, and saving the second image on a recording medium.

[0008] These comprehensive or specific aspects may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0009] The image processing method according to one aspect of the present disclosure makes it possible to reduce the processing load associated with image restoration.

[0010] 1 is a block diagram showing an example of the configuration of an image processing system according to an embodiment. FIG. 2 is a conceptual diagram showing an example of the configuration of an imaging device according to an embodiment. FIG. 3 is a conceptual diagram showing an example of the configuration of a filter array according to an embodiment. FIG. 4 is a graph showing an example of the transmission spectrum of a filter according to an embodiment. FIG. 5 is a graph showing an example of the transmission spectrum of another filter according to an embodiment. FIG. 6 is a schematic diagram showing an example of the transmittance of a first wavelength band of a filter array according to an embodiment. FIG. 7 is a schematic diagram showing an example of the transmittance of a second wavelength band of a filter array according to an embodiment. FIG. 8 is a flowchart showing an example of the operation of an image processing system according to an embodiment. FIG. 9 is a conceptual diagram showing an example of the operation of an image processing system according to an embodiment. FIG. 10 is an explanatory diagram showing a plurality of example combinations of types of images to be displayed and types of images to be saved according to an embodiment. FIG. 11 is a conceptual diagram showing a first display example according to an embodiment. FIG. 12 is an explanatory diagram showing the relationship between a display button, a save button, and modes according to an embodiment. FIG. 13 is a flowchart showing a first specific example of the operation of an image processing system according to an embodiment. FIG. 14 is a flowchart showing a second specific example of the operation of an image processing system according to an embodiment. FIG. 15 is an explanatory diagram showing the relationship between four or more wavelength bands and three wavelength bands according to an embodiment. FIG. 16 is a flowchart showing a third specific example of the operation of an image processing system according to an embodiment. FIG. 17 is a flowchart showing a fourth specific example of the operation of an image processing system according to an embodiment. 10 is a flowchart showing a sixth specific example of the operation of the image processing system in the embodiment. FIG. 11 is a flowchart showing a seventh specific example of the operation of the image processing system in the embodiment. FIG. 12 is a flowchart showing an eighth specific example of the operation of the image processing system in the embodiment. FIG. 13 is a diagram showing an example of data f indicating a restored image of three wavelength bands in the embodiment. FIG. 14 is a diagram showing an example of a matrix H corresponding to mask data of three wavelength bands in the embodiment. FIG. 15 is a diagram showing an example of data f indicating a low-resolution restored image of three wavelength bands in the embodiment. FIG. 16 is a diagram showing an example of a matrix H corresponding to low-resolution mask data of three wavelength bands in the embodiment. FIG. 17 is a flowchart showing a ninth specific example of the operation of the image processing system in the embodiment.1. A conceptual diagram showing a second display example in the embodiment. 2. A conceptual diagram showing a third display example in the embodiment. 3. A conceptual diagram showing a fourth display example in the embodiment. 4. A flowchart showing a tenth specific example of the operation of the image processing system in the embodiment. 5. A flowchart showing an eleventh specific example of the operation of the image processing system in the embodiment. 6. A diagram showing an example of the relationship between the first plurality of pixels included in the image sensor and the first plurality of pixel values, and the relationship between the plurality of pixels included in the image sensor and the second plurality of pixel values. Image I. ２１ ,...,Image I ２ｗ FIG.

[0011] For example, an RGB camera detects light and generates an RGB image represented by information in three wavelength bands corresponding to red, green, and blue. On the other hand, a hyperspectral camera detects light and generates a hyperspectral image represented by information in four or more wavelength bands. Hyperspectral cameras and hyperspectral images are used in various fields, such as food inspection, biological testing, medical drug development, and mineral analysis.

[0012] In addition, from the viewpoints of cost and flexibility, it has been proposed to apply compressed sensing to hyperspectral cameras, i.e., to reconstruct a hyperspectral image from a compressed image. Here, a compressed image is, for example, an image in which information from four or more wavelength bands is superimposed. From the compressed image, a hyperspectral image can be reconstructed according to sparsity.

[0013] However, restoring an image represented by information from four or more wavelength bands, such as a hyperspectral image, from a compressed image requires a huge amount of calculation, resulting in a large processing load and large power and energy consumption.

[0014] Therefore, the image processing method of Example 1 includes acquiring a compressed image, switching between a viewing mode and a storage mode, displaying, in the viewing mode, a first image on a display device, which is one of the compressed image and a display processing image generated based on the compressed image and expressed by information of three or less wavelength bands, and deleting the first image after displaying the first image, and generating, in the storage mode, a second image expressed by information of four or more wavelength bands based on the compressed image, and saving the second image on a recording medium.

[0015] This makes it possible to omit generation of the second image represented by information of four or more wavelength bands when storage is not performed, thereby reducing the processing load and reducing power consumption and energy consumption.

[0016] The image processing method of Example 2 may be the image processing method of Example 1, in which the first image is the compressed image.

[0017] This makes it possible to display the compressed image as is in the viewing mode, thereby further reducing the processing load.

[0018] The image processing method of Example 3 may be the image processing method of Example 1, in which the first image is the display processing image.

[0019] This makes it possible to generate a first image suitable for display in the viewing mode with a low amount of calculation, thereby making it possible to display a first image suitable for display while suppressing the processing load.

[0020] The image processing method of Example 4 may be the image processing method of Example 3, in which the first image is an RGB image expressed by information of three wavelength bands.

[0021] This makes it possible to generate an RGB image as the first image in the viewing mode, thereby making it possible to display a first image that is more suitable for display.

[0022] The image processing method of Example 5 may be the image processing method of Example 3, in which the first image is a monochrome image represented by information of one wavelength band.

[0023] This makes it possible to generate a monochrome image as the first image in the viewing mode, thereby further reducing the processing load.

[0024] The image processing method of Example 6 may be any of the image processing methods of Examples 3 to 5, in which the resolution of the first image is lower than the resolution of the second image.

[0025] This makes it possible to generate a low-resolution first image in the viewing mode, thereby further reducing the processing load.

[0026] Furthermore, the image processing method of Example 7 may be the image processing method of any one of Examples 3 to 6, wherein acquiring the compressed image involves acquiring the compressed image via a plurality of light-receiving regions having a plurality of transmission spectra; generating the first image involves generating the first image based on the compressed image and first mask data including a plurality of values ​​reflecting the plurality of transmission spectra; generating the second image involves generating the second image based on the compressed image and second mask data including a plurality of values ​​reflecting the plurality of transmission spectra; and the plurality of values ​​included in the first mask data may be fewer than the plurality of values ​​included in the second mask data.

[0027] This allows the first image to be generated in the viewing mode using first mask data that includes fewer values ​​than the second mask data used to generate the second image, thereby further reducing the processing load.

[0028] The image processing method of Example 8 is the image processing method of any one of Examples 1 to 7, in which the switching between the viewing mode and the storage mode is performed based on an operation performed by a user.

[0029] This makes it possible to adaptively switch between the viewing mode and the storage mode, thereby making it possible to adaptively reduce the processing load.

[0030] Furthermore, the image processing method of Example 9 may be any of the image processing methods of Examples 1 to 7, further including determining whether or not the first image includes a specific subject in the viewing mode, and when switching between the viewing mode and the storage mode, if it is determined that the first image does not include the specific subject, continuing the viewing mode, and if it is determined that the first image includes the specific subject, switching from the viewing mode to the storage mode.

[0031] This makes it possible to switch between the viewing mode and the storage mode depending on whether the first image includes a specific subject. If the first image includes a specific subject, it becomes possible to store the second image including the specific subject. Therefore, it becomes possible to efficiently store the second image including the specific subject.

[0032] The image processing method of Example 10 may be any of the image processing methods of Examples 1 to 9, further comprising displaying the first image on the display device in the save mode.

[0033] This allows the first image to be displayed in both the viewing mode and the storage mode, so that the first image can be displayed in the storage mode as well, continuing from the viewing mode.

[0034] The image processing method of Example 11 may be the image processing method of Example 10, further comprising the step of storing the first image in the recording medium in the storage mode.

[0035] This allows the first image displayed to be saved in the save mode, and therefore allows the first image displayed to be saved together with the second image in the save mode.

[0036] Furthermore, the image processing method of Example 12 may be the image processing method of Example 10 or 11, further comprising, in the save mode, displaying one or both of the second image and the analysis result of the subject based on the second image.

[0037] This makes it possible to display information corresponding to the second image in the save mode, and therefore makes it possible to display the first image and information corresponding to the second image in the save mode.

[0038] Furthermore, the image processing method of Example 13 may be any of the image processing methods of Examples 3 to 6, and further, in the storage mode, generate the first image based on the second image generated based on the compressed image, and display the first image on the display device.

[0039] This allows the first image to be displayed in both the viewing mode and the storage mode. Therefore, the first image can be displayed in the storage mode, continuing from the viewing mode. Furthermore, in the storage mode, the first image can be efficiently generated based on the second image. Therefore, an increase in the processing load in the storage mode is suppressed.

[0040] Moreover, the image processing system of Example 14 includes an image sensor that acquires a compressed image, and a processing circuit that switches between a viewing mode and a storage mode, and in the viewing mode, the processing circuit displays on a display device a first image that is one of the compressed image and a display processed image that is generated based on the compressed image and is expressed by information of three or less wavelength bands, and deletes the first image after displaying the first image, and in the storage mode, generates a second image that is expressed by information of four or more wavelength bands based on the compressed image, and stores the second image on a recording medium.

[0041] This makes it possible to omit generation of the second image represented by information of four or more wavelength bands when storage is not performed, thereby reducing the processing load and reducing power consumption and energy consumption.

[0042] The program of Example 15 is a program for causing a computer to execute any one of the image processing methods of Examples 1 to 13.

[0043] This makes it possible to realize the image processing method as a program for causing a computer to execute the method.

[0044] Furthermore, these comprehensive or specific aspects may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0045] Hereinafter, embodiments will be described with reference to the drawings. Note that the embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, the arrangement and connection of the components, steps, the order of steps, and the like shown in the following embodiments are merely examples and are not intended to limit the scope of the claims.

[0046] Fig. 1 is a block diagram showing an example of the configuration of an image processing system according to an embodiment. As shown in Fig. 1, the image processing system 100 includes an image sensor 111 and a processing circuit 121. The image processing system 100 may further include a display device 130 and a recording medium 140. The image processing system 100 may also include an imaging device 110 and an image processing device 120. The imaging device 110 may then include the image sensor 111. The image processing device 120 may then include the processing circuit 121.

[0047] The image sensor 111 acquires an image by detecting an optical signal for each pixel and generating an image represented by a plurality of optical signals corresponding to a plurality of pixels. Here, the image sensor 111 acquires a compressed image. Specifically, for example, the image sensor 111 acquires a compressed image in which information of four or more wavelength bands is superimposed for each pixel based on a filter array described below. Note that a wavelength band may be simply referred to as a band.

[0048] The image sensor 111 may be a monochrome photodetector having a plurality of photodetection elements arranged in a matrix. More specifically, the image sensor 111 may be a charge-coupled device (CCD) image sensor, a complementary metal oxide semiconductor (CMOS) image sensor, an infrared array image sensor, a terahertz array image sensor, or a millimeter wave array image sensor.

[0049] Alternatively, the image sensor 111 may be a color-type photodetector. The wavelength range detectable by the image sensor 111 is not limited, and may be, for example, visible light, ultraviolet light, infrared light, terahertz waves, or any combination thereof.

[0050] The processing circuitry 121 is a circuit that performs information processing. For example, the processing circuitry 121 generates a hyperspectral image represented by information of four or more wavelength bands by performing a restoration operation on a compressed image. For example, the hyperspectral image is composed of four or more spectral images corresponding to four or more wavelength bands. The restoration operation may be the same as the restoration operation described in Patent Documents 2 and 3. Specifically, four or more spectral images may be generated as the hyperspectral image based on the following equation (1):

[0051]

[0052] Here, g is data representing a compressed image and is represented, for example, by a one-dimensional array (i.e., a vector). If the compressed image is an image of n×m pixels, data g is represented by a one-dimensional array having n×m elements. f is data representing w spectral images corresponding one-to-one to w wavelength bands and is represented, for example, by a one-dimensional array. When w spectral images constitute a hyperspectral image, w is an integer greater than or equal to 4.

[0053] f １ is the wavelength band W １ Spectral image data corresponding to f ２ is the wavelength band W ２ Spectral image data corresponding to fｗ is the wavelength band W ｗ The spectral image data corresponds to the

[0054] Data f １ , f ２ , ..., f ｗ Each of the spectral images is represented by, for example, a one-dimensional array. If each spectral image is an image of n×m pixels, the data f １ , f ２ , ..., f ｗ are represented by a one-dimensional array having n×m elements, and data f is represented by a one-dimensional array having n×m×w elements. H is a matrix with n×m rows and n×m×w columns, called the system matrix, and corresponds to the mask data.

[0055] The matrix H is the wavelength band W of the filter array described below. １ Transmission spectrum of wavelength band W ２ Transmission spectrum of wavelength band W ｗ The transmittance may be determined based on the transmittance spectrum of the light emitting element.

[0056] The data f that satisfies equation (1) can be estimated using a compressed sensing technique, specifically, by equation (2).

[0057]

[0058] Equation (2) expresses finding data f that minimizes the sum of the first and second terms in the parentheses. Data f can be calculated as final calculation result data by converging the calculation result data through recursive iterative calculation.

[0059] The first term in the parentheses in equation (2) represents the sum of squares of the difference between data g and data Hf obtained by transforming data f in the estimation process using matrix H, and is a so-called residual term. Although the sum of squares is used here, the sum of absolute values, the square root of the sum of squares, or the like may be used instead of the sum of squares.

[0060] The sum of squares of the difference between data Hf and data g is (g １ -r １ ) × (g １ -r １ ) + ... + (g ｎ×ｍ-r ｎ×ｍ ) × (g ｎ×ｍ -r ｎ×ｍ ) where g １ , ..., g ｎ×ｍ is g = (g １ ...g ｎ×ｍ ) Ｔ is an element of data g expressed as １ , ..., r ｎ×ｍ is Hf = (r １ ...r ｎ×ｍ ) Ｔ The elements of data Hf are expressed as follows:

[0061] The second term in the parentheses in Equation (2) is a regularization term, sometimes called a stabilization term. Φ(f) represents a constraint on the regularization of f and is a function that reflects the sparse information of the data f. This function has the effect of smoothing or stabilizing the data f. Φ(f) can be expressed, for example, by a discrete cosine transform (DCT), a wavelet transform, a Fourier transform, a total variation (TV), or any combination thereof.

[0062] τ is a weighting coefficient for the regularization term, and corresponds to the influence of regularization in the reconstruction calculation. The larger the value of τ, the greater the influence of regularization, and the stronger the convergence of the solution in the iterative calculation. Conversely, the smaller the value of τ, the less the influence of regularization, and the weaker the convergence of the solution in the iterative calculation.

[0063] The display device 130 is a display device for displaying information. A compressed image, a monochrome image, an RGB image, a hyperspectral image, or the like is displayed on the display device 130 by the processing circuit 121. The display device 130 may be, for example, a liquid crystal display (LCD) or an organic light-emitting diode (OLED) display.

[0064] A graphical user interface (GUI) may be displayed on the display device 130. The display device 130 may also be a touch panel. A user may input information to the display device 130. The processing circuitry 121 may acquire information from the user via the display device 130. That is, the display device 130 may be an input / output device. Alternatively, the processing circuitry 121 may acquire information from the user via an input device different from the display device 130.

[0065] The recording medium 140 is a storage element for storing information. Compressed images, monochrome images, RGB images, hyperspectral images, etc. are stored in the recording medium 140 by the processing circuit 121. The recording medium 140 can be implemented as, for example, a CD-ROM, a hard disk drive, or a solid state drive.

[0066] 1 shows an example of the configuration of the image processing system 100, but the configuration of the image processing system 100 is not limited to the example of the configuration in FIG. 1. For example, multiple devices may be integrated into one device, or one device may be distributed among multiple devices. Furthermore, multiple devices that are distributed may be able to communicate with each other via wired or wireless communication.

[0067] The processing circuit 121 corresponds to an acquirer that acquires images from the image sensor 111, a switch that switches modes, a generator that generates images, a discard controller that discards images, a display controller that displays images on the display device 130, and a storage controller that saves images in the recording medium 140. The image processing device 120, rather than the processing circuit 121, may include some or all of these components.

[0068] Fig. 2 is a conceptual diagram showing an example of the configuration of the imaging device 110 shown in Fig. 1. The imaging device 110 may have a configuration similar to that of the imaging devices disclosed in Patent Documents 2 and 3. For example, the imaging device 110 includes an image sensor 111, a filter array 112, and an optical system 113.

[0069] The filter array 112 is disposed on the optical path of light incident from the subject, and is disposed between the optical system 113 and the image sensor 111. The filter array 112 functions as the encoding element described in Patent Document 2. The filter array 112 may be integrated with the image sensor 111.

[0070] The arrangement of the filter array 112 is not limited to the arrangement shown in Fig. 2. For example, the filter array 112 may be arranged between the optical system 113 and the image sensor 111, but away from the image sensor 111. Alternatively, the filter array 112 may be arranged between the subject and the optical system 113. Alternatively, the filter array 112 may be arranged within the optical system 113.

[0071] The optical system 113 is disposed on the optical path of light incident from the subject, and is disposed between the subject and the filter array 112. The optical system 113 includes at least one lens, and can form an image of the subject on the imaging surface of the image sensor 111 via the filter array 112.

[0072] The configuration and arrangement of the optical system 113 are not limited to those shown in Fig. 2. For example, the optical system 113 may be arranged between the filter array 112 and the image sensor 111. Furthermore, for example, the optical system 113 may include a plurality of lenses arranged on the optical path. In this case, the filter array 112 may be arranged between adjacent lenses of the plurality of lenses.

[0073] 3 is a conceptual diagram showing an example of the configuration of the filter array 112 shown in FIG. 2. The filter array 112 is made up of a plurality of filters F １１ , ..., F ｎｍ In the example shown in FIG. 3, the filter array 112 includes 48 filters F １１ , ..., F ｎｍ The filter F １１ is 48 filters F １１ , ..., F ｎｍ The filter F is located at the top left of the ｎｍ is 48 filters F １１, ..., F ｎｍ This is the filter located at the bottom right of the list.

[0074] The filter F included in the filter array 112 １１ , ..., F ｎｍ The number of filters F included in the filter array 112 is not limited to 48. １１ , ..., F ｎｍ The number of filters F included in the filter array 112 may be approximately the same as the number of pixels of the image sensor 111, and may be determined depending on the application, for example, in the range from several tens to several tens of millions. １１ , ..., F ｎｍ The number of pixels in the image sensor 111 may be the same as or different from the number of pixels in the image sensor 111. １１ , ..., F ｎｍ may have a one-to-one correspondence with

[0075] For example, the filter array 112 may include n×m filters F corresponding to n×m pixels. １１ , ..., F ｎｍ Waveband W １ , ..., W ｗ In this case, the filter F １１ , ..., F ｎｍ are the transmission spectrum S １１ , ..., S ｎｍ The transmission spectrum S １１ , ..., S ｎｍ Alternatively, the transmission spectrum S １１ , ..., S ｎｍ may be the same. Here, the transmission spectrum may refer to the light transmittance spectrum.

[0076] n×m filters F １１ , ..., F ｎｍ is w wavelength bands W １ , ..., W ｗ For n×m×w transmittances S １１Ｗ１ , ..., S ｎｍＷｗ The transmittance S １１Ｗ１ , ..., S ｎｍＷｗAlternatively, the transmittance S １１Ｗ１ , ..., S ｎｍＷｗ Here, transmittance may refer to light transmittance.

[0077] Wavelength band W α The transmittance of the filter β in the above formula may be expressed by the following formula (3):

[0078]

[0079] where Wαmin is the wavelength band W α is the minimum wavelength value of the wavelength band W α is the maximum wavelength value of h(λ), and h(λ) is a function indicating the transmission spectrum, where λ is the wavelength.

[0080] In addition, the wavelength band W α The transmittance of the filter β in the wavelength band W is not limited to the formula (3). α The transmittance of the filter β in the wavelength band W may be the transmittance obtained by dividing the formula (3) by (Wαmax-Wαmin). α The transmittance of the filter β in the wavelength band W α The wavelength λ that represents α０ Transmittance h (λ α０ ) may also be used.

[0081] Here, the wavelength λ α０ is Wαmin≦λ α０ For example, the wavelength λ α０ is the wavelength band W α The center wavelength may be ((Wαmax−Wαmin) / 2).

[0082] FIG. 4 shows the filter F shown in FIG. １１ 5 is an example of the transmission spectrum of the filter F shown in FIG. ｎｍ 6 is an example of the transmission spectrum of the wavelength band W of the filter array 112 shown in FIG. １ 7 is a diagram showing an example of the transmittance of the wavelength band W of the filter array 112 shown in FIG. ２6 and 7, the shading of each region represents the transmittance of the filter, with lighter regions representing higher transmittance and darker regions representing lower transmittance.

[0083] A plurality of filters F included in the filter array 112 １１ , ..., F ｎｍ The wavelength dependence of the transmittance of each filter is different. １１ In the wavelength band W １ The transmittance of the wavelength band W ２ On the other hand, the transmittance of the filter F ｎｍ In the wavelength band W １ The transmittance of the wavelength band W ２ The transmittance of the filter F is approximately the same as that of the filter F. １１ The wavelength dependence of the transmittance of the filter F ｎｍ The wavelength dependence of the transmittance is different from that of the

[0084] Here, w wavelength bands W １ , ..., W ｗ Two of the wavelength bands W １ and W ２ The transmittance of the w wavelength bands W １ , ..., W ｗ The transmittance of other wavelength bands will not be shown or explained.

[0085] For example, the mask data may include w wavelength bands W １ , W ２ , ..., W ｗ Specifically, the mask data is expressed as w matrices corresponding to w wavelength bands W １ , W ２ , ..., W ｗ is expressed by a matrix of transmittances (transmittance matrix) with n rows and m columns corresponding to pixels with n rows and m columns. Then, by changing the representation format from the mask data expressed by w matrices each having n rows and m columns, a single matrix H with n×m rows and n×m×w columns is obtained.

[0086] H included in the formula (1) is １ H２ ...H ｗ ) can also be expressed as １ , H ２ , ..., H ｗ are submatrices of H. In other words, if the multiple elements contained in H are a(11), ..., a((n × m)(n × m × w)), then

[0087]

[0088] is.

[0089] H １ is the wavelength band W １ The mask data corresponding to H ２ is the wavelength band W ２ Mask data for, . . ., H ｗ is the wavelength band W ｗ This may be interpreted as mask data for

[0090] H １ , H ２ , ..., H ｗ Each of H may be a diagonal matrix. １ , H ２ , ..., H ｗ teeth,

[0091]

[0092] may be.

[0093] Fig. 8 is a flowchart showing an example of the operation of the image processing system 100 shown in Fig. 1. In this example, the image sensor 111 of the imaging device 110 acquires a compressed image by generating a compressed image, and the processing circuitry 121 of the image processing device 120 acquires the compressed image from the image sensor 111 (S101). In other words, the operation of acquiring a compressed image may correspond to the operation of the image sensor 111 acquiring a compressed image by generating a compressed image, or may correspond to the operation of the processing circuitry 121 acquiring a compressed image from the image sensor 111.

[0094] Then, the processing circuitry 121 switches between the viewing mode and the storage mode (S102). Here, the processing circuitry 121 may switch from the viewing mode to the storage mode, or may switch from the storage mode to the viewing mode, or may maintain the viewing mode, or may maintain the storage mode.

[0095] In the viewing mode (visual mode in S103), the processing circuitry 121 displays a first image on the display device 130 (S104). Here, the first image may be a compressed image. Alternatively, the first image may be a display-processed image generated based on the compressed image and represented by information of three or fewer wavelength bands. Then, the processing circuitry 121 deletes the first image after displaying it (S105).

[0096] In the storage mode (S103: storage mode), the processing circuitry 121 generates a second image based on the compressed image (S106). The second image is an image represented by information of four or more wavelength bands. The processing circuitry 121 then stores the second image on the recording medium 140 (S107).

[0097] This makes it possible to omit generation of the second image represented by information of four or more wavelength bands when storage is not performed, thereby reducing the processing load and reducing power consumption and energy consumption.

[0098] For example, the above series of operations (S101 to S107) may be repeated. In the viewing mode, a plurality of first images may be displayed as a moving image, and in the storage mode, a plurality of second images may be stored as a moving image.

[0099] Furthermore, for example, if the first image is a compressed image, the compressed image can be displayed as is in the viewing mode, thereby further reducing the processing load.

[0100] Furthermore, for example, when the first image is a display processing image, the processing circuitry 121 may generate the first image based on a compressed image in the viewing mode. This makes it possible to generate a first image suitable for display in the viewing mode with a low amount of calculation. Therefore, it is possible to display a first image suitable for display while suppressing the processing load.

[0101] Furthermore, for example, the first image may be an RGB image represented by information of three wavelength bands, which makes it possible to generate an RGB image as the first image in the viewing mode, thereby enabling the display of a first image that is more suitable for display.

[0102] Furthermore, for example, the first image may be a monochrome image represented by information of one wavelength band, which makes it possible to generate a monochrome image as the first image in the visibility mode, thereby further reducing the processing load.

[0103] Furthermore, for example, the resolution of the first image may be lower than the resolution of the second image, which allows a low-resolution first image to be generated in the viewing mode, thereby further reducing the processing load.

[0104] Furthermore, for example, the image sensor 111 may acquire a compressed image through a plurality of light-receiving regions having a plurality of transmission spectra. Specifically, the image sensor 111 may acquire a compressed image through a filter array 112. Each of the plurality of light-receiving regions may correspond to each of the plurality of filters included in the filter array 112.

[0105] Alternatively, the metalens described in Non-Patent Document 1 may be used instead of the filter array 112. In this case, the wavelength transmittance varies depending on the location. The light receiving region corresponds to such a location. Alternatively, the CMOS image sensor described in Non-Patent Document 2 may be used instead of the filter array 112. In this case, the sensing region is processed so that a predetermined transmittance is obtained for each pixel. The light receiving region corresponds to such a sensing region.

[0106] The light receiving region may be expressed as an optical element, a modulation element, an encoding element, an optical processing region, a masking region, or the like.

[0107] The processing circuitry 121 may then generate a first image based on the compressed image and the first mask data, where the first mask data includes a plurality of values ​​reflecting a plurality of transmission spectra. The processing circuitry 121 may also generate a second image based on the compressed image and the second mask data, where the second mask data includes a plurality of values ​​reflecting a plurality of transmission spectra. The number of values ​​included in the first mask data is smaller than the number of values ​​included in the second mask data.

[0108] This allows the first image to be generated in the viewing mode using first mask data that includes fewer values ​​than the second mask data used to generate the second image, thereby further reducing the processing load.

[0109] Alternatively, the first mask data may be generated by, for example, integrating matrix elements of the second mask data. This reduces the number of values ​​included in the first mask data. Furthermore, it is possible to efficiently generate both the first mask data for generating the first image and the second mask data for generating the second image from a common set of transmission spectra. This reduces the number of calibrations required.

[0110] Furthermore, for example, the processing circuitry 121 may switch between the viewing mode and the storage mode based on an operation performed by a user. This allows adaptive switching between the viewing mode and the storage mode, thereby enabling adaptive reduction of the processing load.

[0111] Furthermore, for example, in the viewing mode, the processing circuitry 121 may further determine whether the first image includes a specific subject. Here, if it is determined that the first image does not include a specific subject, the processing circuitry 121 may continue the viewing mode. On the other hand, if it is determined that the first image includes a specific subject, the processing circuitry 121 may switch from the viewing mode to the storage mode.

[0112] This makes it possible to switch between the viewing mode and the storage mode depending on whether the first image includes a specific subject. If the first image includes a specific subject, it becomes possible to store the second image including the specific subject. Therefore, it becomes possible to efficiently store the second image including the specific subject.

[0113] Furthermore, for example, the processing circuitry 121 may display the first image on the display device 130 in the storage mode. This allows the first image to be displayed in both the viewing mode and the storage mode. Therefore, the first image can be displayed in the storage mode, continuing from the viewing mode.

[0114] Furthermore, for example, in the save mode, processing circuitry 121 may store the first image on recording medium 140. This allows the displayed first image to be stored in the save mode. Therefore, in the save mode, the displayed first image can be stored together with the second image.

[0115] Furthermore, for example, in the storage mode, the processing circuitry 121 may display one or both of the second image and the analysis result of the subject based on the second image. This makes it possible to display information corresponding to the second image in the storage mode. Therefore, in the storage mode, it is possible to display the first image and information corresponding to the second image.

[0116] Furthermore, for example, in the storage mode, the processing circuitry 121 may generate a first image based on a second image generated based on a compressed image, and then display the first image on the display device 130.

[0117] This allows the first image to be displayed in both the viewing mode and the storage mode. Therefore, the first image can be displayed in the storage mode, continuing from the viewing mode. Furthermore, in the storage mode, the first image can be efficiently generated based on the second image. Therefore, an increase in the processing load in the storage mode is suppressed.

[0118] Fig. 9 is a conceptual diagram showing an example of the operation of the image processing system 100 shown in Fig. 1. For example, the image sensor 111 repeatedly captures images (photographs) based on the frame rate, thereby repeatedly acquiring compressed images.

[0119] The processing circuit 121 switches between a viewing mode and a storage mode by selecting a mode for each frame. The viewing mode is also referred to as a display mode, a viewer mode, or an image capture (photography) mode, and is a mode in which images are not stored. The storage mode is also referred to as a recording mode, and is a mode in which images are stored.

[0120] In the viewing mode, the processing circuitry 121 repeatedly acquires a compressed image via the image sensor 111, restores a first image from the compressed image, displays the first image on the display device 130, and discards the first image. The first image may be an RGB image or a monochrome image. Alternatively, the first image may be the compressed image itself. In this case, the processing circuitry 121 does not need to perform restoration processing in the viewing mode.

[0121] In the storage mode, the processing circuitry 121 repeats the operations of acquiring a compressed image via the image sensor 111, restoring a second image from the compressed image, and storing the second image on the recording medium 140. The second image may be a hyperspectral image. Furthermore, in the storage mode, the processing circuitry 121 may repeat the operations of generating a first image from the second image, displaying the first image on the display device 130, and discarding the first image.

[0122] Alternatively, in the storage mode, the processing circuitry 121 may restore the first image from the compressed image, as in the viewing mode. If the first image is a compressed image, the generation process (restoration process) of the first image may not be performed. Alternatively, in the storage mode, the processing circuitry 121 may display the second image instead of the first image.

[0123] Furthermore, in the storage mode, processing circuitry 121 may store the first image displayed on display device 130 in recording medium 140 in addition to the second image. In other words, processing circuitry 121 may store the first image in recording medium 140 without discarding it.

[0124] FIG. 10 is an explanatory diagram showing a plurality of example combinations of the types of images to be displayed and the types of images to be saved in the example of operation shown in FIG.

[0125] Specifically, the image to be displayed is a first image. The image to be saved may be a second image, or may be both the first and second images. The first image is, for example, a compressed image, a monochrome image, an RGB image, a low-resolution monochrome image, or a low-resolution RGB image. The second image is, for example, a hyperspectral image.

[0126] Furthermore, when the image to be displayed is a monochrome image, an RGB image, a low-resolution monochrome image, or a low-resolution RGB image, the image to be saved may include a compressed image. In other words, in this case, the images to be saved may be two images, the hyperspectral image and the compressed image, or three images, the hyperspectral image, the image to be displayed, and the compressed image.

[0127] In any of these examples, if the hyperspectral image is not saved, ie, the mode is not a save mode, then no image is saved.

[0128] Fig. 11 is a conceptual diagram showing a first display example of the display device 130 shown in Fig. 1. For example, the display device 130 displays the GUI shown in Fig. 11. This GUI includes a first image, a display button, and a save button. The first image included in the GUI is the first image generated (restored) by the processing circuitry 121. The display button and the save button are each switched between an ON state and an OFF state by a user operation.

[0129] The viewing mode and the saving mode are switched by switching the display button and the saving button between the ON state and the OFF state. The display button may be a photographing button or a viewing button. The saving button may be a recording button.

[0130] 12 is an explanatory diagram showing the relationship between the display button, the save button, and the mode in the embodiment. When the display button is in the OFF state, the save button is also in the OFF state, and the mode is not selected. In this case, no image is generated and no image is displayed.

[0131] When the display button is in the ON state, the save button can be switched to the ON state or the OFF state. When the display button is in the ON state and the save button is in the OFF state, the mode is the viewing mode. When the display button is in the ON state and the save button is in the ON state, the mode is the save mode.

[0132] For example, if the display button is in an ON state, the first image is displayed, and if the save button is in an ON state, the second image is saved.

[0133] FIG. 13 is a flowchart showing a first specific example of the operation of the image processing system 100 shown in FIG.

[0134] In this example, the processing circuitry 121 determines whether the display button is in the ON state (S201).

[0135] If the display button is ON (Yes in S201), the processing circuitry 121 acquires a compressed image (S202). Specifically, the image sensor 111 acquires the compressed image by generating it, and the processing circuitry 121 acquires the compressed image from the image sensor 111. Then, the processing circuitry 121 discards the compressed image displayed in the previous frame (S203). Then, the processing circuitry 121 displays the compressed image acquired in the current frame (S204).

[0136] Furthermore, the processing circuitry 121 determines whether the save button is ON (S205). If the save button is ON (Yes in S205), the hyperspectral image is restored using mask data for four or more wavelength bands and saved (S206). Then, the series of processes (S201 to S206) is repeated. On the other hand, if the save button is OFF (No in S205), the hyperspectral image is not restored and the processes (S201 to S205) are repeated.

[0137] If the display button is in the OFF state (No in S201), the processing circuitry 121 discards the compressed image displayed in the previous frame (S209), and the image processing system 100 ends its operation.

[0138] In the example of Fig. 13, when the save button is ON, the hyperspectral image is restored and saved. When the display button is ON and the save button is OFF, the compressed image is displayed and the hyperspectral image is not restored. Therefore, the processing load is reduced.

[0139] FIG. 14 is a flowchart showing a second specific example of the operation of the image processing system 100 shown in FIG.

[0140] Compared to the example of Figure 13, in the example of Figure 14, instead of displaying the compressed image (S204), the processing circuitry 121 restores and displays the RGB image of the current frame from the compressed image of the current frame using mask data of the three wavelength bands corresponding to the RGB image (S204a).

[0141] Furthermore, in association with the restoration and display of the RGB image (S204a), the processing circuitry 121 discards the RGB image displayed in the previous frame (S203a and S209a) instead of discarding the compressed image displayed in the previous frame (S203 and S209). At this time, the processing circuitry 121 may also discard the compressed image used in the previous frame.

[0142] Except for the above, the example in Fig. 14 is the same as the example in Fig. 13. In the example in Fig. 14, when the save button is in the ON state, the hyperspectral image is restored and saved. When the display button is in the ON state and the save button is in the OFF state, the RGB image is restored and displayed, but the hyperspectral image is not restored. Therefore, the processing load is reduced.

[0143] The restoration of an RGB image is performed in the same manner as the restoration of a hyperspectral image, but using mask data of three wavelength bands corresponding to the RGB image instead of mask data of four or more wavelength bands. For example, the restoration calculation described in Patent Document 2 may be used. Specific examples will be described with reference to FIG. 15 .

[0144] 15 is a conceptual diagram showing the relationship between four or more wavelength bands and three wavelength bands. In FIG. 15, W is a wavelength range that can be detected by the image sensor 111. The wavelength range W may be a transmission wavelength range of a band-pass filter included in the image sensor 111.

[0145] The wavelength range W is the number of wavelength bands W (where w is 4 or more) of the hyperspectral image. １ , ..., W ｗ The wavelength range W corresponds to the range covered by the three wavelength bands W of the RGB image. Ｒ , W Ｇ and W Ｂ That is, the wavelength band W １ , ..., W ｗ The range covered by the three wavelength bands W of the RGB image is Ｒ , W Ｇ and W Ｂ corresponds to the range covered by

[0146] For example, the wavelength range W is in the range of 400 nm to 700 nm. １ , ..., W ｗ is determined in 10 nm increments. Therefore, w is 30, and there are w wavelength bands W １ , W ２ , ..., W ｗare 400 nm to 410 nm, 410 nm to 420 nm, ..., 690 nm to 700 nm, respectively. １ , ..., W ｗ To generate a hyperspectral image of １ , ..., W ｗ The mask data is prepared.

[0147] w wavelength bands W １ , W ２ , ..., W ｗ The mask data includes a transmittance matrix for a wavelength band of 400 nm to 410 nm, a transmittance matrix for a wavelength band of 410 nm to 420 nm, . . . , a transmittance matrix for a wavelength band of 690 nm to 700 nm.

[0148] And the wavelength band W of 400 nm to 500 nm Ｂ By integrating the 10 matrices included in the range of Ｂ The transmittance matrix for the wavelength band W of 500 nm to 600 nm is obtained. Ｇ By integrating the 10 matrices included in the range of Ｇ The transmittance matrix for the wavelength band W of 600 nm to 700 nm is obtained. Ｒ By integrating the 10 matrices included in the range of Ｒ The transmittance matrix is ​​obtained.

[0149] In integrating the ten transmittance matrices, the ten transmittances may be added together or averaged for each element. １ , ..., W ｗ From the mask data, three wavelength bands W corresponding to the RGB image are obtained. Ｒ , W Ｇ and W Ｂ That is, to generate an RGB image, mask data of w wavelength bands W １ , ..., W ｗ From the mask data, the three wavelength bands W of the RGB image are Ｒ , W Ｇ and W ＢThe mask data can be prepared.

[0150] H included in the formula (1) is H = (H １ H ２ ...H ｗ ) = (H １ H ２ ...H Ｐ H Ｐ＋１ H Ｐ＋２ ...H Ｑ H Ｑ＋１ H Ｑ＋２ ...H ｗ ) is expressed as the wavelength band W Ｒ A matrix H indicating mask data for Ｒ , wavelength band W Ｇ A matrix H indicating mask data for Ｇ , wavelength band W Ｂ A matrix H indicating mask data for Ｂ may be expressed as equation (4) or equation (5). １ , H ２ , ..., H Ｐ , H Ｐ＋１ , H Ｐ＋２ , ..., H Ｑ , H Ｑ＋１ , H Ｑ＋２ , ..., H ｗ Each of H is a submatrix of H, with elements arranged in n×m rows and n×m columns.

[0151]

[0152] In the above example, P=10, Q=20, and W=30.

[0153] Then, the processing circuit 121 calculates the three wavelength bands W Ｒ , W Ｇ and W Ｂ Based on the mask data and the above-mentioned formulas (1) and (2), an RGB image can be generated. H and f included in formulas (1) and (2) are expressed as follows: H=(H Ｂ H Ｇ H Ｒ )

[0154]

[0155] It may be H Ｂ , HＧ , H Ｒ , are submatrices of H.

[0156] f Ｒ is the wavelength band W Ｒ The spectral image data corresponding to the wavelength band W Ｒ are a plurality of pixel values ​​of the image corresponding to

[0157] f Ｇ is the wavelength band W Ｇ The spectral image data corresponding to the wavelength band W Ｇ are a plurality of pixel values ​​of the image corresponding to

[0158] f Ｂ is the wavelength band W Ｂ The spectral image data corresponding to the wavelength band W Ｂ are a plurality of pixel values ​​of the image corresponding to

[0159] f Ｒ , f Ｇ , f Ｂ The values ​​of the pixels included in the RGB image may be determined based on the n×m pixel values ​​included in each of the pixels.

[0160] w wavelength bands W １ , ..., W ｗ The mask data is a set of n×m transmission spectra S １１ , ..., S ｎｍ Similarly, the three wavelength bands W Ｒ , W Ｂ and W Ｇ The mask data is a set of n×m transmission spectra S １１ , ..., S ｎｍ The transmittance values ​​are n×m×3, which reflect the above.

[0161] In other words, it is possible to efficiently generate both mask data for generating an RGB image and mask data for generating a hyperspectral image from a common set of transmission spectra, thereby reducing the number of calibrations required.

[0162] In the above, w wavelength bands W of the hyperspectral image are １ , ..., W ｗ Using the mask data, the three wavelength bands W of the RGB image are Ｒ , W Ｂ and W Ｇ However, based on the above-mentioned equation (3), three wavelength bands W Ｒ , W Ｂ and W Ｇ By deriving the three transmittances of the three wavelength bands W Ｒ , W Ｇ and W Ｂ The mask data may be derived.

[0163] FIG. 16 is a flowchart showing a third specific example of the operation of the image processing system 100 shown in FIG.

[0164] In comparison with the example of FIG. 14, in the example of FIG. 16, when the save button is in the ON state (Yes in S205), the processing circuitry 121 saves the RGB image in addition to the hyperspectral image (S208).

[0165] Except for the above, the example in Fig. 16 is the same as the example in Fig. 14. In the example in Fig. 16, when the save button is in the ON state, the hyperspectral image is restored and saved. When the display button is in the ON state and the save button is in the OFF state, the RGB image is restored and displayed, but the hyperspectral image is not restored. Therefore, the processing load is reduced.

[0166] 16, if the save button is in the ON state, the displayed RGB image is saved. Therefore, in the save mode, it is possible to save the RGB image valid for display together with the hyperspectral image.

[0167] Fig. 17 is a flowchart showing a fourth specific example of the operation of the image processing system 100 shown in Fig. 1. In the example of Fig. 17, the method of generating an RGB image is different from the example of Fig. 14.

[0168] Specifically, in the example of Fig. 17, the method of generating an RGB image changes depending on whether the Save button is ON. Specifically, if the Save button is OFF (No in S205), the RGB image is restored and displayed using mask data for the three wavelength bands (S204a). This process (S204a in Fig. 17) is the same as the process (S204a in Fig. 14) that is performed in the example of Fig. 14, regardless of whether the Save button is ON.

[0169] On the other hand, if the save button is in the ON state (Yes in S205), the processing circuitry 121 restores the hyperspectral image, then generates an RGB image from the hyperspectral image, and displays it (S207a). Ｒ By integrating multiple spectral images of multiple wavelength bands included in Ｒ In integrating the plurality of spectral images, the values ​​of the plurality of spectral images may be added together, weighted, or averaged for each pixel.

[0170] Similarly, among the multiple spectral images included in the hyperspectral image, the green wavelength band W Ｇ By integrating multiple spectral images of multiple wavelength bands included in the green wavelength band W Ｇ Furthermore, among the multiple spectral images included in the hyperspectral image, the blue wavelength band W Ｂ By integrating multiple spectral images of multiple wavelength bands included in Ｂ Spectral images can be obtained.

[0171] And the red wavelength band W Ｒ Spectral image of the green wavelength band W Ｇ Spectral image of the blue wavelength band W Ｂ By combining this with the spectral image, an RGB image is obtained.

[0172] Except for the above, the example in Fig. 17 is the same as the example in Fig. 14. In the example in Fig. 17, when the save button is in the ON state, the hyperspectral image is restored and saved. When the display button is in the ON state and the save button is in the OFF state, the RGB image is restored and displayed, but the hyperspectral image is not restored. Therefore, the processing load is reduced.

[0173] 17, if the save button is ON, an RGB image is generated from the restored hyperspectral image. The processing load of such a generation process is smaller than the processing load of a restoration process that restores an RGB image from a compressed image. Therefore, the processing load is reduced by the above operation.

[0174] FIG. 18 is a flowchart showing a fifth specific example of the operation of the image processing system 100 shown in FIG.

[0175] Compared to the example of Figure 17, in the example of Figure 18, if the save button is in the ON state (Yes in S205), the processing circuitry 121 saves the RGB image generated from the hyperspectral image in addition to the hyperspectral image (S208a).

[0176] Except for the above, the example in Fig. 18 is the same as the example in Fig. 17. In the example in Fig. 18, when the save button is in the ON state, the hyperspectral image is restored and saved. When the display button is in the ON state and the save button is in the OFF state, the RGB image is restored and displayed, but the hyperspectral image is not restored. Therefore, the processing load is reduced.

[0177] 18, if the save button is in the ON state, the displayed RGB image is saved. Therefore, in the save mode, it is possible to save the RGB image valid for display together with the hyperspectral image.

[0178] FIG. 19 is a flowchart showing a sixth specific example of the operation of the image processing system 100 shown in FIG.

[0179] 14, in the example of Fig. 19, the processing circuitry 121 restores and displays a monochrome image (S204b) instead of restoring and displaying an RGB image (S204a). Specifically, at this time, the processing circuitry 121 restores and displays the monochrome image of the current frame from the compressed image of the current frame using mask data of one wavelength band corresponding to the monochrome image.

[0180] Furthermore, in conjunction with the restoration and display of the monochrome image (S204b), the processing circuitry 121 discards the monochrome image displayed in the previous frame (S203b and S209b) instead of discarding the RGB image displayed in the previous frame (S203a and S209a). At this time, the processing circuitry 121 may also discard the compressed image used in the previous frame.

[0181] Except for the above, the example in Fig. 19 is the same as the example in Fig. 14. In the example in Fig. 19, when the save button is in the ON state, the hyperspectral image is restored and saved. When the display button is in the ON state and the save button is in the OFF state, the RGB image is restored and displayed, but the hyperspectral image is not restored. Therefore, the processing load is reduced.

[0182] The restoration of a monochrome image is performed in a similar manner to the restoration of a hyperspectral image, except that instead of the mask data for four or more wavelength bands, mask data for one wavelength band corresponding to the monochrome image is used. Specifically, the mask data for four or more wavelength bands may be integrated, similar to the restoration of an RGB image.

[0183] In the restoration of RGB images, mask data for four or more wavelength bands is combined into mask data for three wavelength bands with fewer elements, while in the restoration of monochrome images, mask data for four or more wavelength bands is combined into mask data for one wavelength band with even fewer elements.

[0184] Alternatively, mask data for one wavelength band may be derived by deriving one transmittance for one wavelength band from the transmission spectrum for each pixel based on the above-described formula (3). The one wavelength band may correspond to the wavelength range W detectable by the image sensor 111.

[0185] By using a monochrome image, the amount of calculation is reduced compared to when an RGB image is used, and the processing load is further reduced.

[0186] If the save button is in the ON state (Yes in S205), the processing circuitry 121 may save the displayed monochrome image in addition to the hyperspectral image to the recording medium 140. This makes it possible to save the monochrome image valid for display together with the hyperspectral image in the save mode.

[0187] Fig. 20 is a flowchart showing a seventh specific example of the operation of the image processing system 100 shown in Fig. 1. In the example of Fig. 20, the method of generating a monochrome image is different from the example of Fig. 19.

[0188] Specifically, in the example of Fig. 20, the method of generating a monochrome image changes depending on whether the Save button is ON. Specifically, if the Save button is OFF (No in S205), a monochrome image is restored and displayed using mask data for one wavelength band (S204b). This process (S204b in Fig. 20) is the same as the process (S204b in Fig. 19) that is performed in the example of Fig. 19, regardless of whether the Save button is ON.

[0189] On the other hand, if the save button is ON (Yes in S205), the processing circuitry 121 restores the hyperspectral image, then generates and displays a monochrome image from the hyperspectral image (S207b). Specifically, the monochrome image is obtained by integrating multiple spectral images included in the hyperspectral image. In integrating multiple spectral images, multiple values ​​of the multiple spectral images may be added or averaged for each pixel.

[0190] Except for the above, the example in Fig. 20 is the same as the example in Fig. 19. In the example in Fig. 20, when the save button is in the ON state, the hyperspectral image is restored and saved. When the display button is in the ON state and the save button is in the OFF state, the monochrome image is restored and displayed, and the hyperspectral image is not restored. Therefore, the processing load is reduced.

[0191] 20, if the save button is ON, a monochrome image is generated from the restored hyperspectral image. The processing load of such a generation process is smaller than the processing load of a restoration process that restores a monochrome image from a compressed image. Therefore, the processing load is reduced by the above operation.

[0192] If the save button is in the ON state (Yes in S205), the processing circuitry 121 may save the displayed monochrome image in addition to the hyperspectral image to the recording medium 140. This makes it possible to save the monochrome image valid for display together with the hyperspectral image in the save mode.

[0193] FIG. 21 is a flowchart showing an eighth specific example of the operation of the image processing system 100 shown in FIG.

[0194] 14, in the example of Fig. 21, the processing circuitry 121 restores and displays a low-resolution RGB image (S204c) instead of restoring and displaying an RGB image (S204a). Specifically, at this time, the processing circuitry 121 restores and displays a low-resolution RGB image of the current frame from the compressed image of the current frame using low-resolution mask data of three wavelength bands corresponding to the RGB image.

[0195] Furthermore, in association with the restoration and display of the low-resolution RGB image (S204c), the processing circuitry 121 discards the low-resolution RGB image displayed in the previous frame (S203c and S209c) instead of discarding the RGB image displayed in the previous frame (S203a and S209a). At this time, the processing circuitry 121 may also discard the compressed image used in the previous frame.

[0196] Except for the above, the example in FIG. 21 is the same as the example in FIG. 14 . In the example in FIG. 21 , when the save button is in the ON state, the hyperspectral image is restored and saved. When the display button is in the ON state and the save button is in the OFF state, the low-resolution RGB image is restored and displayed, and the hyperspectral image is not restored. Therefore, the processing load is reduced. Furthermore, the data volume of the low-resolution RGB image is smaller than the data volume of the original RGB image. Therefore, the processing load is reduced.

[0197] If the save button is in the ON state (Yes in S205), the processing circuitry 121 may save the displayed low-resolution RGB image in addition to the hyperspectral image to the recording medium 140. This makes it possible to save the low-resolution RGB image effective for display together with the hyperspectral image in the save mode.

[0198] There are three methods for restoring a low-resolution RGB image: The first method is to restore the low-resolution RGB image by spatially averaging both the mask data and the compressed image and performing a restoration operation using compressed sensing.

[0199] Specifically, first, mask data for three wavelength bands corresponding to an RGB image is derived using the method described with reference to FIGS. 14 and 15 . Low-resolution mask data is obtained by spatially averaging the mask data. Similarly, a low-resolution compressed image is obtained by spatially averaging the compressed image. Then, a low-resolution RGB image is derived by performing a restoration operation for compressed sensing using the low-resolution mask data and the low-resolution compressed image.

[0200] The spatial averaging is performed by, for example, averaging four values ​​for each 2×2 pixel block into one value. Averaging is not limited to 2×2 pixel blocks, and may be performed in units of other blocks.

[0201] The second method restores the low-resolution RGB image by spatially subsampling both the mask data and the compressed image and performing a restoration operation using compressed sensing, i.e., thinning is performed instead of the averaging used in the first method.

[0202] Subsampling in the spatial direction is performed by, for example, extracting one value at the top left of the four values ​​for each 2x2 pixel block. Subsampling is not limited to 2x2 pixel blocks, and may be performed in units of other blocks. Furthermore, the value may be extracted at a different position, not limited to the top left.

[0203] The third method is a method of restoring a low-resolution RGB image by linear operations that are determined on the premise that a plurality of spatially adjacent pixels have the same pixel value spectrum. The third method will be specifically described below.

[0204] First, assuming that 2x2 pixels have the same pixel value spectrum, it is considered to restore three low-resolution spectral images of three wavelength bands corresponding to a low-resolution RGB image as a restored image. The relationship between data g representing the compressed image and data f representing the spectral image, i.e., the restored image, is expressed as g = Hf, as in Equation (1), where H is the system matrix described above. For simplicity of explanation, f is converted to the format expressed in Figure 22.

[0205] 22 is a diagram showing an example of data f representing a restored image of three wavelength bands, where f(a, b, c) represents the pixel value of the cth wavelength band at pixel position (a, b).

[0206] That is, in equation (1), the order used is the pixel value of the first pixel in the first wavelength band, the pixel value of the second pixel in the first wavelength band, ..., the pixel value of the first pixel in the second wavelength band, the pixel value of the second pixel in the second wavelength band, .... In contrast, in the example of Fig. 22, the order used is the pixel value of the first pixel in the first wavelength band, the pixel value of the first pixel in the second wavelength band, ..., the pixel value of the second pixel in the first wavelength band, the pixel value of the second pixel in the second wavelength band, ....

[0207] Fig. 23 is a diagram showing an example of matrix H corresponding to mask data of three wavelength bands. The column arrangement of matrix H is also converted in accordance with the conversion of data f. In Fig. 23, h(a, b, c) indicates the transmittance of the cth wavelength band at pixel position (a, b).

[0208] 24 is a diagram showing an example of data f representing a low-resolution restored image of three wavelength bands. In the example of FIG. 24, for example, pixel values ​​f(1,1,1), f(1,2,1), f(2,1,1), and f(2,2,1) have the same value according to the premise, so these four pixel values ​​can be aggregated into a single pixel value f(1,1,1). Similarly, by aggregating every four pixel values ​​into a single pixel value, the number of rows of data f is reduced to 1 / 4.

[0209] 25 is a diagram showing an example of matrix H corresponding to low-resolution mask data for three wavelength bands. As the number of rows of data f is reduced, the number of columns of matrix H is also reduced. Note that the number of rows of matrix H is maintained at the number of pixels in the compressed image. For example, aggregated pixel value f(1, 1, 1) affects four pixel values ​​included in data g. The leftmost column of matrix H shows this effect.

[0210] As a result of the above transformation, the number of rows in data f becomes smaller than the number of rows in data g. Therefore, the number of unknowns decreases, and data f can be derived by linear calculation. For example, f=H ＋ The data f can be derived by linear operation of g. Here, H ＋ denotes the pseudo-inverse matrix of the matrix H.

[0211] In the above, the data f of the three wavelength bands is derived on the assumption that 2×2 pixels have the same pixel value spectrum. However, the pixels having the same pixel value spectrum are not limited to 2×2 pixels, and the data f is not limited to data of the three wavelength bands.

[0212] For example, when deriving data f for N wavelength bands on the assumption that a×b pixels have the same pixel value spectrum, if a×b=N, then f=H －１ Data f can be derived by linear operation of g. If a×b>N, then f=H ＋ The data f can be derived by linear operation of g.

[0213] Fig. 26 is a flowchart showing a ninth specific example of the operation of the image processing system 100 shown in Fig. 1. In the example of Fig. 26, the method of generating a low-resolution RGB image is different from the example of Fig. 21.

[0214] Specifically, in the example of Fig. 26, the method for generating a low-resolution RGB image changes depending on whether the Save button is ON. Specifically, if the Save button is OFF (No in S205), the low-resolution RGB image is restored and displayed using low-resolution mask data for the three wavelength bands (S204c). This process (S204c in Fig. 26) is the same as the process (S204c in Fig. 20) that is performed in the example of Fig. 21 regardless of whether the Save button is ON.

[0215] On the other hand, if the save button is in the ON state (Yes in S205), the processing circuitry 121 restores the hyperspectral image, and then generates and displays a low-resolution RGB image from the hyperspectral image (S207c). Ｒ By integrating multiple spectral images of multiple wavelength bands included in Ｒ A low-resolution spectral image of

[0216] In integrating a plurality of spectral images, for example, a plurality of values ​​of the plurality of spectral images may be added together or averaged for each block of 2×2 pixels. The blocks are not limited to 2×2 pixel blocks, and integration may be performed in units of other blocks.

[0217] Similarly, among the multiple spectral images included in the hyperspectral image, the green wavelength band W Ｇ By integrating multiple spectral images of multiple wavelength bands included in the green wavelength band W Ｇ A low-resolution spectral image of the blue wavelength band W is obtained. Ｂ By integrating multiple spectral images of multiple wavelength bands included in Ｂ A low-resolution spectral image of

[0218] And the red wavelength band W Ｒ Low-resolution spectral image of the green wavelength band W Ｇ Low-resolution spectral image of the blue wavelength band W Ｂ A low-resolution RGB image is obtained by combining the low-resolution spectral image of

[0219] Except for the above, the example in Fig. 26 is the same as the example in Fig. 21. In the example in Fig. 26, when the save button is in the ON state, the hyperspectral image is restored and saved. When the display button is in the ON state and the save button is in the OFF state, the low-resolution RGB image is restored and displayed, and the hyperspectral image is not restored. Therefore, the processing load is reduced.

[0220] In the example of FIG. 26, if the save button is in the ON state, a low-resolution RGB image is generated from the restored hyperspectral image.

[0221] If the save button is in the ON state (Yes in S205), the processing circuitry 121 may save the displayed low-resolution RGB image in addition to the hyperspectral image to the recording medium 140. This makes it possible to save the low-resolution RGB image effective for display together with the hyperspectral image in the save mode.

[0222] 27 is a conceptual diagram showing a second display example of the display device 130 shown in FIG. 1. For example, in the viewing mode, the processing circuitry 121 displays an RGB image as a first image on the display device 130. In the viewing mode, the displayed RGB image is an RGB image restored from a compressed image. In the viewing mode, the hyperspectral image is not restored and is not displayed.

[0223] For example, the RGB image is updated based on the frame rate. That is, the RGB image corresponds to a moving image. The RGB image is checked by the user, and the position, orientation, angle of view, etc. of the image capturing device 110 are adjusted so that the subject to be captured is included in the hyperspectral image. After the adjustment, the mode is switched from the viewing mode to the storage mode.

[0224] 28 is a conceptual diagram showing a third display example of the display device 130 shown in FIG. 1. For example, in the storage mode, the processing circuitry 121 displays an RGB image and a hyperspectral image as a first image and a second image, respectively, on the display device 130. The processing circuitry 121 also displays the hyperspectral image in the form of a plurality of spectral images on the display device 130. In this example, the hyperspectral image includes 12 spectral images corresponding to the 12 wavelength bands, respectively.

[0225] In the storage mode, the displayed RGB image may be an RGB image restored from a compressed image or an RGB image generated from a hyperspectral image. For example, an RGB image generated by combining one or more spectral images corresponding to R, one or more spectral images corresponding to G, and one or more spectral images corresponding to B may be displayed.

[0226] After the mode is switched from the storage mode to the viewing mode, the display of the hyperspectral image last restored in the storage mode may continue in the viewing mode.

[0227] Fig. 29 is a conceptual diagram showing a fourth display example of the display device 130 shown in Fig. 1. For example, in the storage mode, the processing circuitry 121 displays an RGB image and an analysis result of the subject based on the hyperspectral image on the display device 130.

[0228] The analysis result of the subject based on the hyperspectral image may be an image showing the recognition result of the subject based on the hyperspectral image, or may be an image obtained by processing the RGB image and the hyperspectral image, or may be statistical information of the hyperspectral image and displayed in text format.

[0229] In the storage mode, the processing circuitry 121 may further display the hyperspectral image on the display device 130. That is, in the storage mode, the processing circuitry 121 may display the RGB image, the hyperspectral image, and the analysis result of the subject based on the hyperspectral image on the display device 130.

[0230] Alternatively, in the storage mode, the processing circuitry 121 may display the hyperspectral image, instead of the RGB image, on the display device 130. That is, in the storage mode, the processing circuitry 121 may display the hyperspectral image and the analysis result of the subject based on the hyperspectral image on the display device 130.

[0231] In addition, in the storage mode, the processing circuitry 121 may store the analysis results of the subject based on the hyperspectral image in addition to the hyperspectral image in the recording medium 140.

[0232] After the mode is switched from the storage mode to the viewing mode, the display of the analysis result last obtained in the storage mode may continue in the viewing mode.

[0233] FIG. 30 is a flowchart showing a tenth specific example of the operation of the image processing system 100 shown in FIG.

[0234] In this example, the processing circuitry 121 acquires a compressed image (S301). Specifically, the image sensor 111 acquires the compressed image by generating the compressed image, and the processing circuitry 121 acquires the compressed image from the image sensor 111.

[0235] Next, the processing circuitry 121 performs recognition processing on the compressed image (S302). As the recognition processing, the recognition processing described in Patent Document 3 may be used.

[0236] Specifically, for example, the processing circuitry 121 may apply preprocessing to the compressed image to improve recognition accuracy. The preprocessing may include region extraction, smoothing, feature extraction, edge detection, or any combination thereof. Then, the processing circuitry 121 may use a learning model to recognize objects included in the preprocessed compressed image. The learning model may be a deep learning model such as a convolutional neural network (CNN) or a recurrent neural network (RNN).

[0237] Then, the processing circuitry 121 determines whether or not a specific subject is included in the compressed image based on the result of the recognition process (S303). That is, the processing circuitry 121 determines whether or not a specific subject is captured in the compressed image.

[0238] If the compressed image contains a specific subject (Yes in S303), the processing circuitry 121 restores the hyperspectral image using mask data of four or more wavelength bands and stores the hyperspectral image in the recording medium 140 (S304).Then, the processing circuitry 121 generates an RGB image from the hyperspectral image and displays the RGB image on the display device 130 (S305).

[0239] If the compressed image does not contain a specific subject (No in S303), the processing circuitry 121 restores the RGB image using the mask data of the three wavelength bands and displays the RGB image on the display device 130 (S306).

[0240] Then, the processing circuitry 121 discards the displayed RGB image (S307). After that, the image processing system 100 ends the processing. The image processing system 100 may repeat the series of operations (S301 to S307).

[0241] In the above operation, the viewing mode and the storage mode are switched depending on whether or not the subject is included in the compressed image. That is, if the subject is included in the compressed image, the storage mode is used. If the subject is not included in the compressed image, the viewing mode is used. This reduces the processing load. Also, the recognition process is performed on the compressed image without restoring the hyperspectral image. This reduces the overall processing volume and processing time.

[0242] The image processing system 100 that performs the above-described recognition process may be applied to foreign object inspection in a factory. Specifically, the image processing system 100 may use a learning model to perform foreign object recognition processing on a compressed image. If a foreign object is captured in the compressed image, the image processing system 100 may restore and save a hyperspectral image from the compressed image. The image processing system 100 may then present the hyperspectral image to a user, or may remove the foreign object based on the hyperspectral image.

[0243] Furthermore, the image processing system 100 that performs the above-described recognition process may be applied to facial skin condition recognition. Specifically, the image processing system 100 may perform facial recognition processing on a compressed image using a learning model. If a face is captured in the compressed image, the image processing system 100 may restore and store a hyperspectral image from the compressed image. The image processing system 100 may then perform facial skin condition recognition based on the hyperspectral image.

[0244] Furthermore, the image processing system 100 that performs the above-described recognition process may be applied to recognition process on a conveyor belt. Specifically, the image processing system 100 may perform recognition process on a two-dimensional barcode placed near an object based on a compressed image. If a two-dimensional barcode is captured in the compressed image, the image processing system 100 may restore a hyperspectral image for processing the object from the compressed image.

[0245] FIG. 31 is a flowchart showing an eleventh specific example of the operation of the image processing system 100 shown in FIG.

[0246] In this example, the processing circuitry 121 acquires a compressed image (S301). This process is the same as the process in the example of FIG.

[0247] Next, the processing circuitry 121 restores the RGB image using the mask data of the three wavelength bands, and displays the RGB image on the display device 130 (S306). This processing is the same as the processing performed when the specific subject is not included in the compressed image in the example of Fig. 30.

[0248] Next, the processing circuitry 121 performs a recognition process on the RGB image (S302a). Then, based on the results of the recognition process, the processing circuitry 121 determines whether the RGB image contains a specific subject (S303a). In the example of Fig. 30, a compressed image is used, while in the example of Fig. 31, an RGB image is used. Except for the difference between a compressed image and an RGB image, these processes are the same as the processes in the example of Fig. 30.

[0249] If the compressed image contains a specific subject (Yes in S303a), the processing circuitry 121 restores the hyperspectral image using mask data of four or more wavelength bands and stores the hyperspectral image on the recording medium 140 (S304). This process is the same as the process in the example of Fig. 30. If the compressed image does not contain a specific subject (No in S303a), the hyperspectral image is not restored or stored.

[0250] Then, the processing circuitry 121 discards the displayed RGB image (S307). After that, the image processing system 100 ends the processing. The image processing system 100 may repeat the series of operations (S301 to S307).

[0251] In the example of Fig. 31, an RGB image is used for the recognition process. Therefore, color information can also be used for the recognition process. This makes it possible to perform more complex recognition processes with high accuracy.

[0252] Although the image processing system and the like have been described according to the embodiments, the aspects of the image processing system and the like are not limited to the embodiments. Modifications conceivable by those skilled in the art may be made to the embodiments, and multiple components in the embodiments may be combined in any manner.

[0253] For example, a process performed by a specific component in an embodiment may be performed by another component instead of the specific component. Furthermore, the order of multiple processes may be changed, or multiple processes may be performed in parallel. Furthermore, ordinal numbers such as "first" and "second" used in the description may be changed, removed, or newly assigned as appropriate. These ordinal numbers do not necessarily correspond to a meaningful order, and may be used to identify elements.

[0254] Also, for example, a phrase "at least one of a first element, a second element, and a third element" corresponds to the first element, the second element, the third element, or any combination thereof.

[0255] Furthermore, a method including steps performed by each component of an image processing system or the like may be executed by any system or device. In other words, the method may be executed by the image processing system or the like described above, or may be executed by another system or device.

[0256] For example, a part or all of the method may be executed by a computer including a processor, a memory, an input / output circuit, etc. In this case, the method may be executed by the computer executing a program for causing the computer to execute the method.

[0257] For example, the above program causes a computer to execute an image processing method including acquiring a compressed image, switching between a viewing mode and a storage mode, displaying, in the viewing mode, a first image on a display device, which is one of the compressed image and a display processing image generated based on the compressed image and expressed by information of three or less wavelength bands, and deleting the first image after displaying the first image, and generating, in the storage mode, a second image expressed by information of four or more wavelength bands based on the compressed image, and saving the second image on a recording medium.

[0258] The above program may also be recorded on a non-transitory computer-readable recording medium such as a CD-ROM.

[0259] Furthermore, each component of the image processing system may be configured with dedicated hardware, general-purpose hardware that executes the above-mentioned programs, or a combination of these. The general-purpose hardware may be configured with a memory in which the programs are recorded and a general-purpose processor that reads and executes the programs from the memory. Here, the memory may be a semiconductor memory or a hard disk, and the general-purpose processor may be a CPU.

[0260] Furthermore, the dedicated hardware may be configured with a memory and a dedicated processor, etc. For example, the dedicated processor may refer to the memory and execute the above-described method.

[0261] Furthermore, each component of the image processing system or the like may be an electric circuit. These electric circuits may form a single electric circuit as a whole, or may be separate electric circuits. These electric circuits may correspond to dedicated hardware, or may correspond to general-purpose hardware that executes the above-mentioned programs or the like.

[0262] (Others) Modifications of the embodiment of the present disclosure may be as follows.

[0263] (Variation a) An apparatus comprising: an image sensor including a first plurality of pixels; and a controller; after the image sensor receives first light from a filter array including four or more filters having different transmission spectra in wavelength ranges, the image sensor outputs a first plurality of pixel values ​​corresponding to the first plurality of pixels; after the image sensor receives second light from the filter array, the image sensor outputs a second plurality of pixel values ​​corresponding to the first plurality of pixels; the image sensor receives the second light after receiving the first light; the controller performs a first process before the controller receives an instruction; and the controller performs a second process after the controller receives the instruction; in the first process, the controller generates a third plurality of pixel values ​​corresponding to a first wavelength band, a fourth plurality of pixel values ​​corresponding to a second wavelength band, and a fifth plurality of pixel values ​​corresponding to a third wavelength band based on the first plurality of pixel values ​​and first information; and the total number of the third plurality of pixel values ​​is the same as the total number of the first plurality of pixel values. the total number of the fourth plurality of pixel values ​​is the same as the total number of the first plurality of pixel values; the total number of the fifth plurality of pixel values ​​is the same as the total number of the first plurality of pixel values; the controller generates a first image based on the third plurality of pixel values, the fourth plurality of pixel values, and the fifth plurality of pixel values ​​in the first processing; the controller does not generate four or more second images that correspond one-to-one to four or more wavelength bands based on the first plurality of pixel values ​​and second information in the first processing; the controller generates a sixth plurality of pixel values ​​corresponding to the first wavelength band, a seventh plurality of pixel values ​​corresponding to the second wavelength band, and an eighth plurality of pixel values ​​corresponding to the third wavelength band based on the second plurality of pixel values ​​and the first information in the second processing; the total number of the sixth plurality of pixel values ​​is the same as the total number of the first plurality of pixel values; the total number of the seventh plurality of pixel values ​​is the same as the total number of the first plurality of pixel values;the total number of the eighth plurality of pixel values ​​is the same as the total number of the first plurality of pixel values; the controller generates a third image based on the sixth plurality of pixel values, the seventh plurality of pixel values, and the eighth plurality of pixel values ​​in the second process; the controller generates four or more fourth images in one-to-one correspondence with the four or more wavelength bands based on the second plurality of pixel values ​​and the second information in the second process; the wavelength range is divided into the first wavelength band, the second wavelength band, and the third wavelength band; the wavelength range is divided into the four or more wavelength bands; the width of the first wavelength band is a first width, the width of the second wavelength band is a second width, and the width of the third wavelength band is a third width; and the bandwidth of each of the four or more wavelength bands is smaller than the first width, smaller than the second width, and smaller than the third width.

[0264] (Variation b) The device according to Variation a, further including a memory, wherein the first information and the second information are stored in the memory before the image sensor receives the first light.

[0265] (Variation c) The device of Variation a or Variation b, wherein the instructions indicate displaying four or more images on a display device that correspond one-to-one to the four or more wavelength bands.

[0266] (Description of Modification A) The first plurality of pixel values ​​may be a matrix g with n×m rows and 1 column expressed by equation (1). In this description, the first plurality of pixel values ​​will be described as a matrix g1 with n×m rows and 1 column.

[0267] The second plurality of pixel values ​​may be a matrix g having n×m rows and 1 column, as shown in Equation (1). In this description, the second plurality of pixel values ​​will be referred to as a matrix g2 having n×m rows and 1 column.

[0268] FIG. 32 is a diagram showing an example of the relationship between the first plurality of pixels included in the image sensor and the first plurality of pixel values, and the relationship between the plurality of pixels included in the image sensor and the second plurality of pixel values.

[0269] The first plurality of pixels may be pixel p(1,1), . . . , pixel p(n,m).

[0270] The first plurality of pixel values ​​may be a pixel value g1(1,1) output by pixel p(1,1), . . . , a pixel value g1(n,m) output by pixel p(n,m).

[0271] The second plurality of pixel values ​​may be a pixel value g2(1,1) output by pixel p(1,1), . . . , a pixel value g2(n,m) output by pixel p(n,m).

[0272] The wavelength range may be the wavelength range W shown in FIG.

[0273] The first information is a matrix H1=(H Ｂ H Ｇ H Ｒ ) may be used. Ｂ , H Ｇ , H Ｒ Each of H is a submatrix of H1.

[0274] The third plurality of pixel values ​​corresponding to the first wavelength band are wavelength band W Ｒ The matrix f of n × m rows and 1 column corresponding to １Ｒ the fourth plurality of pixel values ​​corresponding to the second wavelength band are included in wavelength band W Ｇ The matrix f of n × m rows and 1 column corresponding to １Ｇ the fifth plurality of pixel values ​​corresponding to the third wavelength band are included in the wavelength band W Ｂ The matrix f of n × m rows and 1 column corresponding to １Ｂ It may be expressed as n×m components contained in

[0275] Equation (1) is

[0276]

[0277] This can be expressed as:

[0278] The first image is f １Ｒ , f １Ｇ , f １Ｂ The first RGB image may be generated based on the

[0279] The second information is a matrix H2=(H １ H２ ...H ｗ ) where H1, H2, ..., Hw are each submatrices of H2.

[0280] The above four or more wavelength bands are the wavelength bands W shown in FIG. １ , ..., wavelength band W ｗ may be.

[0281] In the modification example a, "the controller does not generate four or more second images corresponding one-to-one to four or more wavelength bands based on the first plurality of pixel values ​​and the second information in the first processing" means that the controller does not generate four or more second images corresponding one-to-one to four or more wavelength bands based on the first plurality of pixel values ​​and the second information in the first processing. Ｔ and H2, four or more wavelength bands (i.e., wavelength bands W １ , ..., wavelength band W ｗ ) in one-to-one correspondence with the first image.

[0282]

[0283] Based on the equation (2) corresponding to the wavelength band W １ f corresponding to １１ , ..., wavelength band W ｗ f corresponding to １ｗ This makes it possible to reduce the amount of calculation of the controller.

[0284] The sixth plurality of pixel values ​​corresponding to the first wavelength band are Ｒ The matrix f of n × m rows and 1 column corresponding to ２Ｒ the seventh plurality of pixel values ​​corresponding to the second wavelength band are n×m components included in wavelength band W Ｇ The matrix f of n × m rows and 1 column corresponding to ２Ｇ the eighth plurality of pixel values ​​corresponding to the third wavelength band are included in wavelength band W Ｂ The matrix f of n × m rows and 1 column corresponding to ２Ｂ It may be expressed as n×m components contained in

[0285] Equation (1) is

[0286]

[0287] This can be expressed as:

[0288] The third image is f ２Ｒ , f ２Ｇ , f ２Ｂ The second RGB image may be generated based on the

[0289] In the modification example a, "the controller generates four or more fourth images corresponding one-to-one to the four or more wavelength bands based on the second plurality of pixel values ​​and the second information in the second processing" means that the controller generates four or more fourth images corresponding one-to-one to the four or more wavelength bands based on the second plurality of pixel values ​​and the second information in the second processing. Ｔ and H2, four or more wavelength bands (i.e., wavelength bands W １ , ..., wavelength band W ｗ ) to generate four or more fourth images in one-to-one correspondence with the first and second images.

[0290]

[0291] Based on the equation (2) corresponding to the wavelength band W １ f corresponding to ２１ , f ２２ , ..., wavelength band W ｗ f corresponding to ２ｗ Calculate.

[0292] The controller ２１ Based on the wavelength band W １ Image I corresponding to ２１ , ..., f ２ｗ Based on the wavelength band W ｗ Image I corresponding to ２ｗ Generate.

[0293] Figure 35 is Image I ２１ ,...,Image I ２ｗ FIG.

[0294] Image I ２１ is the pixel value f ２１ Pixel p21(1,1), ..., pixel value is f ２１(n, m), and so on, image I ２ｗ is the pixel value f ２ｗ Pixel p2w(1,1), ... pixel value is f ２ｗ (n, m) includes pixel p2w(n, m).

[0295] The present disclosure is applicable to an image processing method for restoring an image, and can be used in image processing systems, imaging systems, camera systems, analysis systems, recognition systems, and the like.

[0296] 100 Image processing system 110 Imaging device 111 Image sensor 112 Filter array 113 Optical system 120 Image processing device 121 Processing circuit 130 Display device 140 Recording medium

Claims

1. An image processing method comprising: acquiring a compressed image; switching between a viewing mode and a storage mode; in the viewing mode, displaying on a display device a first image which is one of the compressed image and a display processing image generated based on the compressed image and expressed by information of three or less wavelength bands, and deleting the first image after displaying the first image; in the storage mode, generating a second image based on the compressed image and expressed by information of four or more wavelength bands, and storing the second image on a recording medium.

2. The image processing method according to claim 1, wherein the first image is the compressed image.

3. The image processing method according to claim 1, wherein the first image is the display processing image.

4. The image processing method according to claim 3, wherein the first image is an RGB image represented by information of three wavelength bands.

5. The image processing method according to claim 3, wherein the first image is a monochrome image represented by information of one wavelength band.

6. The image processing method according to any one of claims 3 to 5, wherein the resolution of the first image is lower than the resolution of the second image.

7. An image processing method according to any one of claims 3 to 5, wherein: acquiring the compressed image involves acquiring the compressed image via a plurality of light receiving areas having a plurality of transmission spectra; generating the first image involves generating the first image based on the compressed image and first mask data including a plurality of values ​​reflecting the plurality of transmission spectra; generating the second image involves generating the second image based on the compressed image and second mask data including a plurality of values ​​reflecting the plurality of transmission spectra; and the plurality of values ​​included in the first mask data are less than the plurality of values ​​included in the second mask data.

8. The image processing method according to any one of claims 1 to 5, wherein the switching between the viewing mode and the storage mode is performed based on an operation performed by a user.

9. An image processing method according to any one of claims 1 to 5, further comprising: in the viewing mode, determining whether or not the first image includes a specific object; and, in switching between the viewing mode and the storage mode, continuing the viewing mode if it is determined that the first image does not include the specific object; and switching from the viewing mode to the storage mode if it is determined that the first image includes the specific object.

10. The image processing method according to any one of claims 1 to 5, further comprising the step of displaying the first image on the display device in the save mode.

11. The image processing method according to claim 10, further comprising the step of: in said save mode, saving said first image on said recording medium.

12. The image processing method according to claim 10, further comprising the steps of: displaying, in said storage mode, one or both of said second image and an analysis result of the subject based on said second image.

13. The image processing method according to any one of claims 3 to 5, further comprising, in the storage mode, generating the first image based on the second image generated based on the compressed image, and displaying the first image on the display device.

14. An image processing system comprising: an image sensor that acquires a compressed image; and a processing circuit that switches between a viewing mode and a storage mode, wherein the processing circuit, in the viewing mode, displays on a display device a first image which is one of the compressed image and a display processed image generated based on the compressed image and expressed by information of three or less wavelength bands, and deletes the first image after displaying the first image; and, in the storage mode, generates a second image based on the compressed image and expressed by information of four or more wavelength bands, and stores the second image on a recording medium.

15. A program for causing a computer to execute the image processing method according to any one of claims 1 to 5.

Images