Image processing method and image processing system

Abstract

The image processing method includes: acquiring a compressed image (S101); switching between a visual confirmation mode and a saving mode (S102); in the visual confirmation mode, displaying a first image on a display device (S104), and deleting the first image after displaying the first image (S105), the first image being one of the compressed image and a display-processed image, the display-processed image being an image generated based on the compressed image and represented by information of three or fewer wavebands; and in the saving mode, generating a second image represented by information of four or more wavebands based on the compressed image (S106), and saving the second image on a recording medium (S107).

Description

Technical Field

[0001] This disclosure relates to image processing methods, etc. Background Technology

[0002] A scheme for applying compressed sensing to hyperspectral cameras, i.e., restoring hyperspectral images from compressed images, is proposed. Patent documents 1-3 and non-patent documents 1 and 2 relate to this technology.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: International Publication No. 2022 / 202236

[0006] Patent Document 2: International Publication No. 2021 / 192891

[0007] Patent Document 3: International Publication No. 2020 / 080045

[0008] Non-patent literature

[0009]

[0010] Non-patent document 2: Ahasan Ahamed et. Al., “Reconstruction-based spectroscopy using CMOS image sensors with random photon-trappingnanostructure per sensor”, Proc. SPIE 11971, High-Speed ​​Biomedical Imaging and Spectroscopy VII, 1197106, 2 March 2022 Summary of the Invention

[0011] However, the computational demands for restoring images from compressed images, such as hyperspectral images, which contain information from more than four bands, are enormous. Therefore, the processing load is high, and the power and energy consumption are also significant.

[0012] Therefore, image processing methods are provided that can suppress the processing load associated with image restoration. This reduces the computational load on one or more computers executing the image processing method. In other words, it improves the functionality of the one or more computers executing the image processing method.

[0013] One technical solution disclosed herein relates to an image processing method comprising: acquiring a compressed image; switching between a visual confirmation mode and a save mode; in the visual confirmation mode, displaying a first image on a display device, and deleting the first image after displaying the first image, wherein the first image is one of the compressed image and a display-processed image, the display-processed image being an image generated based on the compressed image and representing information from three or fewer bands; and in the save mode, generating a second image based on the compressed image and representing information from four or more bands, and saving the second image on a recording medium.

[0014] Furthermore, these specific technical solutions can be implemented either through systems, devices, methods, integrated circuits, computer programs, or non-transient recording media such as CD-ROMs that can be read by computers, or through any combination of systems, devices, methods, integrated circuits, computer programs, and recording media.

[0015] The image processing method disclosed herein can suppress the processing load associated with image restoration. Attached Figure Description

[0016] Figure 1 This is a block diagram illustrating an example configuration of the image processing system in the implementation method.

[0017] Figure 2 This is a conceptual diagram illustrating an example of the configuration of the imaging device in the embodiment.

[0018] Figure 3 This is a conceptual diagram illustrating an example of the configuration of a filter array in an embodiment.

[0019] Figure 4 This is a graph showing an example of the transmission spectrum of the filter in the embodiment.

[0020] Figure 5 This is a graph representing an example of the transmission spectrum of another filter in the embodiment.

[0021] Figure 6 This is a schematic diagram illustrating an example of the transmittance of the first band of the filter array in the embodiment.

[0022] Figure 7 This is a schematic diagram illustrating an example of the transmittance of the second band of the filter array in the embodiment.

[0023] Figure 8 This is a flowchart illustrating an example of the operation of the image processing system in the implementation method.

[0024] Figure 9This is a conceptual diagram illustrating an example of the operation of the image processing system in the implementation method.

[0025] Figure 10 This is an explanatory diagram illustrating multiple combinations of the types of display object images and the types of saved object images in the implementation method.

[0026] Figure 11 This is a conceptual diagram representing the first display example in the implementation method.

[0027] Figure 12 This is an explanatory diagram showing the relationship between the display button, the save button, and the mode in the implementation method.

[0028] Figure 13 This is a flowchart illustrating the operation of the image processing system in the implementation method, specifically a first example.

[0029] Figure 14 This is a flowchart illustrating a second specific example of the operation of the image processing system in the implementation method.

[0030] Figure 15 This is an explanatory diagram showing the relationship between four or more bands and three bands in the implementation method.

[0031] Figure 16 This is a flowchart illustrating a third specific example of the operation of the image processing system in the implementation method.

[0032] Figure 17 This is a flowchart illustrating the operation of the image processing system in the implementation method, representing a fourth specific example.

[0033] Figure 18 This is a flowchart illustrating the operation of the image processing system in the implementation method, representing a fifth specific example.

[0034] Figure 19 This is a flowchart illustrating the operation of the image processing system in the implementation method, specifically a sixth example.

[0035] Figure 20 This is a flowchart illustrating the seventh specific example of the operation of the image processing system in the implementation method.

[0036] Figure 21 This is a flowchart illustrating the operation of the image processing system in the implementation method, representing the eighth specific example.

[0037] Figure 22 This is a diagram illustrating an example of data f representing the restored image of three bands in an implementation method.

[0038] Figure 23 This is a diagram illustrating an example of matrix H corresponding to the mask data for the three bands in the implementation method.

[0039] Figure 24 This is a diagram illustrating an example of data f representing a low-resolution restored image of three bands in an implementation method.

[0040] Figure 25 This is a diagram illustrating an example of matrix H corresponding to low-resolution mask data for three bands in an implementation scheme.

[0041] Figure 26 This is a flowchart illustrating the ninth specific example of the operation of the image processing system in the implementation method.

[0042] Figure 27 This is a conceptual diagram illustrating a second display example in the implementation method.

[0043] Figure 28 This is a conceptual diagram representing the third display example in the implementation method.

[0044] Figure 29 This is a conceptual diagram representing the fourth display example in the implementation method.

[0045] Figure 30 This is a flowchart illustrating the operation of the image processing system in the implementation method in the tenth specific example.

[0046] Figure 31 This is a flowchart illustrating the eleventh specific example of the operation of the image processing system in the implementation method.

[0047] Figure 32 This is a diagram illustrating an example of the relationship between the first plurality of pixels included in the image sensor and the values ​​of the first plurality of pixels, and the relationship between the plurality of pixels included in the image sensor and the values ​​of the second plurality of pixels.

[0048] Figure 33 It represents image I 21 Image I 2w An example diagram. Detailed Implementation

[0049] For example, an RGB camera detects light and generates an RGB image, which represents information from three bands corresponding to red, green, and blue. On the other hand, a hyperspectral camera detects light and generates a hyperspectral image, which represents information from four or more bands. Hyperspectral cameras and hyperspectral images are used in a wide variety of fields, including food inspection, biological examination, medical drug development, and mineral composition analysis.

[0050] Furthermore, from the perspectives of cost and flexibility, a scheme for applying compressed sensing to hyperspectral cameras, i.e., reconstructing hyperspectral images from compressed images, is proposed. Here, the compressed image is, for example, an image that overlaps information from four or more spectral bands. Hyperspectral images can be reconstructed from compressed images according to sparsity.

[0051] However, the computational demands for restoring images from compressed images, such as hyperspectral images, which contain information from more than four bands, are enormous. Therefore, the processing load is high, and the power and energy consumption are also significant.

[0052] Therefore, the image processing method of Example 1 includes: acquiring a compressed image; switching between a visual confirmation mode and a save mode; in the visual confirmation mode, displaying a first image on a display device, and deleting the first image after displaying the first image, wherein the first image is one of the compressed image and a display processing image, the display processing image being an image generated based on the compressed image and representing information from three or fewer bands; and in the save mode, generating a second image based on the compressed image and representing information from four or more bands, and saving the second image on a recording medium.

[0053] Therefore, the generation of a second image, which represents information from more than four bands, can be omitted without saving the data. This reduces processing load and power consumption.

[0054] Alternatively, the image processing method of Example 2 can also be the same as the image processing method of Example 1, where the first image is the compressed image.

[0055] This allows compressed images to be displayed directly in visual confirmation mode. Consequently, the processing load can be further reduced.

[0056] Alternatively, the image processing method in Example 3 can also be the same as the image processing method in Example 1, where the first image is the display processing image.

[0057] Therefore, a first image suitable for display can be generated with low computational load in visual confirmation mode. Thus, a first image suitable for display can be displayed while suppressing processing load.

[0058] Alternatively, the image processing method in Example 4 can also be the same as the image processing method in Example 3, where the first image is an RGB image represented by information from three bands.

[0059] Therefore, an RGB image can be generated as the first image in visual confirmation mode. Thus, a first image more suitable for display can be displayed.

[0060] Alternatively, the image processing method of Example 5 can also be used in the image processing method of Example 3, wherein the first image is a monochrome image represented by information from one band.

[0061] Therefore, a monochrome image can be generated as the first image in visual confirmation mode. This further reduces the processing load.

[0062] Alternatively, in any of the image processing methods in Examples 3 to 5, the resolution of the first image is lower than the resolution of the second image.

[0063] This allows for the generation of a low-resolution first image in visual confirmation mode. Consequently, the processing load can be further reduced.

[0064] Alternatively, the image processing method of Example 7 can also be implemented in any of the image processing methods of Examples 3 to 6, wherein in obtaining the compressed image, the compressed image is obtained via multiple light-receiving regions having multiple transmission spectra; in generating the first image, the first image is generated based on the compressed image and first mask data including multiple values ​​reflecting the multiple transmission spectra; and in generating the second image, the second image is generated based on the compressed image and second mask data including multiple values ​​reflecting the multiple transmission spectra, wherein the multiple values ​​included in the first mask data are fewer than the multiple values ​​included in the second mask data.

[0065] Therefore, in visual confirmation mode, a first image can be generated using first mask data, which includes fewer values ​​than the second mask data used in the generation of the second image. This further reduces processing load.

[0066] Furthermore, in any of the image processing methods in Examples 1 to 7, the image processing method in Example 8 switches between the visual confirmation mode and the save mode based on the user's operation during the switching between the visual confirmation mode and the save mode.

[0067] Therefore, it is possible to adaptively switch between visual verification mode and save mode. As a result, it is possible to adaptively suppress processing load.

[0068] In addition, the image processing method of Example 9 may also be in any of the image processing methods of Examples 1 to 7, further comprising: in the visual confirmation mode, determining whether the first image includes a specific subject; in the switching between the visual confirmation mode and the save mode, if it is determined that the first image does not include the specific subject, continuing the visual confirmation mode; and if it is determined that the first image includes the specific subject, switching from the visual confirmation mode to the save mode.

[0069] Therefore, the visual confirmation mode and the save mode can be switched depending on whether the first image includes a specific subject. Furthermore, if the first image includes a specific subject, a second image including that specific subject can be saved. Thus, a second image including a specific subject can be saved efficiently.

[0070] Furthermore, the image processing method of Example 10 may also be further described in any of the image processing methods of Examples 1 to 9, wherein the first image is displayed on the display device in the save mode.

[0071] Therefore, the first image can be displayed in both visual confirmation mode and save mode. Thus, it is possible to continue displaying the first image from visual confirmation mode into save mode as well.

[0072] Furthermore, the image processing method of Example 11 can also be further improved in the image processing method of Example 10 by saving the first image to the recording medium in the saving mode.

[0073] Therefore, the first image displayed can be saved in save mode. Thus, the first image and the second image displayed can be saved together in save mode.

[0074] Furthermore, the image processing method of Example 12 may also be further described in the image processing method of Example 10 or 11, in the saving mode, by displaying one or both of the second image and the analysis results of the photographed object based on the second image.

[0075] Therefore, it is possible to display the information corresponding to the second image in save mode. Thus, it is possible to display the first image and the information corresponding to the second image in save mode.

[0076] Furthermore, in any of the image processing methods in Examples 3 to 6, the image processing method of Example 13 may further include, in the storage mode, generating the first image based on the second image generated from the compressed image, and displaying the first image on the display device.

[0077] Therefore, the first image can be displayed in both visual confirmation mode and save mode. Thus, the first image can continue to be displayed in save mode from visual confirmation mode. Furthermore, the first image can be efficiently generated based on the second image in save mode. Therefore, the increase in processing load can be suppressed in save mode.

[0078] Additionally, the image processing system of Example 14 includes: an image sensor that acquires a compressed image; and a processing circuit that switches between a visual confirmation mode and a storage mode. In the visual confirmation mode, the processing circuit displays a first image on a display device and deletes the first image after displaying it. The first image is one of the compressed image and a display processing image, the display processing image being generated based on the compressed image and representing information from three or fewer bands. In the storage mode, a second image representing information from four or more bands is generated based on the compressed image, and the second image is stored on a recording medium.

[0079] Therefore, the generation of a second image, which represents information from more than four bands, can be omitted without saving the data. This reduces processing load and power consumption.

[0080] In addition, the program in Example 15 is a program used to make the computer execute any of the image processing methods in Examples 1 to 13.

[0081] Therefore, it can be implemented as a program for causing a computer to perform the above-described image processing method.

[0082] Furthermore, these specific or concrete technical solutions can be implemented either through systems, devices, methods, integrated circuits, computer programs, or non-transient recording media such as CD-ROMs that can be read by computers, or through any combination of systems, devices, methods, integrated circuits, computer programs, and recording media.

[0083] The embodiments will now be described using the accompanying drawings. Furthermore, the embodiments described below are either inclusive or specific examples. The numerical values, shapes, materials, constituent elements, the arrangement of constituent elements, connection methods, steps, and the order of steps shown in the following embodiments are merely examples and are not intended to limit the claims.

[0084] Figure 1 This is a block diagram illustrating an example configuration of the image processing system in the implementation embodiment. For example... Figure 1 As shown, the image processing system 100 includes an image sensor 111 and a processing circuit 121. Additionally, the image processing system 100 may also include a display device 130 and a recording medium 140. Furthermore, the image processing system 100 may also include an imaging device 110 and an image processing device 120. The imaging device 110 may also include the image sensor 111. The image processing device 120 may also include the processing circuit 121.

[0085] Image sensor 111 acquires an image by detecting light signals at each pixel and generating an image represented by multiple light signals corresponding to multiple pixels. Here, image sensor 111 acquires a compressed image. Specifically, for example, image sensor 111 acquires a compressed image based on a filter array described later, in which information of four or more bands are superimposed at each pixel. Furthermore, bands are sometimes represented only as bands.

[0086] Image sensor 111 can also be a monochromatic photodetector having multiple photodetector elements arranged in a matrix. More specifically, image sensor 111 can also be a CCD (Charge-Coupled Device) image sensor, a CMOS (Complementary Metal Oxide Semiconductor) image sensor, an infrared array image sensor, a terahertz array image sensor, or a millimeter-wave array image sensor.

[0087] Alternatively, the image sensor 111 can also be a color photodetector. The wavelength range that can be detected by the image sensor 111 is not limited, and can be, for example, visible light, ultraviolet light, infrared light, terahertz waves, or any combination thereof.

[0088] Processing circuit 121 is a circuit that performs information processing. For example, processing circuit 121 generates a hyperspectral image representing information from four or more bands by performing a restoration operation on a compressed image. For example, the hyperspectral image consists of four or more spectroscopic images corresponding to four or more bands. The restoration operation can be the same as that described in Patent Documents 2 and 3. Specifically, four or more spectroscopic images can be generated as a hyperspectral image based on the following equation (1).

[0089]

[0090] Here, g represents the data of the compressed image, for example, represented by a one-dimensional arrangement (i.e., a vector). If the compressed image is an n×m pixel image, then the data g is represented by a one-dimensional arrangement with n×m elements. F represents the data of w spectroscopic images that correspond one-to-one with w bands, for example, represented by a one-dimensional arrangement. In the case where w spectroscopic images constitute a hyperspectral image, w is an integer greater than or equal to 4.

[0091] f1 represents the data of the spectroscopic image corresponding to band W1, f2 represents the data of the spectroscopic image corresponding to band W2, ..., f w To be compatible with band W w The corresponding spectroscopic image data.

[0092] Data f1, f2, ..., f wFor example, they can be represented by a one-dimensional arrangement. If each spectroscopic image is an n×m pixel image, then the data f1, f2, ..., f... w Each element is represented by a one-dimensional permutation of n×m elements, and the data f is represented by a one-dimensional permutation of n×m×w elements. H is an n×m row, n×m×w column matrix, called the system matrix, which corresponds to the mask data.

[0093] Matrix H can also be based on the transmission spectra of band W1, band W2, ..., band W of the filter array described later. w It is determined by the transmission spectrum.

[0094] The data f that satisfies equation (1) can be estimated using the method of compressed sensing, specifically, it can be estimated using equation (2).

[0095]

[0096] Equation (2) represents finding the data f that minimizes the sum of the first and second terms within the parentheses. By using recursive iterative operations to converge the data of the calculation results, the data f can be calculated as the final calculation result.

[0097] The first term within the parentheses in equation (2) represents the sum of squares of the differences between the data Hf obtained by transforming the data f in the estimation process through matrix H and the data g, which is the so-called residual term. Here, the sum of squares is used, but the sum of absolute values ​​or the square root of the sum of squares can also be used instead.

[0098] The sum of squares of the differences between data Hf and data g is (g1 - r1) × (g1 - r1) + ... + (g n×m -r n×m )×(g n×m -r n×m Here, g1, ..., g n×m For g = (g1...g n×m ) T The elements representing the data g. Additionally, r1, ..., r n×m For Hf = (r1...r n×m ) T The data Hf represents the elements.

[0099] The second term within the parentheses in equation (2) is the regularization term, sometimes also called the stabilization term. Φ(f) represents the constraint condition in the regularization of f, and is a function that reflects the sparse information of the data f. This function has the effect of smoothing or stabilizing the data f. Φ(f) can be represented, for example, by the Discrete Cosine Transform (DCT), Wavelet Transform, Fourier Transform, Total Variation (TV), or any combination thereof.

[0100] τ is the weight coefficient of the regularization term, corresponding to the influence of regularization in the restoration operation. The larger the value of τ, the higher the influence of regularization, and the stronger the convergence of the solution in the iterative operation. Conversely, the smaller the value of τ, the lower the influence of regularization, and the weaker the convergence of the solution in the iterative operation.

[0101] Display device 130 is a display used to display information. Compressed images, monochrome images, RGB images, or hyperspectral images are displayed on display device 130 via processing circuit 121. Display device 130 can be, for example, a liquid crystal display (LCD) or an organic light-emitting diode (OLED) display.

[0102] A graphical user interface (GUI) can also be displayed on the display device 130. Alternatively, the display device 130 can also be a touch panel. Furthermore, information can be input by the user on the display device 130. The processing circuit 121 can also obtain information from the user via the display device 130. That is, the display device 130 can also be an input / output device. Alternatively, the processing circuit 121 can also obtain information from the user via an input device different from the display device 130.

[0103] Recording medium 140 is a storage element used to store information. Compressed images, monochrome images, RGB images, or hyperspectral images are stored in recording medium 140 via processing circuitry 121. Recording medium 140 can be installed, for example, by a CD-ROM, hard disk drive, or solid-state hard disk drive.

[0104] also, Figure 1 An example of the configuration of the image processing system 100 is shown, but the configuration of the image processing system 100 is not limited to... Figure 1 Examples of configurations include: multiple devices can be integrated into one device, or one device can be distributed into multiple devices. Furthermore, the distributed devices can communicate with each other via wired or wireless communication.

[0105] The processing circuit 121 corresponds to an image acquisition unit that acquires images from the image sensor 111, a mode switcher, an image generator, a discard controller for discarding images, a display controller for displaying images on the display device 130, and a storage controller for storing images on the recording medium 140. Alternatively, the image processing apparatus 120 may not be the processing circuit 121, but may possess some or all of these components.

[0106] Figure 2 It means Figure 1The diagram shows a conceptual example of the configuration of the imaging device 110. The imaging device 110 may have the same configuration as the imaging devices disclosed in Patent Documents 2 and 3. For example, the imaging device 110 includes an image sensor 111, a filter array 112, and an optical system 113.

[0107] The filter array 112 is disposed in the optical path of light incident from the subject, between the optical system 113 and the image sensor 111. The filter array 112 functions as the coding element described in Patent Document 2. The filter array 112 can also be integrated with the image sensor 111.

[0108] Furthermore, the configuration of the filter array 112 is not limited to Figure 2 The filter array 112 can be configured, for example, between the optical system 113 and the image sensor 111, away from the image sensor 111. Alternatively, the filter array 112 can be configured between the subject and the optical system 113. Alternatively, the filter array 112 can be configured within the optical system 113.

[0109] The optical system 113 is disposed in the optical path of light incident from the subject and is positioned between the subject and the filter array 112. The optical system 113 includes at least one lens capable of forming an image of the subject on the imaging surface of the image sensor 111 via the filter array 112.

[0110] Furthermore, the configuration and arrangement of the optical system 113 are not limited to Figure 2 The optical system 113 can be configured and arranged in various ways. For example, it can be positioned between the filter array 112 and the image sensor 111. Alternatively, the optical system 113 can also include multiple lenses arranged in the optical path. In this case, the filter array 112 can also be positioned between adjacent lenses of the multiple lenses.

[0111] Figure 3 It means Figure 2 The diagram shows a conceptual example of the configuration of the filter array 112. The filter array 112 includes multiple filters F arranged in a matrix. 11 ... F nm .exist Figure 3 In the example shown, the filter array 112 includes 48 filters F arranged in 6 rows and 8 columns. 11 ... F nm Filter F 11 It has 48 filters F 11 ... F nm The filter configured in the upper left corner, filter F nm It has 48 filters F 11 ... F nmThe filter configuration is located in the lower right corner.

[0112] In addition, the filter array 112 includes filter F 11 ... F nm The number is not limited to 48. For example, the filter array 112 includes filter F. 11 ... F nm The number of filters can also be similar to the number of pixels in the image sensor 111, for example, it can range from tens to tens of millions, depending on the application. The filter array 112 includes filters F... 11 ... F nm The number of pixels can be the same as or different from the number of pixels in image sensor 111. The multiple pixels included in image sensor 111 can also be related to the filter F. 11 ... F nm One-to-one correspondence.

[0113] For example, filter array 112 includes n×m filters F corresponding to n×m pixels. 11 ... F nm In bands W1, ..., W w In the middle, filter F 11 ... F nm They each have a transmission spectrum S 11 ... S nm Transmission spectrum S 11 ... S nm They can also be completely different. Or, the transmission spectrum S 11 ... S nm The included transmission spectra can also be the same. Here, transmission spectrum can mean light transmittance spectrum.

[0114] n×m filters F 11 ... F nm For w bands W1, ..., W w Having n×m×w transmittance S 11W1 ... S nmWw Transmittance S 11W1 ... S nmWw They can all be different. Or, the transmittance S 11W1 ... S nmWw The included transmittance values ​​can also be the same. Here, transmittance can mean light transmittance.

[0115] Band W α The transmittance of filter β in the filter can also be given by the following formula (3).

[0116]

[0117] Here, Wαmin represents the band W. α The minimum wavelength value, Wαmax is the band W α The maximum wavelength value, h(λ) is a function representing the transmission spectrum, and λ is the wavelength.

[0118] In addition, band W α The transmittance of filter β in the image is not limited to equation (3). For example, in band W... α The transmittance of filter β in the formula can also be obtained by dividing equation (3) by (Wαmax - Wαmin). Additionally, for example, in band W... α The transmittance of filter β in the image can also be represented by the wavelength band W. α wavelength λ α0 The transmittance h (λ) at the following conditions α0 ).

[0119] Here, wavelength λ α0 To satisfy Wαmin≤λ α0 Wavelengths ≤ Wαmax. For example, wavelength λ. α0 It can also be band W. α The center wavelength ((Wαmax-Wαmin) / 2).

[0120] Figure 4 yes Figure 3 The filter F shown 11 Examples of transmission spectra. Figure 5 yes Figure 3 The filter F shown nm Examples of transmission spectra. Figure 6 It means Figure 3 A graph showing an example of the transmittance of band W1 of the filter array 112 shown. Figure 7 It means Figure 3 A graph showing an example of the transmittance of band W2 of the filter array 112 shown. Figure 6 and Figure 7 In the diagram, the density of each region represents the transmittance of the filter; the lighter the region, the higher the transmittance, and the darker the region, the lower the transmittance.

[0121] The filter array 112 includes multiple filters F 11 ... F nm The wavelength dependence of transmittance varies from one another. For example, in filter F... 11 In the meantime, the transmittance of band W1 is considerably lower than that of band W2. On the other hand, in filter F... nm In the middle band, the transmittance of band W1 is approximately the same as that of band W2. That is, the transmittance of filter F... 11The wavelength dependence of transmittance on filter F nm The wavelength dependence of transmittance differs between the two.

[0122] Furthermore, here are the w bands W1, ..., W w The transmittance of two bands W1 and W2 in the image is illustrated and explained. The transmittance of the w bands W1, ..., W2 is also explained. w Transmittance figures and explanations for other bands are omitted.

[0123] Additionally, for example, mask data consists of w bands W1, W2, ..., W w The corresponding w matrices represent this. Specifically, the mask data represents w bands W1, W2, ..., W... w Each band is represented by a transmittance matrix (transmittance matrix) corresponding to n rows and m columns of pixels. Furthermore, by changing the representation of mask data represented by w matrices with n rows and m columns respectively, a matrix H with n×m rows and n×m×w columns is obtained.

[0124] The H included in equation (1) can also be expressed as H = (H1H2……H) w H1, H2, ..., H w These are the smaller matrices of H. That is, when the multiple components included in H are set as a(11), ..., a((n×m)(n×m×w)), then...

[0125] .

[0126] H1 can also be interpreted as the mask data corresponding to band W1, and H2 can also be interpreted as the mask data for band W2. w This can also be interpreted as for band W w Mask data.

[0127] H1, H2, ..., H w Each can also be a diagonal matrix. That is, H1, H2, ..., H w It can also be

[0128] .

[0129] Figure 8 It means Figure 1The flowchart illustrates an example of the operation of the image processing system 100. In this example, a compressed image is obtained by generating a compressed image through the image sensor 111 of the imaging device 110, and the processing circuit 121 of the image processing device 120 obtains the compressed image from the image sensor 111 (S101). That is, the action of obtaining the compressed image can correspond to either the action of generating a compressed image through the image sensor 111 or the action of the processing circuit 121 obtaining the compressed image from the image sensor 111.

[0130] Then, the processing circuit 121 switches between the visual confirmation mode and the save mode (S102). Here, the processing circuit 121 can switch from the visual confirmation mode to the save mode, or from the save mode to the visual confirmation mode, or it can maintain both the visual confirmation mode and the save mode.

[0131] In visual confirmation mode (“visual confirmation mode” in S103), processing circuit 121 displays the first image on display device 130 (S104). Here, the first image may also be a compressed image. Alternatively, the first image may also be a display processing image generated based on a compressed image and represented by information from three or fewer bands. Then, processing circuit 121 deletes the first image after displaying it (S105).

[0132] In save mode ("save mode" in S103), processing circuit 121 generates a second image based on the compressed image (S106). The second image is an image represented by information from four or more bands. Then, processing circuit 121 saves the second image to recording medium 140 (S107).

[0133] Therefore, without saving the data, the generation of a second image representing information from four or more bands can be omitted. This reduces processing load and power consumption.

[0134] For example, the above series of actions (S101 to S107) can be performed repeatedly. In visual confirmation mode, multiple first images can be displayed as moving images, and in save mode, multiple second images can be saved as moving images.

[0135] Furthermore, for example, if the first image is a compressed image, the compressed image can be directly displayed in visual confirmation mode. Therefore, the processing load can be further reduced.

[0136] Furthermore, for example, if the first image is a display processing image, the processing circuit 121 can also generate the first image based on the compressed image in visual confirmation mode. This allows for the generation of a display-suitable first image with low computational load in visual confirmation mode. Therefore, a display-suitable first image can be displayed while suppressing processing load.

[0137] Alternatively, the first image could be an RGB image representing information from three bands. Therefore, an RGB image can be generated as the first image in visual confirmation mode. Consequently, a first image more suitable for display can be displayed.

[0138] Alternatively, the first image could be a monochrome image representing information from a single band. Therefore, a monochrome image can be generated as the first image in visual confirmation mode. This further reduces processing overhead.

[0139] Furthermore, the resolution of the first image can be lower than that of the second image. This allows for the generation of a low-resolution first image in visual verification mode, thus further reducing processing load.

[0140] Alternatively, for example, image sensor 111 can also acquire compressed images via multiple light-receiving areas having multiple transmission spectra. Specifically, image sensor 111 can also acquire compressed images via filter array 112. The multiple light-receiving areas can each correspond to a specific filter of the multiple filters included in filter array 112.

[0141] Alternatively, a superlens as described in Non-Patent Document 1 may be used instead of the filter array 112. In this case, the wavelength transmittance varies depending on the position. The light-receiving area corresponds to such a position. Alternatively, a CMOS image sensor as described in Non-Patent Document 2 may be used instead of the filter array 112. In this case, the sensing area is processed to obtain a predetermined transmittance per pixel. The light-receiving area corresponds to such a sensing area.

[0142] The light-receiving area can also be represented as an optical element, a modulation element, an encoding element, an optical processing area, or a shielding area.

[0143] Furthermore, the processing circuit 121 can also generate a first image based on the compressed image and the first mask data. Here, the first mask data includes multiple values ​​reflecting multiple transmission spectra. Additionally, the processing circuit 121 can also generate a second image based on the compressed image and the second mask data. Here, the second mask data includes multiple values ​​reflecting multiple transmission spectra. Furthermore, the multiple values ​​included in the first mask data are fewer than the multiple values ​​included in the second mask data.

[0144] Therefore, in visual confirmation mode, a first image can be generated using first mask data that includes fewer values ​​than the second mask data used in the generation of the second image. This further reduces processing load.

[0145] Alternatively, the first mask data can be generated by integrating the matrix elements of the second mask data, for example. This reduces the number of values ​​included in the first mask data. Furthermore, both the first mask data for generating the first image and the second mask data for generating the second image can be efficiently generated from common multiple transmission spectra. Therefore, the number of calibrations can be reduced.

[0146] Furthermore, the processing circuit 121 can switch between visual confirmation mode and save mode based on user operations. This allows for adaptive switching between visual confirmation mode and save mode, thus adaptively suppressing processing load.

[0147] Additionally, for example, the processing circuit 121 may further determine whether the first image includes a specific subject in the visual confirmation mode. Here, if it is determined that the first image does not include a specific subject, the processing circuit 121 may continue with the visual confirmation mode. On the other hand, if it is determined that the first image includes a specific subject, the processing circuit 121 may switch from the visual confirmation mode to the save mode.

[0148] Therefore, the visual confirmation mode and the save mode can be switched depending on whether the first image includes a specific subject. Furthermore, if the first image includes a specific subject, a second image including that specific subject can be saved. Thus, a second image including a specific subject can be saved efficiently.

[0149] Alternatively, for example, the processing circuit 121 can also display the first image on the display device 130 in save mode. Thus, the first image can be displayed in both visual confirmation mode and save mode. Therefore, it is possible to continue displaying the first image from visual confirmation mode into save mode as well.

[0150] Alternatively, for example, the processing circuit 121 can also save the first image to the recording medium 140 in save mode. Thus, the displayed first image can be saved in save mode. Therefore, it is possible to save the displayed first image together with the second image in save mode.

[0151] Alternatively, for example, the processing circuit 121 can also display one or both of the second image and the analysis results of the photographed subject based on the second image in the save mode. Thus, information corresponding to the second image can be displayed in the save mode. Therefore, the first image and information corresponding to the second image can be displayed in the save mode.

[0152] Alternatively, for example, the processing circuit 121 can also generate a first image based on a second image generated from a compressed image in save mode. Furthermore, the processing circuit 121 can also display the first image on the display device 130.

[0153] Therefore, the first image can be displayed in both visual confirmation mode and save mode. Thus, the first image can be displayed continuously from visual confirmation mode into save mode. Furthermore, the first image can be efficiently generated based on the second image in save mode. Therefore, the increase in processing load can be suppressed in save mode.

[0154] Figure 9 It means Figure 1 The diagram illustrates a conceptual example of the operation of the image processing system 100. For example, the image sensor 111 repeatedly acquires compressed images by repeatedly taking pictures (photographing) based on the frame rate.

[0155] The processing circuit 121 switches between visual confirmation mode and save mode by selecting a mode on a per-frame basis. Visual confirmation mode, also known as display mode, viewing mode, or shooting (photography) mode, is a mode in which images are not saved. Save mode, also known as video recording mode, is a mode in which images are saved.

[0156] In visual confirmation mode, processing circuit 121 repeatedly performs the following operations: acquiring a compressed image via image sensor 111, reconstructing a first image from the compressed image, displaying the first image on display device 130, and discarding the first image. The first image can be an RGB image or a monochrome image. Alternatively, the first image can be the compressed image itself. In this case, processing circuit 121 may also not perform the reconstruction process in visual confirmation mode.

[0157] In save mode, the processing circuit 121 repeatedly performs the following operations: acquiring a compressed image via the image sensor 111, reconstructing a second image from the compressed image, and saving the second image to the recording medium 140. The second image may be a hyperspectral image. Furthermore, in save mode, the processing circuit 121 may also repeatedly perform the following operations: generating a first image from the second image, displaying the first image on the display device 130, and discarding the first image.

[0158] Alternatively, in save mode, processing circuit 121 can restore the first image from the compressed image in the same way as in visual confirmation mode. If the first image is a compressed image, the generation process (restoration process) of the first image may not be performed. Alternatively, in save mode, processing circuit 121 may display the second image instead of the first image.

[0159] In addition, in the save mode, the processing circuit 121 can also save the first image displayed on the display device 130 to the recording medium 140 based on the second image. That is, the processing circuit 121 can also save the first image to the recording medium 140 without discarding it.

[0160] Figure 10 It means Figure 9 The diagram illustrates multiple combinations of display object image types and save object image types in the action example shown.

[0161] Specifically, the displayed object image is the first image. The saved object image can be either the second image or a combination of the first and second images. The first image can be, for example, a compressed image, a monochrome image, an RGB image, a low-resolution monochrome image, or a low-resolution RGB image. The second image can be, for example, a hyperspectral image.

[0162] Alternatively, when the object image to be displayed is a monochrome image, an RGB image, a low-resolution monochrome image, or a low-resolution RGB image, a compressed image can be included in the saved object image. That is, in this case, the saved object image can be either a hyperspectral image and a compressed image, or a hyperspectral image, the object image to be displayed, and a compressed image.

[0163] In any of these multiple examples, no image is saved if the hyperspectral image is not saved, that is, if the mode is not the save mode.

[0164] Figure 11 It means Figure 1 This is a conceptual diagram of a first display example of the display device 130. For example, the display device 130 displays... Figure 11 The GUI shown includes a first image, a display button, and a save button. The first image included in the GUI is a first image generated (restored) by the processing circuit 121. The display button and the save button are toggled between ON and OFF states by user operation.

[0165] The visual confirmation mode and save mode are switched by toggling the ON and OFF states of the display and save buttons. The display button can also be a shooting button or a visual confirmation button. The save button can also be a recording button.

[0166] Figure 12 This is an explanatory diagram illustrating the relationship between the display button, save button, and mode in the implementation method. When the display button is in the OFF state, the save button is also in the OFF state, and the mode is in the unselected state. In this case, no image is generated or displayed.

[0167] With the display button in the ON state, the save button can be toggled between ON and OFF. When the display button is ON and the save button is OFF, the mode is visual confirmation mode. When both the display button and the save button are ON, the mode is save mode.

[0168] For example, if the display button is in the ON state, the first image is displayed; furthermore, if the save button is in the ON state, the second image is saved.

[0169] Figure 13 It means Figure 1 A flowchart of a first specific example of the operation of the image processing system 100 shown.

[0170] In this example, the processing circuit 121 determines whether the display button is in the ON state (S201).

[0171] When the display button is in the ON state ("Yes" in S201), the processing circuit 121 acquires the compressed image (S202). Specifically, the compressed image is acquired by generating a compressed image using the image sensor 111, and the processing circuit 121 acquires the compressed image from the image sensor 111. Then, the processing circuit 121 discards the compressed image displayed in the previous frame (S203). Then, the processing circuit 121 displays the compressed image acquired in the current frame (S204).

[0172] Further, the processing circuit 121 determines whether the save button is in the ON state (S205). If the save button is in the ON state ("Yes" in S205), the hyperspectral image is restored and saved using mask data from four or more bands (S206). Then, a series of processes are repeated (S201 to S206). On the other hand, if the save button is in the OFF state ("No" in S205), the hyperspectral image is not restored, and the processing is repeated (S201 to S205).

[0173] When the display button is in the OFF state ("No" in S201), the processing circuit 121 discards the compressed image displayed in the previous frame (S209). Then, the image processing system 100 ends its operation.

[0174] exist Figure 13In the example, when the save button is ON, the hyperspectral image is restored and saved. Furthermore, if the display button is ON and the save button is OFF, a compressed image is displayed, and the hyperspectral image is not restored. Therefore, the processing load can be reduced.

[0175] Figure 14 It means Figure 1 A flowchart of a second specific example of the operation of the image processing system 100 shown.

[0176] and Figure 13 Compared to the examples, in Figure 14 In the example, the processing circuit 121 replaces the display of the compressed image (S204), and uses the mask data of the three bands corresponding to the RGB image to restore the RGB image of the current frame from the compressed image of the current frame and display it (S204a).

[0177] Furthermore, along with the restoration and display of the RGB image (S204a), the processing circuit 121 discards the compressed image displayed in the previous frame (S203 and S209) instead of the RGB image displayed in the previous frame (S203a and S209a). Additionally, the processing circuit 121 can also discard the compressed image used in the previous frame at that time.

[0178] In addition to the above, Figure 14 Examples and Figure 13 The examples are the same. In Figure 14 In the example, when the save button is ON, the hyperspectral image is restored and saved. Furthermore, if the display button is ON and the save button is OFF, the RGB image is restored and displayed, but the hyperspectral image is not restored. Therefore, the processing load can be reduced.

[0179] For the restoration of RGB images, instead of mask data for four or more bands, mask data for three bands corresponding to the RGB image is used, and the restoration of the RGB image is performed in the same manner as the restoration of hyperspectral images. For example, the restoration operation described in Patent Document 2 can also be used. Specifically, using... Figure 15 Please provide an explanation.

[0180] Figure 15 This is a conceptual diagram illustrating the relationship between four or more bands and three bands. Figure 15 In this context, W represents the wavelength range that can be detected by the image sensor 111. The wavelength range W can also be the transmission wavelength range of the bandpass filter that the image sensor 111 has.

[0181] The wavelength range W is determined by w bands (w is 4 or more) of the hyperspectral image: W1, ..., W2. wThe coverage area corresponds to this. Additionally, the wavelength range W corresponds to the three bands W of the RGB image. R W G and W B The coverage area corresponds to this. That is, it is determined by bands W1, ..., W... w The coverage area is determined by the three bands W of the RGB image. R W G and W B The coverage area corresponds to this.

[0182] For example, the wavelength range W is from 400nm to 700nm. Additionally, there are, for example, w bands W1, ..., W2. w Determined using a 10nm scale. Therefore, w is 30, and there are w bands W1, W2, ..., W w The wavelengths are 400nm–410nm, 410nm–420nm, ..., 690nm–700nm, respectively. To generate such w wavelength bands W1, ..., W2... w Prepare hyperspectral images with w bands W1, ..., W2. w Mask data.

[0183] w bands W1, W2, ..., W w The mask data includes the transmittance matrix for the 400nm–410nm band, the transmittance matrix for the 410nm–420nm band, ..., and the transmittance matrix for the 690nm–700nm band.

[0184] Furthermore, by integrating the 400nm–500nm band W B The range includes 10 matrices to obtain band W. B The transmittance matrix. Additionally, by integrating the 500nm–600nm band W... G The range includes 10 matrices to obtain band W. G The transmittance matrix. Additionally, by integrating the 600nm–700nm band W... R The range includes 10 matrices to obtain band W. R The transmittance matrix.

[0185] In integrating the 10 transmittance matrices, the 10 transmittances can be either added element-wise or averaged element-wise. Thus, from w bands W1, ..., W... w The mask data yields three bands W corresponding to the RGB image. R W G and W B The mask data. That is, in order to generate an RGB image, it is possible to obtain the mask data from w bands W1, ..., W2. wPrepare the mask data for the 3 bands W of the RGB image. R W G and W B Mask data.

[0186] When H included in equation (1) is expressed as H = (H1H2……H) w ) = (H1H2……H P H P＋1 H P＋2 ...H Q H Q＋ 1H Q＋2 ...H w When ), it indicates that for band W R The matrix H of mask data R , indicates that for band W G The matrix H of mask data G , indicates that for band W B The matrix H of mask data B It can also be expressed as equation (4) or equation (5). H1, H2, ..., H P H P＋1 H P＋2 ... H Q H Q＋1 H Q＋2 ... H w Each of the following is a small matrix H, with components arranged in n×m rows and n×m columns.

[0187]

[0188] In the example above, P = 10, Q = 20, and W = 30.

[0189] Furthermore, the processing circuit 121 can operate based on three wavebands W R W G and W B The mask data and the above equations (1) and (2) are used to generate an RGB image. H and f included in equations (1) and (2) can also be:

[0190] H = (H B H G H R )

[0191] .

[0192] H B H G H R These are small matrices of H.

[0193] f R To be compatible with band WR The corresponding spectroscopic image data, that is, the data of band W R The corresponding pixel values ​​of the image.

[0194] f G To be compatible with band W G The corresponding spectroscopic image data, that is, the data of band W G The corresponding pixel values ​​of the image.

[0195] f B To be compatible with band W B The corresponding spectroscopic image data, that is, the data of band W B The corresponding pixel values ​​of the image.

[0196] It can also be based on f R f G f B Each of the n×m pixel values ​​determines the total number of pixel values ​​in an RGB image.

[0197] w bands W1, ..., W w The mask data has n×m transmission spectra S for n×m pixels. 11 ... S nm The transmittance of n×m×w elements (where w is 4 or more) is used as the value. Similarly, the three bands W R W B and W G The mask data has n×m transmission spectra S for n×m pixels. 11 ... S nm The transmittance of n×m×3 is used as the value.

[0198] That is, it can efficiently generate both mask data for generating RGB images and mask data for generating hyperspectral images from multiple common transmission spectra. Therefore, it can reduce the number of calibrations.

[0199] Furthermore, in the above, the w bands W1, ..., W of the hyperspectral image are used. w From the mask data, export the three bands W of the RGB image. R W B and W G The mask data. However, it is also possible to derive three bands W from the transmission spectrum per pixel based on the above equation (3). R W B and W G The three transmittance values ​​are used to derive the three bands W. R W G and W B Mask data.

[0200] Figure 16 It means Figure 1 A flowchart illustrating a third specific example of the operation of the image processing system 100 shown.

[0201] and Figure 14 Compared to the examples, in Figure 16 In the example, when the save button is in the ON state ("Yes" in S205), the processing circuit 121 saves the RGB image based on the hyperspectral image (S208).

[0202] In addition to the above, Figure 16 Examples and Figure 14 The examples are the same. In Figure 16 In the example, when the save button is ON, the hyperspectral image is restored and saved. Furthermore, if the display button is ON and the save button is OFF, the RGB image is restored and displayed, but the hyperspectral image is not restored. Therefore, the processing load can be reduced.

[0203] In addition, Figure 16 In the example, if the save button is ON, the displayed RGB image is saved. Therefore, in save mode, the valid RGB image and the hyperspectral image can be saved together.

[0204] Figure 17 It means Figure 1 A flowchart illustrating a fourth specific example of the operation of the image processing system 100 shown. Figure 17 In the example, the method for generating RGB images is similar to... Figure 14 The examples are different.

[0205] Specifically, in Figure 17 In the example, the method for generating the RGB image changes depending on whether the save button is ON. Specifically, if the save button is OFF ("No" in S205), the RGB image is reconstructed and displayed using mask data from three bands (S204a). This process ( Figure 17 S204a) and in Figure 14 In the example, this process is performed regardless of whether the save button is ON. Figure 14 It is the same as S204a).

[0206] On the other hand, if the save button is in the ON state ("Yes" in S205), the processing circuit 121 generates an RGB image from the hyperspectral image and displays it after restoring the hyperspectral image (S207a). Specifically, by integrating the red band W from the multiple spectroscopic images included in the hyperspectral image... RMultiple spectroscopic images of multiple bands are included, and the red band W is obtained. R The spectroscopic image. In the integration of multiple spectroscopic images, the values ​​of multiple spectroscopic images can be added together at each pixel, or a weighted addition operation can be performed on the values ​​of multiple spectroscopic images at each pixel, or the values ​​of multiple spectroscopic images can be averaged at each pixel.

[0207] Similarly, by analyzing the green band W in multiple spectroscopic images included in the hyperspectral image... G The multiple spectroscopic images of the included multiple bands are integrated to obtain the green band W. G The spectroscopic image. Additionally, by analyzing the blue band W in multiple spectroscopic images included in the hyperspectral image... B The multiple spectroscopic images of the included multiple bands are integrated to obtain the blue band W. B The spectroscopic image.

[0208] Then, through the red band W R Spectroscopic image, green band W G Spectroscopic images and the blue band W B The RGB image is obtained by combining the spectral images.

[0209] In addition to the above, Figure 17 Examples and Figure 14 The examples are the same. In Figure 17 In the example, when the save button is ON, the hyperspectral image is restored and saved. Furthermore, if the display button is ON and the save button is OFF, the RGB image is restored and displayed, but the hyperspectral image is not restored. Therefore, the processing load can be reduced.

[0210] In addition, Figure 17 In the example, if the save button is ON, an RGB image is generated from the restored hyperspectral image. This generation process has a lower processing load than the restoration process of restoring an RGB image from a compressed image. Therefore, the above actions can suppress the processing load.

[0211] Figure 18 It means Figure 1 A flowchart of a fifth specific example of the operation of the image processing system 100 shown.

[0212] and Figure 17 Compared to the examples, in Figure 18 In the example, when the save button is in the ON state ("Yes" in S205), the processing circuit 121 saves the RGB image generated from the hyperspectral image based on the hyperspectral image (S208a).

[0213] In addition to the above, Figure 18 Examples and Figure 17 The examples are the same. In Figure 18 In the example, when the save button is ON, the hyperspectral image is restored and saved. Furthermore, if the display button is ON and the save button is OFF, the RGB image is restored and displayed, but the hyperspectral image is not restored. Therefore, the processing load can be reduced.

[0214] In addition, Figure 18 In the example, if the save button is ON, the displayed RGB image is saved. Therefore, in save mode, the valid RGB image and the hyperspectral image can be saved together.

[0215] Figure 19 It means Figure 1 A flowchart of a sixth specific example of the operation of the image processing system 100 shown.

[0216] and Figure 14 Compared to the examples, in Figure 19 In the example, processing circuit 121 replaces the restoration and display of the RGB image (S204a) and restores and displays the monochrome image (S204b). Specifically, at that time, processing circuit 121 uses mask data of a band corresponding to the monochrome image to restore and display the monochrome image of the current frame from the compressed image of the current frame.

[0217] Additionally, along with the restoration and display of the monochrome image (S204b), the processing circuit 121 discards the monochrome image displayed in the previous frame instead of the RGB image displayed in the previous frame (S203a and S209a) (S203b and S209b). Furthermore, the processing circuit 121 can also discard the compressed image used in the previous frame at that time.

[0218] In addition to the above, Figure 19 Examples and Figure 14 The examples are the same. In Figure 19 In the example, when the save button is ON, the hyperspectral image is restored and saved. Furthermore, if the display button is ON and the save button is OFF, the RGB image is restored and displayed, but the hyperspectral image is not restored. Therefore, the processing load can be reduced.

[0219] For monochrome image restoration, instead of using mask data for four or more bands, mask data for one band corresponding to the monochrome image is used, and the monochrome image is restored in the same way as for hyperspectral image restoration. Specifically, mask data for four or more bands can also be integrated, similar to RGB image restoration.

[0220] In RGB image restoration, mask data with more than four bands is integrated into mask data with fewer than three bands. In monochrome image restoration, mask data with more than four bands is integrated into mask data with even fewer than one band.

[0221] Alternatively, a band of mask data can be derived by deriving a transmittance for each pixel from the transmission spectrum based on equation (3) above. This band can also correspond to a wavelength range W that can be detected by the image sensor 111.

[0222] By using monochrome images, the computational load can be reduced compared to using RGB images, and the processing load can be further suppressed.

[0223] When the save button is in the ON state ("Yes" in S205), the processing circuit 121 can also save the displayed monochrome image to the recording medium 140 based on the hyperspectral image. Thus, it is possible to save the displayed monochrome image and the hyperspectral image together in save mode.

[0224] Figure 20 It means Figure 1 A flowchart illustrating a seventh specific example of the operation of the image processing system 100 shown. Figure 20 In the example, the method for generating monochrome images is similar to... Figure 19 The examples are different.

[0225] Specifically, in Figure 20 In the example, the method for generating the monochrome image changes depending on whether the save button is in the ON state. Specifically, if the save button is in the OFF state ("No" in S205), the monochrome image is reconstructed and displayed using mask data for one band (S204b). This process ( Figure 20 S204b) and in Figure 19 In the example, this process is performed regardless of whether the save button is ON. Figure 19 It is the same as S204b.

[0226] On the other hand, if the save button is in the ON state ("Yes" in S205), the processing circuit 121 generates a monochrome image from the hyperspectral image and displays it after restoring the hyperspectral image (S207b). Specifically, a monochrome image is obtained by integrating multiple spectroscopic images included in the hyperspectral image. In the integration of multiple spectroscopic images, the values ​​of the multiple spectroscopic images can be added together for each pixel, or the values ​​of the multiple spectroscopic images can be averaged for each pixel.

[0227] In addition to the above, Figure 20 Examples and Figure 19The examples are the same. In Figure 20 In the example, when the save button is ON, the hyperspectral image is restored and saved. Conversely, if the display button is ON and the save button is OFF, the monochrome image is restored and displayed, but the hyperspectral image is not restored. Therefore, the processing load can be suppressed.

[0228] In addition, Figure 20 In the example, if the save button is ON, a monochrome image is generated from the restored hyperspectral image. This generation process has a lower processing load than the restoration process of restoring a monochrome image from a compressed image. Therefore, the processing load can be suppressed through the above actions.

[0229] When the save button is in the ON state ("Yes" in S205), the processing circuit 121 can also save the displayed monochrome image to the recording medium 140 based on the hyperspectral image. Thus, it is possible to save the displayed monochrome image and the hyperspectral image together in save mode.

[0230] Figure 21 It means Figure 1 The flowchart shows an eighth specific example of the operation of the image processing system 100.

[0231] and Figure 14 Compared to the examples, in Figure 21 In the example, processing circuit 121 replaces the restoration and display of the RGB image (S204a) and restores and displays the low-resolution RGB image (S204c). Specifically, at that time, processing circuit 121 uses low-resolution mask data of three bands corresponding to the RGB image to restore and display the low-resolution RGB image of the current frame from the compressed image of the current frame.

[0232] Additionally, along with the restoration and display of the low-resolution RGB image (S204c), the processing circuit 121 discards the RGB image displayed in the previous frame (S203a and S209a) instead of the low-resolution RGB image displayed in the previous frame (S203c and S209c). Furthermore, at that time, the processing circuit 121 can also discard the compressed image used in the previous frame.

[0233] In addition to the above, Figure 21 Examples and Figure 14 The examples are the same. In Figure 21In the example, when the save button is ON, the hyperspectral image is restored and saved. Conversely, if the display button is ON and the save button is OFF, the low-resolution RGB image is restored and displayed, but the hyperspectral image is not restored. Therefore, the processing load is reduced. Furthermore, the low-resolution RGB image has less data than the original RGB image. Therefore, the processing load is reduced.

[0234] When the save button is in the ON state (Yes in S205), the processing circuit 121 can also save the displayed low-resolution RGB image to the recording medium 140 based on the hyperspectral image. Thus, it is possible to save the display-valid low-resolution RGB image and the hyperspectral image together in save mode.

[0235] There are three methods for restoring low-resolution RGB images. The first method is as follows: by averaging both the mask data and the compressed image in the spatial direction, a compression sensing restoration operation is performed to restore the low-resolution RGB image.

[0236] Specifically, firstly, by using Figure 14 and Figure 15 The method described above is used to export mask data for three bands corresponding to the RGB image. Low-resolution mask data is obtained by averaging this mask data in the spatial direction. Similarly, a low-resolution compressed image is obtained by averaging the compressed image in the spatial direction. Then, a compression sensing restoration operation is performed using the low-resolution mask data and the low-resolution compressed image to export the low-resolution RGB image.

[0237] For spatial averaging, for example, averaging four values ​​into one value by dividing the data into 2×2 pixel blocks. Averaging is not limited to 2×2 pixel blocks; it can also be performed on other block types.

[0238] The second method is as follows: by subsampling both the mask data and the compressed image in the spatial direction, a compression sensing restoration operation is performed to restore the low-resolution RGB image. That is, instead of averaging in the first method, interleaving is performed.

[0239] For spatial subsampling, this can be performed by extracting the top-left value from four values ​​in each 2×2 pixel block. Subsampling is not limited to 2×2 pixel blocks; it can also be performed in other blocks. Furthermore, values ​​from other positions can be extracted, not just the top-left.

[0240] The third method is as follows: it restores a low-resolution RGB image by performing linear operations based on the premise that multiple spatially adjacent pixels have the same pixel value spectrum. The third method will be explained in detail below.

[0241] First, assuming that 2×2 pixels have the same pixel value spectrum, the study reconstructs three low-resolution spectroscopic images of three bands corresponding to the low-resolution RGB image as the reconstructed image. The relationship between the data g representing the compressed image and the data f representing the spectroscopic image, i.e., the reconstructed image, is expressed as in equation (1) by g = Hf. H is the system matrix mentioned above. For simplification, f is transformed into... Figure 22 The form in which it is expressed.

[0242] Figure 22 This is a diagram showing an example of data f representing a reconstructed image of three bands. Figure 22 f(a, b, c) represents the pixel value of the c-th band at pixel position (a, b).

[0243] That is, in equation (1), the pixel values ​​of the first band of the first pixel, the pixel values ​​of the first band of the second pixel, ..., the pixel values ​​of the second band of the first pixel, the pixel values ​​of the second band of the second pixel, ... are used in the order of... In contrast, in... Figure 22 In the example, the pixel values ​​of the first band of the first pixel, the pixel values ​​of the second band of the first pixel, ..., the pixel values ​​of the first band of the second pixel, the pixel values ​​of the second band of the second pixel, ... are used in the order of the first band of the first pixel, ...

[0244] Figure 23 This is a diagram showing an example of matrix H corresponding to mask data for three bands. With the transformation of the data f described above, the column arrangement of matrix H is also transformed. Figure 23 h(a, b, c) represents the transmittance of the c-th band at pixel position (a, b).

[0245] Figure 24 This is a diagram illustrating an example of data f representing a low-resolution restored image across three bands. Figure 24 In the example, pixel values ​​f(1,1,1), f(1,2,1), f(2,1,1), and f(2,2,1) have the same value according to the premise, so these four pixel values ​​can be summarized into a single pixel value f(1,1,1). By summing the four pixel values ​​into one pixel value in the same way, the number of rows in data f is reduced to 1 / 4.

[0246] Figure 25This is a diagram showing an example of matrix H corresponding to low-resolution mask data for three bands. As the number of rows in data f is reduced, the number of columns in matrix H is also reduced. Furthermore, the number of rows in matrix H is maintained at the number of pixels in the compressed image. And, for example, the aggregated pixel value f(1, 1, 1) affects the four pixel values ​​included in data g. The leftmost column of matrix H represents this effect.

[0247] With the transformation described above, the number of rows in data f becomes smaller than the number of rows in data g. Therefore, the number of unknowns decreases, and data f can be derived through linear operations. For example, it can be derived by f = H. ＋ Linear operations on g derive data f. Here, H ＋ Let H be the pseudo-inverse matrix.

[0248] In the above, the assumption was that 2×2 pixels had the same pixel value spectrum, and data f for 3 bands was derived. However, pixels with the same pixel value spectrum are not limited to 2×2 pixels, and data f is not limited to data for 3 bands.

[0249] For example, when deriving data f for N bands based on the premise that a×b pixels have the same pixel value spectrum, if a×b=N, then it is possible to obtain data f=H. －1 Linear operations on g derive the data f. Furthermore, if a × b > N, then it is possible to derive the data f through f = H. ＋ Linear operations on g derive data f.

[0250] Figure 26 It means Figure 1 A flowchart illustrating a ninth specific example of the operation of the image processing system 100. Figure 26 In the example, the method for generating low-resolution RGB images is similar to... Figure 21 The examples are different.

[0251] Specifically, in Figure 26 In the example, the method for generating the low-resolution RGB image changes depending on whether the save button is ON. Specifically, if the save button is OFF ("No" in S205), the low-resolution RGB image is reconstructed and displayed using low-resolution mask data for three bands (S204c). This process ( Figure 26 S204c) and in Figure 21 In the example, this process is performed regardless of whether the save button is ON. Figure 20 It is the same as S204c.

[0252] On the other hand, if the save button is in the ON state ("Yes" in S205), the processing circuit 121 generates a low-resolution RGB image from the hyperspectral image after restoring the hyperspectral image and displays it (S207c). Specifically, by analyzing the red band W in the multiple spectroscopic images included in the hyperspectral image... R The multiple spectroscopic images of the included multiple bands are integrated to obtain the red band W. R Low-resolution spectrophotometer images.

[0253] In the integration of multiple spectroscopic images, the values ​​of the multiple spectroscopic images can be added together in 2×2 pixel blocks, or the values ​​of the multiple spectroscopic images can be averaged in 2×2 pixel blocks. The blocks are not limited to 2×2 pixel blocks; integration can also be performed on other block-by-block basis.

[0254] Similarly, by analyzing the green band W in multiple spectroscopic images included in the hyperspectral image... G The multiple spectroscopic images of the included multiple bands are integrated to obtain the green band W. G Low-resolution spectroscopic images. Additionally, by analyzing the blue band W in multiple spectroscopic images included in the hyperspectral image... B The multiple spectroscopic images of the included multiple bands are integrated to obtain the blue band W. B Low-resolution spectrophotometer images.

[0255] Then, through the red band W R Low-resolution spectrophotometer images, green band W G Low-resolution spectrophotometer images and blue band W B By combining low-resolution spectrophotometers, a low-resolution RGB image is obtained.

[0256] In addition to the above, Figure 26 Examples and Figure 21 The examples are the same. In Figure 26 In the example, when the save button is ON, the hyperspectral image is restored and saved. Conversely, if the display button is ON and the save button is OFF, the low-resolution RGB image is restored and displayed, but the hyperspectral image is not restored. Therefore, the processing load is reduced.

[0257] In addition, Figure 26 In the example, if the save button is ON, a low-resolution RGB image is generated from the restored hyperspectral image.

[0258] When the save button is in the ON state ("Yes" in S205), the processing circuit 121 can also save the displayed low-resolution RGB image to the recording medium 140 based on the hyperspectral image. Thus, it is possible to save the display-valid low-resolution RGB image and the hyperspectral image together in save mode.

[0259] Figure 27 It means Figure 1 This is a conceptual diagram of a second display example of the display device 130. For example, the processing circuit 121 displays an RGB image on the display device 130 as a first image in visual confirmation mode. In visual confirmation mode, the displayed RGB image is an RGB image restored from a compressed image. In visual confirmation mode, hyperspectral images are not restored and are not displayed.

[0260] For example, the RGB image is updated based on the frame rate. That is, the RGB image corresponds to the moving image. The RGB image is confirmed by the user, and the position, orientation, and field of view of the shooting device 110 are adjusted so that the subject being photographed is included in the hyperspectral image. Furthermore, after the adjustment, the mode is switched from visual confirmation mode to save mode.

[0261] Figure 28 It means Figure 1 This is a conceptual diagram of a third display example of the display device 130. For example, in save mode, the processing circuit 121 displays an RGB image and a hyperspectral image as a first image and a second image, respectively, on the display device 130. Furthermore, the processing circuit 121 displays the hyperspectral image on the display device 130 as a plurality of spectroscopic images. In this example, the hyperspectral image includes 12 spectroscopic images, each corresponding to one of the 12 wavelength bands.

[0262] In save mode, the displayed RGB image can be either an RGB image restored from a compressed image or an RGB image generated from a hyperspectral image. For example, an RGB image generated by combining one or more spectroscopic images corresponding to R, one or more spectroscopic images corresponding to G, and one or more spectroscopic images corresponding to B can also be displayed.

[0263] In addition, after switching from save mode to visual confirmation mode, the display of the hyperspectral image that was last restored in save mode can continue in visual confirmation mode.

[0264] Figure 29 It means Figure 1 This is a conceptual diagram of a fourth display example of the display device 130. For example, the processing circuit 121 displays the RGB image and the analysis results of the photographed subject based on the hyperspectral image on the display device 130 in a save mode.

[0265] The analysis results of a subject based on a hyperspectral image can also be an image representing the identification results of the subject identified based on the hyperspectral image. Alternatively, the analysis results can be images obtained by processing RGB and hyperspectral images. Furthermore, the analysis results can also be statistical information from the hyperspectral image, or they can be displayed in text format.

[0266] The processing circuit 121 can also further display the hyperspectral image on the display device 130 in the save mode. That is, the processing circuit 121 can also display the RGB image, the hyperspectral image, and the analysis results of the subject based on the hyperspectral image on the display device 130 in the save mode.

[0267] Alternatively, the processing circuit 121 may display the hyperspectral image on the display device 130 instead of the RGB image in the save mode. That is, the processing circuit 121 may also display the hyperspectral image and the analysis results of the photographed object based on the hyperspectral image on the display device 130 in the save mode.

[0268] Alternatively, the processing circuit 121 can also save the analysis results of the subject based on the hyperspectral image to the recording medium 140 in the save mode, instead of the hyperspectral image.

[0269] In addition, after switching from save mode to visual confirmation mode, the last analysis results obtained in save mode can be displayed in visual confirmation mode.

[0270] Figure 30 It means Figure 1 The flowchart shows a tenth specific example of the operation of the image processing system 100.

[0271] In this example, the processing circuit 121 acquires the compressed image (S301). Specifically, the compressed image is acquired by generating a compressed image through the image sensor 111, and the processing circuit 121 acquires the compressed image from the image sensor 111.

[0272] Next, the processing circuit 121 performs recognition processing on the compressed image (S302). As the recognition processing, the recognition processing described in Patent Document 3 can be used.

[0273] Specifically, for example, processing circuit 121 can apply preprocessing to the compressed image to improve recognition accuracy. Preprocessing may include region extraction, smoothing, feature extraction, edge detection, or any combination thereof. Then, processing circuit 121 can also use a learning model to perform recognition processing on the subject included in the preprocessed compressed image. As the learning model, deep learning models such as convolutional neural networks (CNNs) or recurrent neural networks (RNNs) can be used.

[0274] Then, based on the result of the recognition processing, the processing circuit 121 determines whether the compressed image includes a specific subject (S303). That is, the processing circuit 121 determines whether a specific subject is reflected in the compressed image.

[0275] If the compressed image includes a specific subject ("Yes" in S303), the processing circuit 121 uses mask data of four or more bands to restore the hyperspectral image and saves the hyperspectral image to the recording medium 140 (S304). Then, the processing circuit 121 generates an RGB image from the hyperspectral image and displays the RGB image on the display device 130 (S305).

[0276] If the compressed image does not include a specific subject ("No" in S303), the processing circuit 121 uses mask data of three bands to restore the RGB image and displays the RGB image on the display device 130 (S306).

[0277] Then, the processing circuit 121 discards the displayed RGB image (S307). Then, the image processing system 100 ends the processing. The image processing system 100 may also repeatedly perform a series of actions (S301 to S307).

[0278] In the above operations, the visual verification mode and the save mode are switched depending on whether the compressed image contains the subject. That is, the save mode is used when the compressed image includes the subject, and the visual verification mode is used when the compressed image does not include the subject. Therefore, the processing load can be reduced. In addition, recognition processing is performed using the compressed image without restoring the hyperspectral image. As a result, the overall processing volume and processing time can be reduced.

[0279] The image processing system 100 that performs the above-described identification process can also be applied to foreign object inspection in factories. Specifically, the image processing system 100 can also use a learning model to perform foreign object identification processing on compressed images. Furthermore, if foreign objects are reflected in a compressed image, the image processing system 100 can also recover a hyperspectral image from the compressed image and save it. Moreover, the image processing system 100 can both prompt the user with the hyperspectral image and remove foreign objects based on the hyperspectral image.

[0280] Furthermore, the image processing system 100 that performs the above-described recognition processing can also be applied to facial skin condition recognition. Specifically, the image processing system 100 can also use a learning model to perform facial recognition processing on compressed images. Moreover, when a face is reflected in a compressed image, the image processing system 100 can also recover and save a hyperspectral image from the compressed image. Furthermore, the image processing system 100 can also perform facial skin condition recognition based on the hyperspectral image.

[0281] Furthermore, the image processing system 100 that performs the above-described recognition process can also be applied to recognition processing in a conveyor belt. Specifically, the image processing system 100 can also perform recognition processing on a two-dimensional barcode placed near an object based on a compressed image. The image processing system 100 can also recover a hyperspectral image for processing the object from the compressed image if a two-dimensional barcode is reflected in the compressed image.

[0282] Figure 31 It means Figure 1 The flowchart shows the eleventh specific example of the operation of the image processing system 100.

[0283] In this example, processing circuit 121 obtains the compressed image (S301). This processing is related to... Figure 30 The processing is the same as in the example.

[0284] Next, the processing circuit 121 uses mask data from three bands to reconstruct the RGB image and displays it on the display device 130 (S306). This processing is consistent with... Figure 30 The same processing is performed in the example where the specific subject is not included in the compressed image.

[0285] Next, the processing circuit 121 performs recognition processing on the RGB image (S302a). Then, based on the result of the recognition processing, the processing circuit 121 determines whether the RGB image includes a specific subject (S303a). Figure 30 In the example, compressed images are used, but... Figure 31 In the example, RGB images are used. Aside from the differences between compressed and RGB images, these processing methods are similar to... Figure 30 The processing is the same as in the example.

[0286] When a specific subject is included in the compressed image ("Yes" in S303a), the processing circuit 121 uses mask data from four or more bands to reconstruct the hyperspectral image and records the hyperspectral image on the recording medium 140 (S304). This processing is related to... Figure 30The processing is the same as in the example. If the specific subject is not included in the compressed image ("No" in S303a), the hyperspectral image is not restored and is not saved.

[0287] Then, the processing circuit 121 discards the displayed RGB image (S307). Then, the image processing system 100 ends the processing. The image processing system 100 may also repeatedly perform a series of actions (S301 to S307).

[0288] exist Figure 31 In the example, RGB images are used in the recognition process. Therefore, color information can also be used in the recognition process. This enables more complex recognition processes with high accuracy.

[0289] The image processing system and the like have been described above according to the embodiments, but the technical solutions of the image processing system and the like are not limited to the embodiments. Modifications that can be conceived by those skilled in the art can be implemented in the embodiments, and multiple constituent elements in the embodiments can be combined arbitrarily.

[0290] For example, a process performed by a specific component in an implementation may be performed by another component instead of that specific component. Furthermore, the order of multiple processes may be changed, or multiple processes may be performed in parallel. Additionally, the first and second ordinal numbers used in the description may be appropriately replaced, removed, or reassigned. These ordinal numbers do not necessarily correspond to a meaningful order and may also be used to identify components.

[0291] Additionally, for example, at least one of the first element, the second element, and the third element, this expression corresponds to the first element, the second element, the third element, or any combination thereof.

[0292] Furthermore, the method for performing the steps of each component, including the image processing system, can be executed by any system or device. That is, the method can be executed by the aforementioned image processing system or by other systems or devices.

[0293] For example, part or all of this method can be executed by a computer equipped with a processor, memory, and input / output circuits. In that case, the method can also be executed by the computer executing a program designed to make the computer perform the method.

[0294] For example, the above procedure causes a computer to execute an image processing method, which includes: acquiring a compressed image; switching between a visual confirmation mode and a save mode; in the visual confirmation mode, displaying a first image on a display device, the first image being one of the compressed image and a display-processed image, the display-processed image being an image generated based on the compressed image and represented by information from three or fewer bands; deleting the first image after displaying it; and in the save mode, generating a second image based on the compressed image, recording the second image on a recording medium, the second image being represented by information from four or more bands.

[0295] In addition, the above program can also be recorded on non-transitory computer-readable recording media such as CD-ROM.

[0296] Furthermore, the components of an image processing system, etc., can be composed of dedicated hardware, general-purpose hardware that executes the aforementioned programs, or a combination thereof. Additionally, the general-purpose hardware can consist of a memory storing programs and a general-purpose processor that reads the programs from the memory and executes them. Here, the memory can be a semiconductor memory or a hard disk, etc., and the general-purpose processor can be a CPU, etc.

[0297] Alternatively, dedicated hardware can also consist of memory and a dedicated processor. For example, a dedicated processor can refer to memory to execute the above methods.

[0298] Furthermore, the constituent elements of an image processing system, etc., can also be circuits. These circuits can either be configured as a single circuit or be separate circuits. Additionally, these circuits can correspond to dedicated hardware or general-purpose hardware that executes the aforementioned programs, etc.

[0299] (other)

[0300] The embodiments of this disclosure may also be modified as shown below.

[0301] (Variation a)

[0302] A device,

[0303] The device includes:

[0304] An image sensor, comprising a plurality of pixels; and

[0305] Controller

[0306] After the image sensor receives first light from a filter array comprising four or more filters with different transmission spectra across a wavelength range, the image sensor outputs a first plurality of pixel values ​​corresponding to the first plurality of pixels.

[0307] After the image sensor receives the second light from the filter array, the image sensor outputs a second plurality of pixel values ​​corresponding to the first plurality of pixels.

[0308] After the image sensor receives the first light, the image sensor receives the second light.

[0309] The controller performs a first process before receiving a command.

[0310] The controller performs a second process after receiving the command.

[0311] In the first processing, the controller generates a third plurality of pixel values ​​corresponding to the first band, a fourth plurality of pixel values ​​corresponding to the second band, and a fifth plurality of pixel values ​​corresponding to the third band, based on the first plurality of pixel values ​​and the first information.

[0312] The total number of the third plurality of pixel values ​​is the same as the total number of the first plurality of pixel values.

[0313] The total number of the fourth plurality of pixel values ​​is the same as the total number of the first plurality of pixel values.

[0314] The total number of the fifth plurality of pixel values ​​is the same as the total number of the first plurality of pixel values.

[0315] In the first processing, the controller generates a first image based on the third plurality of pixel values, the fourth plurality of pixel values, and the fifth plurality of pixel values.

[0316] In the first processing, the controller does not generate more than four second images corresponding one-to-one with more than four bands based on the first plurality of pixel values ​​and the second information.

[0317] In the second processing, the controller generates a sixth plurality of pixel values ​​corresponding to the first band, a seventh plurality of pixel values ​​corresponding to the second band, and an eighth plurality of pixel values ​​corresponding to the third band, based on the second plurality of pixel values ​​and the first information.

[0318] The total number of the sixth plurality of pixel values ​​is the same as the total number of the first plurality of pixel values.

[0319] The total number of the seventh plurality of pixel values ​​is the same as the total number of the first plurality of pixel values.

[0320] The total number of the eighth plurality of pixel values ​​is the same as the total number of the first plurality of pixel values.

[0321] In the second processing, the controller generates a third image based on the sixth plurality of pixel values, the seventh plurality of pixel values, and the eighth plurality of pixel values.

[0322] In the second processing, the controller generates four or more fourth images that correspond one-to-one with the four or more bands, based on the second plurality of pixel values ​​and the second information.

[0323] The wavelength range is divided into the first band, the second band, and the third band.

[0324] The wavelength range is divided into four or more bands.

[0325] The width of the first band is a first width, the width of the second band is a second width, and the width of the third band is a third width.

[0326] The bandwidth of each of the four or more bands is smaller than the first bandwidth, smaller than the second bandwidth, and smaller than the third bandwidth.

[0327] (Variation b)

[0328] The device according to the modified example a

[0329] The device also includes a memory.

[0330] Before the image sensor receives the first light, the first information and the second information are stored in the memory.

[0331] (Variation c)

[0332] According to the apparatus of either Modification a or Modification b, the command means to display four or more images corresponding one-to-one with the four or more bands on the display.

[0333] (Explanation of variation a)

[0334] The first plurality of pixel values ​​can also be an n×m matrix g with 1 column, represented by equation (1). In this description, the first plurality of pixel values ​​are recorded as an n×m matrix g1 with 1 column.

[0335] The second plurality of pixel values ​​can also be an n×m matrix g with 1 column, as represented by equation (1). In this description, the second plurality of pixel values ​​are recorded as an n×m matrix g2 with 1 column.

[0336] Figure 32This is a diagram illustrating an example of the relationship between the first plurality of pixels included in the image sensor and the values ​​of the first plurality of pixels, and the relationship between the plurality of pixels included in the image sensor and the values ​​of the second plurality of pixels.

[0337] The first plurality of pixels can be pixels p(1,1), ..., pixels p(n,m).

[0338] The first multiple pixel values ​​can be the pixel value g1(1,1) output by pixel p(1,1), ..., the pixel value g1(n,m) output by pixel p(n,m).

[0339] The second set of multiple pixel values ​​can be the pixel value g2(1,1) output by pixel p(1,1), ..., the pixel value g2(n,m) output by pixel p(n,m).

[0340] The above wavelength range can be Figure 15 The wavelength range W is shown.

[0341] The first piece of information mentioned above can be a matrix H1 = (Hm × m rows, n × m × 3 columns) B H G H R H B H G H R Each is a small matrix of size H1.

[0342] The aforementioned third or more pixel values ​​corresponding to the first band can also be represented as those of band W. R The corresponding n×m row, 1 column matrix f 1R The included n×m components,

[0343] The aforementioned fourth set of pixel values ​​corresponding to the second band can also be represented as those of band W. G The corresponding n×m row, 1 column matrix f 1G The included n×m components,

[0344] The aforementioned fifth and multiple pixel values ​​corresponding to the third band can also be represented as those of band W. B The corresponding n×m row, 1 column matrix f 1B It includes n×m components.

[0345] Equation (1) can be expressed as:

[0346] .

[0347] The first image mentioned above can also be based on f 1R f 1G f 1B The first RGB image generated.

[0348] The second piece of information mentioned above can also be a matrix H2 = (H1H2……H) with n×m rows and n×m×w columns. w H1, H2, ..., Hw are each a submatrix of H2.

[0349] The above four or more bands can also be used as Figure 15 The bands W1, ..., W shown are bands W1, ..., band W2. w .

[0350] In variation a, the statement "the controller, in the first processing, does not generate more than four second images corresponding one-to-one with more than four bands based on the first plurality of pixel values ​​and the second information" means that the controller, in the first processing, does not base its actions on g1 = (g1(1,1)...g1(n,m)). T H2 generates more than 4 bands (i.e., bands W1, ..., band W). w There are four or more second images that correspond one-to-one. That is, the controller is not based on...

[0351]

[0352] The corresponding equation (2) is used to calculate f for band W1. 11 ... and band W w The corresponding f 1w This reduces the computational load on the controller.

[0353] The sixth or more pixel values ​​corresponding to the first band mentioned above can also be represented as those of band W. R The corresponding n×m row, 1 column matrix f 2R The included n×m components,

[0354] The seventh or more pixel values ​​corresponding to the second band mentioned above can also be represented as those of band W. G The corresponding n×m row, 1 column matrix f 2G The included n×m components,

[0355] The aforementioned eighth or more pixel values ​​corresponding to the third band can also be represented as those of band W. B The corresponding n×m row, 1 column matrix f 2B It includes n×m components.

[0356] Equation (1) can be expressed as:

[0357] .

[0358] The third image mentioned above can also be based on f 2R f2G f 2B The generated second RGB image.

[0359] In variation example a, "the controller generates four or more fourth images corresponding one-to-one with the four or more bands based on the second plurality of pixel values ​​and the second information in the second processing" means that the controller, in the second processing, generates four or more fourth images based on g2 = (g2(1,1)...g2(n,m)). T And H2, generating more than 4 bands (that is, band W1, ..., band W...). w There are four or more fourth images that correspond one-to-one. That is, the controller is based on...

[0360]

[0361] The corresponding equation (2) is used to calculate f for band W1. 21 f 22 ... and band W w The corresponding f 2w .

[0362] The controller is based on f 21 Generate image I corresponding to band W1 21 ..., based on f 2w Generation and band W w Corresponding image I 2w .

[0363] Figure 35 shows the representation of image I. 21 Image I 2w An example diagram.

[0364] Image I 21 Including pixel value f 21 The pixel value of (1,1) is p21(1,1), ..., the pixel value of which is f. 21 The pixels p21(n,m) of (n,m), ..., the image I 2w Including pixel value f 2w The pixel p2w(1,1), ... has a value of f. 2w The pixel p2w(n,m) of (n,m).

[0365] Industrial availability

[0366] This disclosure is applicable to image processing methods for image restoration and can be used in image processing systems, shooting systems, camera systems, analysis systems, and recognition systems, etc.

[0367] Explanation of reference numerals in the attached figures

[0368] 100 Image Processing System

[0369] 110 camera device

[0370] 111 Image Sensor

[0371] 112 filter array

[0372] 113 Optical System

[0373] 120 Image Processing Device

[0374] 121 Processing Circuit

[0375] 130 display device

[0376] 140 Recording media

Claims

1. An image processing method, comprising: Obtain a compressed image; Switch between visual confirmation mode and save mode; In the visual confirmation mode, a first image is displayed on the display device, and the first image is deleted after being displayed. The first image is one of the compressed image and the display processing image, which is an image generated based on the compressed image and represented by information from three or fewer bands. as well as In the storage mode, a second image representing information from four or more bands is generated based on the compressed image, and the second image is stored on the recording medium.

2. The image processing method according to claim 1, The first image is the compressed image.

3. The image processing method according to claim 1, The first image is the image that has been processed for display.

4. The image processing method according to claim 3, The first image is an RGB image represented by information from three bands.

5. The image processing method according to claim 3, The first image is a monochrome image represented by information from one wavelength band.

6. The image processing method according to any one of claims 3 to 5, The resolution of the first image is lower than that of the second image.

7. The image processing method according to any one of claims 3 to 5, In acquiring the compressed image, the compressed image is obtained through multiple light-receiving regions having multiple transmission spectra. In the generation of the first image, the first image is generated based on the compressed image and first mask data including multiple values ​​reflecting the multiple transmission spectra. In the generation of the second image, the second image is generated based on the compressed image and second mask data including multiple values ​​reflecting the multiple transmission spectra. The first mask data includes fewer values ​​than the second mask data includes fewer values.

8. The image processing method according to any one of claims 1 to 5, The switching between the visual confirmation mode and the save mode is based on the user's actions.

9. The image processing method according to any one of claims 1 to 5, Also included are: In the visual confirmation mode, it is determined whether the first image includes a specific subject. During the switching between the visual confirmation mode and the save mode, If it is determined that the first image does not include the specific subject, the visual confirmation mode continues. If it is determined that the first image includes the specific subject, the system switches from the visual confirmation mode to the save mode.

10. The image processing method according to any one of claims 1 to 5, Furthermore, in the save mode, the first image is displayed on the display device.

11. The image processing method according to claim 10, Furthermore, in the storage mode, the first image is stored on the recording medium.

12. The image processing method according to claim 10, Furthermore, in the save mode, one or both of the second image and the analysis results of the subject based on the second image are displayed.

13. The image processing method according to any one of claims 3 to 5, Furthermore, in the save mode, the first image is generated based on the second image generated from the compressed image, and the first image is displayed on the display device.

14. An image processing system, comprising: Image sensor, which acquires compressed images; and The processing circuit switches between visual confirmation mode and save mode. The processing circuit, In the visual confirmation mode, a first image is displayed on the display device, and then deleted after the first image is displayed. The first image is one of the compressed image and the display-processed image, which is an image generated based on the compressed image and represented by information from three or fewer bands. In the storage mode, a second image representing information from four or more bands is generated based on the compressed image, and the second image is stored on the recording medium.

15. A program for causing a computer to perform the image processing method according to any one of claims 1 to 5.

Images