Signal processing method and signal processing device

Abstract

A signal processing method executed using a computer includes: acquiring a compressed image including compressed information of a subject; acquiring a first parameter group and a restoration matrix used in a first restoration process for generating a restoration target image from the compressed image; generating the restoration target image on the basis of the compressed image, the first parameter group, and the restoration matrix; acquiring a second parameter group used in a second restoration process for generating a restored image from the compressed image; generating the restored image on the basis of the compressed image, the second parameter group, and the restoration matrix; and correcting the second parameter group on the basis of the restoration target image and the restoration image.

Description

Technical Field

[0001] The present disclosure relates to a signal processing method and a signal processing device. Background Art

[0002] Compressed sensing is a technology that restores more data than the observed data by assuming that the data distribution of the observed object is sparse in a certain space (such as frequency space). Compressed sensing can be applied to a camera system that restores an image containing more information from a small amount of observed data. When compressed sensing is applied to a camera system, for example, a filter that has the function of encoding the image of light spatially or wavelength-wise can be used. Such a camera system can obtain a compressed image by photographing the object through a filter, and generate a restored image containing more information than the compressed image through calculation. As a result, various effects such as high resolution of the image, multi-wavelength, shortened shooting time, or high sensitivity can be obtained.

[0003] Patent Document 1 discloses an example of applying compressed sensing technology to a hyperspectral camera that acquires images in multiple wavelength bands, each of which is a narrow band. According to the technology disclosed in Patent Document 1, a high-resolution multi-wavelength hyperspectral image can be generated.

[0004] Patent Document 2 discloses a super-resolution method for generating a high-resolution image from less observation information using compressed sensing technology.

[0005] Patent Document 3 discloses a method of restoring an image having a higher resolution than an acquired image by applying a convolutional neural network (CNN) to the acquired image.

[0006] Prior art literature

[0007] Patent Literature

[0008] Patent Document 1: U.S. Patent No. 9599511

[0009] Patent Document 2: Japanese Patent Application Publication No. 2017-208641

[0010] Patent Document 3: U.S. Patent Application Publication No. 2019 / 0340497 Summary of the invention

[0011] The present disclosure provides a technology for improving processing efficiency or performance by restoring a restored image with a larger amount of information from a compressed image.

[0012] A signal processing method related to a technical solution of the present disclosure is a signal processing method executed by a computer. The method includes: obtaining a compressed image containing compressed information of a subject; obtaining a first parameter group and a restoration matrix used in a first restoration process for generating a restored target image based on the compressed image; generating the restored target image based on the compressed image, the first parameter group and the restoration matrix; obtaining a second parameter group used in a second restoration process for generating a restored image based on the compressed image; generating the restored image based on the compressed image, the second parameter group and the restoration matrix; and correcting the second parameter group based on the restored target image and the restored image.

[0013] The inclusive or specific technical solutions disclosed in the present invention may be implemented by a system, device, method, integrated circuit, computer program or a computer-readable recording disk or other recording medium, or by any combination of systems, devices, methods, integrated circuits, computer programs and recording media. Computer-readable recording media may include volatile recording media or non-volatile recording media such as CD-ROM (Compact Disc-Read Only Memory). A device may also be composed of more than one device. In the case where a device is composed of more than two devices, the two or more devices may be configured in one device or separately configured in two or more separate devices. In this specification and the scope of protection requested, "device" may refer not only to one device but also to a system composed of multiple devices.

[0014] According to the technology disclosed in the present invention, it is possible to realize an improvement in the efficiency or performance of a process of generating a restored image having a larger amount of information from a compressed image. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 Detailed description of the invention is given below.

[0016] Figure 2A It is a diagram schematically showing a configuration example of an imaging system.

[0017] Figure 2B This is a diagram showing a configuration example of an imaging device in which a filter array and an image sensor are arranged separately.

[0018] Figure 2C This is a diagram showing another configuration example of an imaging device in which a filter array and an image sensor are arranged separately.

[0019] Figure 2D This is a diagram showing still another configuration example of an imaging device in which the filter array and the image sensor are arranged separately.

[0020] Figure 3A It is a diagram schematically showing an example of a filter array.

[0021] Figure 3B This is a diagram showing an example of spatial distribution of light transmittance in each of a plurality of wavelength bands included in the target wavelength range.

[0022] Figure 3C Yes means Figure 3A Graph showing an example of the spectral transmittance of region A1 included in the optical filter array shown.

[0023] Figure 3D Yes means Figure 3A Graph showing an example of the spectral transmittance of region A2 included in the filter array shown.

[0024] Figure 4A It is used to explain the target wavelength range W and the multiple wavelength bands W1, W2, ..., W contained therein. N A diagram showing an example of the relationship between .

[0025] Figure 4B It is used to explain the target wavelength range W and the multiple wavelength bands W1, W2, ..., W contained therein. N A diagram of another example of the relationship of .

[0026] Figure 5 This is a block diagram showing a structural example of a signal processing device.

[0027] Figure 6 This is a conceptual diagram showing an example of the first restoration process and the second restoration process.

[0028] Figure 7 This is a conceptual diagram showing another example of the first restoration process and the second restoration process.

[0029] Figure 8 This is a flowchart showing an example of a method for adjusting the second parameter group.

[0030] Fig. 9 This is a diagram showing an example of a GUI for allowing the user to confirm whether or not the restoration target image can also be adopted.

[0031] Fig.10 FIG. 1 is a diagram showing an example of a restoration target image and a restored image displayed on a display device.

[0032] Fig.11 Yes means Fig.10 Flowchart of a variation of the method shown.

[0033] Fig.12 The diagram shows an example in which a partial region in an image is designated by a user.

[0034] Fig.13 This is a diagram showing an example of a displayed warning.

[0035] Fig.14 This is a diagram showing an example of a GUI for changing parameters.

[0036] Fig.15 This is a diagram showing an example of a one-dimensional vector of n×m rows and one column based on image data which is two-dimensional data of n×m pixels. DETAILED DESCRIPTION

[0037] The embodiments described below are all inclusive or specific examples. The numerical values, shapes, materials, constituent elements, configuration positions and connection forms of constituent elements, steps, the order of steps, and the layout of the display screen, etc. shown in the following embodiments are examples and are not intended to limit the technology disclosed herein. Among the constituent elements of the following embodiments, constituent elements that are not recorded in the independent claims representing the highest concept are described as arbitrary constituent elements. Each figure is a schematic diagram and is not necessarily a strict illustration. Furthermore, in each figure, the same reference numerals are given to substantially the same or similar constituent elements. There are cases where repeated descriptions are omitted or simplified.

[0038] In the present disclosure, all or part of a circuit, unit, device, component or part, or all or part of a functional block in a block diagram may be performed by one or more electronic circuits including a semiconductor device, a semiconductor integrated circuit (IC) or an LSI (large scale integration). LSI or IC may be integrated into one chip or may be composed of a combination of multiple chips. For example, functional blocks other than storage elements may be integrated into one chip. Here, it is referred to as LSI or IC, but the name may vary depending on the degree of integration, and may also be referred to as system LSI, VLSI (very large scale integration) or ULSI (ultra large scale integration). Field Programmable Gate Array (FPGA) that can be programmed after the manufacture of LSI, or a reconfigurable logic device (reconfigurable logic device) that can reconfigure the bonding relationship within LSI or set the circuit division within LSI may also be used for the same purpose.

[0039] Furthermore, all or part of the functions or operations of a circuit, unit, device, component or part can be executed by software processing. In this case, the software is recorded in one or more non-transitory recording media such as ROM, optical disk, hard disk drive, etc. When the software is executed by a processing device (processor), the functions determined by the software are executed by the processing device and peripheral devices. The system or device may also have one or more non-transitory recording media with software recorded, a processing device and necessary hardware devices, such as an interface.

[0040] In the present disclosure, data or signals representing an image may be simply referred to as an “image”.

[0041] [1. Signal processing method]

[0042] In the process of restoring a restored image containing more information from a compressed image with less information, various algorithms can be applied. For example, various algorithms based on the above-mentioned compressed sensing technology or various algorithms based on machine learning such as deep learning can be used. Each algorithm has inherent characteristics. For example, when a certain algorithm is used, high-precision restoration can be performed, but the amount of calculation is large, and there are cases where a long time is required in the restoration process. On the other hand, when other algorithms are used, restoration can be performed in a shorter time, but there are cases where the restoration accuracy deteriorates.

[0043] Here, as an example, a system for inspecting products transported by conveying devices such as belt conveyors based on hyperspectral images is considered. In such a system, for example, an imaging device having a filter such as that disclosed in Patent Document 1 can be used. Such an imaging device sequentially photographs the product via a filter that encodes the image of light with respect to the wavelength to generate a compressed image. By applying an algorithm using compressed sensing or machine learning to the generated compressed image, a restored image (such as a hyperspectral image) can be generated. Based on the generated restored image, it is possible to check whether the product has no abnormalities or whether there are no foreign objects mixed in the product. In such a system, real-time processing is required. Therefore, it is requested to perform restoration processing in a short time using a high-speed algorithm. However, in such an algorithm, generally, more parameters need to be set to appropriate values ​​for accurate restoration, and a method for efficiently optimizing the parameters is required.

[0044] Based on the above-mentioned considerations, the present disclosure provides a technique for efficiently optimizing one or more parameters in a restoration processing algorithm actually used at a site of an inspection or the like.

[0045] Figure 1 1 is a flowchart showing a signal processing method according to an embodiment of the present disclosure. The signal processing method is executed by a computer. Figure 1 The illustrated signal processing method includes the processing from the following steps S11 to S16.

[0046] (S11) A compressed image including compressed information of a subject is acquired.

[0047] (S12) A first parameter group and a restoration matrix used in a first restoration process for generating a restoration target image from a compressed image are obtained.

[0048] (S13) A restored target image is generated based on the compressed image, the first parameter group and the restoration matrix.

[0049] (S14) A second parameter group used in a second restoration process for generating a restored image from the compressed image is obtained.

[0050] (S15) A restored image is generated based on the compressed image, the second parameter group and the restoration matrix.

[0051] (S16) Based on the restoration target image and the restored image, the second parameter group is corrected.

[0052] Here, a "compressed image" is an image with a relatively small amount of information obtained by video recording. A compressed image may be, for example, image data in which information of multiple wavelength bands is compressed into one image information, but is not limited thereto. A compressed image may also be image data used to restore an MRI (Magnetic Resonance Imaging) image. Alternatively, a compressed image may also be image data used to generate a high-resolution image.

[0053] The "restored target image" is data of an image that is a target for restoration. The restored target image is generated from the compressed image by the first restoration processing based on the first algorithm. As the first algorithm, for example, an algorithm with a large amount of computation but high restoration performance can be used. For example, an algorithm that performs a restoration processing based on compressed sensing can be used as the first algorithm. Before performing the first restoration processing, the first parameter group is set. The first parameter group includes one or more parameters. The first parameter group can include multiple parameters or a single parameter. The parameters included in the first parameter group are sometimes referred to as "first parameters". The first parameter group can be set by the user or automatically by the system. The value of the set first parameter group can be saved in a storage medium such as a memory. In the following description, the restored target image is sometimes referred to as the "target image".

[0054] A "restored image" is an image restored for the purpose of inspection or analysis. The restored image is generated from the compressed image by a second restoration process based on a second algorithm. As the second algorithm, an algorithm with a smaller computational load than the first algorithm can be used. For example, an algorithm that is faster than the first algorithm or an algorithm that uses less memory can be used as the second algorithm. A second parameter group is set before performing the second restoration process. The second parameter group includes one or more parameters. The second parameter group can include multiple parameters or a single parameter. The parameters included in the second parameter group are sometimes referred to as "second parameters". The second parameter group may also include more parameters than the first parameter group. For example, the number of parameters in the second parameter group may be more than twice, more than five times, or more than ten times the number of parameters in the first parameter group. The second parameter group may also include, for example, more than 10, more than 30, or more than 50 parameters.

[0055] The "restored matrix" is matrix data used in the first restoration process and the second restoration process. The restored matrix can be stored in a storage medium such as a memory in the form of a table, for example. Therefore, the restored matrix is ​​sometimes referred to as a "restored table". The restored matrix can be, for example, a matrix that reflects the characteristics of a filter used in imaging based on compressed sensing.

[0056] The correction of the second parameter group in step S16 may include correcting one or more parameters included in the second parameter group so that the restored image is close to the restored target image. For example, the correction of the second parameter group may include obtaining an error evaluation value between the target restored image and the restored image, and correcting one or more parameters included in the second parameter group so that the error evaluation value is minimized. Thus, the second parameter group can be adjusted so that the restored image is close to the restored target image.

[0057] The final second parameter group may be determined by repeatedly performing the processing of step S15 and the processing of step S16 for a plurality of times. That is, the signal processing method may also include determining the final second parameter group by repeatedly performing the steps of correcting the second parameter group based on the restored target image and the restored image and generating the restored image using the corrected second parameter group for a plurality of times. Thus, for example, the second parameter group can be optimized so that the restored image is substantially consistent with the restored target image.

[0058] The second restoration process may also include a process based on a learned model trained by machine learning, such as deep learning. Such a process is high-speed and can generate a restored image in a short time. In the algorithm based on machine learning, a large number of parameters are required to be optimized. In this embodiment, the parameters can be optimized efficiently based on a high-precision restored target image.

[0059] The first restoration process may also not include processing based on a learned model trained by machine learning. The first restoration process may, for example, include an iterative operation that minimizes or maximizes an evaluation function based on a compressed image and a restoration matrix. An algorithm that performs such an iterative operation can perform a high-precision restoration, but on the other hand, the amount of calculation is large, and sometimes a restored image cannot be generated in a short time. Therefore, the first algorithm that performs the first restoration process is not used in an actual environment such as an inspection, but is used to generate a restored target image that is referenced to correct the second parameter group in the second algorithm used in an actual environment. By using the high-precision restored target image obtained by the first restoration process to correct the second parameter group, the restoration performance of the second restoration process can be improved.

[0060] The compressed image may also be an image obtained by encoding the spectral information of the object. In other words, the compressed image may also be an image obtained by compressing the information of multiple wavelength bands of the object into one monochrome image. In this case, the restored target image and the restored image may also include information of multiple images corresponding to the multiple wavelength bands. Thus, images of multiple wavelength bands (e.g., hyperspectral images) can be restored from the compressed image.

[0061] The method may further include causing the display device to display a graphical user interface (GUI) for the user to input the first parameter group. Thus, the user can set the first parameter group via the GUI.

[0062] The above method may also include: repeatedly performing a process of correcting the second parameter group based on the restored target image and the restored image and generating the restored image using the corrected second parameter group for a predetermined number of times as long as the termination condition is not satisfied; calculating an error evaluation value between the restored image and the restored target image after the predetermined number of repetitions; and, when the error evaluation value is greater than a threshold, causing the display device to display a GUI that urges the user to re-enter the first parameter group, change the predetermined number of times, and change the termination condition. Thus, when the error evaluation value between the restored image and the restored target image is not less than the threshold, the user can change the generation condition of the restored target image.

[0063] Calculating the error evaluation value may also include: extracting a first region in the restored target image; extracting a second region corresponding to the first region in the restored image; and determining the error evaluation value based on the difference between the first region and the second region. The first region and the second region may be determined based on a region specified by a user, for example. Thus, the second parameter group can be corrected so that the error between the regions extracted from the restored image and the restored target image becomes smaller.

[0064] A signal processing device according to another embodiment of the present disclosure includes one or more processors and a memory storing a computer program executed by the one or more processors. The computer program causes the one or more processors to execute the signal processing method. That is, the computer program causes the one or more processors to execute: (a) obtaining a compressed image containing compressed information of a subject; (b) obtaining a first parameter group and a restoration matrix used in a first restoration process for generating a restored target image based on the compressed image; (c) generating the restored target image based on the compressed image, the first parameter group and the restoration matrix; (d) obtaining a second parameter group used in a second restoration process for generating a restored image based on the compressed image; (e) generating the restored image based on the compressed image, the second parameter group and the restoration matrix; and (f) correcting the second parameter group based on the restored target image and the restored image.

[0065] With the above configuration, the second parameter for generating the restored image can be appropriately corrected based on the restored image generated by the first restoration process and the restored image generated by the second restoration process.

[0066] [2. Camera system]

[0067] Next, a configuration example of an imaging system that can be used in the exemplary embodiment of the present disclosure will be described.

[0068] Figure 2A 1 is a diagram schematically showing a configuration example of an imaging system. The imaging system includes an imaging device 100 and a signal processing device 200 (hereinafter referred to as "processing device 200"). The imaging device 100 has the same configuration as the imaging device disclosed in Patent Document 1. The imaging device 100 includes an optical system 140, a filter array 110, and an image sensor 160. The optical system 140 and the filter array 110 are arranged on the optical path of light incident from an object 70 as a subject. Figure 2A The filter array 110 in the example is arranged between the optical system 140 and the image sensor 160 .

[0069] exist Figure 2AIn the figure, an apple is illustrated as an example of the object 70. The object 70 is not limited to an apple, but may be any object. The image sensor 160 generates data of a compressed image 10 that compresses information of multiple wavelength bands into a two-dimensional monochrome image. The processing device 200 generates image data of each of the multiple wavelength bands included in a specified target wavelength domain based on the data of the compressed image 10 generated by the image sensor 160. The generated image data of the multiple wavelength bands is sometimes referred to as a "hyperspectral (HS) data cube" or "hyperspectral image data". Here, the number of wavelength bands included in the target wavelength domain is assumed to be N (N is an integer greater than or equal to 4). In the following description, the generated image data of the multiple wavelength bands are referred to as restored images 20W1, 20W2, ..., 20W N , they are sometimes collectively referred to as a "hyperspectral image 20" or a "hyperspectral data cube 20". In this specification, a collection of data or signals representing an image, that is, data or signals representing the pixel value of each pixel, is sometimes simply referred to as an "image".

[0070] The filter array 110 in this embodiment is an array of multiple filters with light transmittance arranged in rows and columns. The multiple filters include multiple types of filters with different spectral transmittances, that is, wavelength dependence of light transmittance. The filter array 110 modulates the intensity of incident light according to the wavelength and outputs it. Sometimes this process performed by the filter array 110 is called "encoding", and the filter array 110 is called a "coding element" or "coding masker".

[0071] exist Figure 2A In the example shown, the filter array 110 is arranged near or in close contact with the image sensor 160. Here, "near" means close to the extent that the image of the light from the optical system 140 is formed on the surface of the filter array 110 in a state of a certain degree of clarity. "In close contact" means that the two are close to the extent that there is almost no gap. The filter array 110 and the image sensor 160 may also be integrated.

[0072] The optical system 140 includes at least one lens. Figure 2A In FIG. 1 , the optical system 140 is shown as a single lens, but the optical system 140 may be a combination of a plurality of lenses. The optical system 140 forms an image on an imaging plane of the image sensor 160 via the filter array 110 .

[0073] The filter array 110 may be configured separately from the image sensor 160 . FIG. 2B to FIG. 2D 1 is a diagram showing a configuration example of an imaging device 100 in which a filter array 110 and an image sensor 160 are arranged separately. Figure 2BIn the example of FIG. 1 , the filter array 110 is disposed between the optical system 140 and the image sensor 160 and at a position separated from the image sensor 160. Figure 2C In the example of FIG. 1 , the filter array 110 is disposed between the object 70 and the optical system 140. Figure 2D In the examples of FIG. 1 , the imaging device 100 includes two optical systems 140A and 140B, and the filter array 110 is disposed between them. As in these examples, an optical system including one or more lenses may be disposed between the filter array 110 and the image sensor 160 .

[0074] The image sensor 160 is a monochrome light detection device having a plurality of light detection elements (also referred to as "pixels" in this specification) arranged in a two-dimensional plane. The image sensor 160 may be, for example, a CCD (Charge-Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor) sensor, or an infrared array sensor. The light detection element includes, for example, a photodiode. The image sensor 160 does not necessarily need to be a monochrome sensor. For example, a color sensor having a filter of red (R) / green (G) / blue (B), R / G / B / infrared (IR), or R / G / B / transparent (W) may also be used. By using a color sensor, the amount of information related to the wavelength can be increased, and the accuracy of the reconstruction of the hyperspectral image 20 can be improved. The wavelength range of the acquisition object can be determined arbitrarily, and is not limited to the visible wavelength range, but may also be a wavelength range of ultraviolet, near infrared, mid-infrared, or far infrared.

[0075] The processing device 200 may be a computer including one or more processors and one or more storage media such as a memory. The processing device 200 generates a plurality of restored images 20W1, 20W2, ..., 20W based on the compressed image 10 acquired by the image sensor 160. N The processing device 200 may also be incorporated into the imaging device 100.

[0076] Figure 3A 1 is a diagram schematically showing an example of a filter array 110. The filter array 110 has a plurality of regions arranged in a two-dimensional plane. In this specification, the region is sometimes referred to as a "cell". In each region, a filter having a spectral transmittance set individually is arranged. Let the wavelength of the incident light be λ, and the spectral transmittance is represented by a function T(λ). The spectral transmittance T(λ) can take a value greater than 0 and less than 1.

[0077] exist Figure 3AIn the example shown, the filter array 110 has 48 rectangular areas arranged in 6 rows and 8 columns. This is just an example, and more areas than this may be provided in actual use. The number of such areas may be, for example, the same as the number of pixels of the image sensor 160. The number of filters included in the filter array 110 is determined, for example, in the range of several tens to several tens of millions depending on the use.

[0078] Figure 3B represents a plurality of wavelength bands W1, W2, ..., W included in the target wavelength region. N An example of the spatial distribution of the transmittance of each light. Figure 3B In the example shown, the difference in depth of each area indicates the difference in transmittance. The lighter the area, the higher the transmittance, and the darker the area, the lower the transmittance. Figure 3B As shown, the spatial distribution of light transmittance differs depending on the wavelength band.

[0079] Figure 3C and Figure 3D They are respectively Figure 3A FIG. 1 is a diagram showing an example of spectral transmittance of region A1 and region A2 included in the filter array 110 shown. The spectral transmittance of region A1 and the spectral transmittance of region A2 are different from each other. In this way, the spectral transmittance of the filter array 110 is different according to the region. However, it is not necessary that the spectral transmittance of all regions is different. In the filter array 110, the spectral transmittance of at least a part of the multiple regions is different from each other. The filter array 110 includes two or more filters with different spectral transmittances. In a certain example, the number of patterns of spectral transmittance of the multiple regions included in the filter array 110 can be the same as or greater than the number N of wavelength bands included in the target wavelength domain. The filter array 110 can also be designed so that the spectral transmittance of more than half of the regions is different.

[0080] Figure 4A and Figure 4B It is used to explain the target wavelength range W and the multiple wavelength bands W1, W2, ..., W included therein. N The target wavelength domain W can be set to various ranges according to the application. The target wavelength domain W can be, for example, the wavelength domain of visible light from about 400nm to about 700nm, the wavelength domain of near infrared from about 700nm to about 2500nm, or the wavelength domain of near ultraviolet from about 10nm to about 400nm. Alternatively, the target wavelength domain W can also be a wavelength domain such as mid-infrared or far-infrared. In this way, the wavelength domain used is not limited to the visible light domain. In this specification, it is not limited to visible light, and all radiation including infrared and ultraviolet rays are referred to as "light".

[0081] exist Figure 4AIn the example shown, N is an arbitrary integer greater than or equal to 4, and the target wavelength range W is divided into N equal parts, each of which is a wavelength band W1, a wavelength band W2, ..., a wavelength band W N However, this is not limited to such an example. The multiple wavelength bands included in the target wavelength domain W can also be set arbitrarily. For example, the bandwidth can be set to be uneven according to the wavelength band. There can also be gaps or overlaps between adjacent wavelength bands. Figure 4B In the example shown, the bandwidths differ according to the wavelength bands, and there is a gap between two adjacent wavelength bands. In this way, the method of determining the plurality of wavelength bands is arbitrary.

[0082] exist FIG. 3A to FIG. 3D In the example shown, a grayscale transmittance distribution is assumed in which the transmittance of each region can take any value between 0 and 1. However, it is not necessary to have a grayscale transmittance distribution. For example, a two-level transmittance distribution can be adopted in which the transmittance of each region can take a value of about 0 or about 1. In the two-level transmittance distribution, each region transmits most of the light in at least two wavelength domains among the multiple wavelength domains included in the target wavelength domain, and does not transmit most of the light in the remaining wavelength domains. Here, "most of" means about 80% or more.

[0083] Alternatively, a portion of the entire cell, for example, half of the cell, may be replaced with a transparent area. Such a transparent area allows all wavelength bands W1 to W2 included in the target wavelength range W to be covered. N The light of the filter array 110 is transmitted with a higher transmittance of the same degree, for example, a transmittance of more than 80%. In such a structure, the plurality of transparent regions can be arranged, for example, in a checkerboard shape. That is, the regions with different light transmittances according to the wavelength and the transparent regions can be arranged alternately in two arrangement directions of the plurality of regions in the filter array 110.

[0084] Data indicating the spatial distribution of the spectral transmittance of the filter array 110 can be acquired in advance based on design data or actual measurement calibration, and stored in a storage medium included in the processing device 200. This data is used in the calculation process described later.

[0085] The filter array 110 can be formed, for example, using a multilayer film, an organic material, a diffraction grating structure, or a fine structure containing metal. In the case of using a multilayer film, for example, a dielectric multilayer film or a multilayer film containing a metal layer can be used. In this case, each unit cell is formed in at least one different way in the thickness, material, and stacking order of each multilayer film. Thus, different spectral characteristics can be achieved according to the unit cell. By using a multilayer film, a steep rise and fall in spectral transmittance can be achieved. The structure using organic materials can be achieved by making the pigments or dyes contained different according to the unit cell, or stacking different types of materials. The structure using a diffraction grating structure can be achieved by setting a diffraction structure with different diffraction spacing or depth for each unit cell. In the case of using a fine structure containing metal, it can be made using the spectral characteristics brought about by the plasma exciton effect.

[0086] Next, an example of signal processing performed by the processing device 200 is described. The processing device 200 reconstructs a multi-wavelength hyperspectral image 20 based on the compressed image 10 output from the image sensor 160 and the spatial distribution characteristics of the transmittance of each wavelength of the filter array 110. Here, multi-wavelength refers to, for example, more wavelength domains than the wavelength domain of RGB (red, green, and blue) three colors obtained by a conventional color camera. The number of wavelength domains can be, for example, about 4 to 100. The number of wavelength domains is called the "band number". Depending on the application, the number of bands may also exceed 100.

[0087] The data to be obtained is the data of the hyperspectral image 20, and the data is denoted by f. If the number of bands is N, then f is the image data f1 of the image corresponding to the wavelength band W1, the image data f2 of the image corresponding to the wavelength band W2, ..., and the image data f3 of the image corresponding to the wavelength band W1. N The image data f of the corresponding image N The merged data. Here, Figure 2A As shown in FIG. 1 , let the horizontal direction of the image be the x direction, and let the vertical direction of the image be the y direction. If the number of pixels in the x direction of the image data to be obtained is m, and the number of pixels in the y direction is n, then image data f1, image data f2, ..., image data f N are two-dimensional data of n×m pixels respectively. Therefore, the data f is three-dimensional data with n×m×N elements. This three-dimensional data is called "hyperspectral image data" or "hyperspectral data cube". On the other hand, the image data g of the compressed image 10 obtained by encoding and multiplexing the filter array 110 is two-dimensional data of n×m pixels. The image data g can be represented by the following formula (1).

[0088] [Formula 1]

[0089]

[0090] g included in the formula (1) is a one-dimensional vector of n×m rows and one column based on the image data g which is the above-mentioned two-dimensional data of n×m pixels.

[0091] f1 included in the formula (1) is a one-dimensional vector of n×m rows and 1 column based on the image data f1 which is the two-dimensional data of n×m pixels mentioned above, f2 included in the formula (1) is a one-dimensional vector of n×m rows and 1 column based on the image data f2 which is the two-dimensional data of n×m pixels mentioned above, ..., f included in the formula (1) is a one-dimensional vector of n×m rows and 1 column based on the image data f2 which is the two-dimensional data of n×m pixels mentioned above, ..., N The image data f is based on the above-mentioned two-dimensional data of n×m pixels. N A one-dimensional vector with n×m rows and 1 column.

[0092] Fig.15 This is a diagram showing an example of a one-dimensional vector of n×m rows and one column based on image data which is two-dimensional data of n×m pixels. i(1,1) represents the pixel value of a pixel located at coordinates (x,y)=(1,1), ..., i(n,m) represents the pixel value of a pixel located at coordinates (x,y)=(n,m).

[0093] The f contained in the formula (1) is a one-dimensional vector with n×m×N rows and 1 column. The matrix H represents the components f1, f2, ..., f N The encoding and intensity modulation are performed using different encoding information (also called "masking information") for each wavelength band, and then these are added. Therefore, H is a matrix of n×m rows and n×m×N columns. This matrix H is sometimes called a "restored matrix".

[0094] As long as g and matrix H are given, f should be calculated by solving the inverse problem of equation (1). However, since the number of elements of the obtained data f is n×m×N, which is greater than the number of elements of the obtained data g, n×m, this problem is a bad setting problem and cannot be solved in the original state. Therefore, the processing device 200 uses the redundancy of the image contained in the data f and uses the compressed sensing method to solve it. Specifically, the obtained data f is estimated by solving the following equation (2).

[0095] [Formula 2]

[0096]

[0097] Here, f' represents the data of the inferred f. The first term in the brackets of the above formula represents the deviation between the inferred result Hf and the acquired data g, which is the so-called residual term. Here, the sum of squares is set as the residual term, but the absolute value or the square root of the sum of squares can also be used as the residual term. The second term in the brackets is a regularization term or a stabilization term. Formula (2) means to find f that minimizes the sum of the first term and the second term. The function in the brackets in formula (2) is called an evaluation function. The processing device 200 can converge the solution through recursive iterative operations, and calculate the f that minimizes the evaluation function as the final solution f'.

[0098] The first term in the brackets of formula (2) means the operation of finding the sum of squares of the differences between the acquired data g and Hf obtained by transforming the f of the inference process by the matrix H. The second term Φ(f) is a constraint condition for the regularization of f, and is a function that reflects the sparse information of the inferred data. This function has the effect of smoothing or stabilizing the inferred data. The regularization term can be represented by, for example, discrete cosine transform (DCT), wavelet transform, Fourier transform or total variation (TV) of f. For example, when the total variation is used, stable inferred data that suppresses the influence of noise of the observed data g can be obtained. The sparsity of the object 70 in the space of each regularization term is different depending on the texture of the object 70. It is also possible to select a regularization term that makes the texture of the object 70 sparser in the space of the regularization term. Alternatively, multiple regularization terms may be included in the operation. τ is a weight coefficient. The larger the weight coefficient τ, the more redundant data is reduced and the higher the compression ratio. The smaller the weight coefficient τ, the weaker the convergence to the solution. The weight coefficient τ is set to an appropriate value that allows f to converge to a certain extent and does not lead to over-compression.

[0099] In addition, Figure 2B and Figure 2C In the configuration, the image encoded by the filter array 110 is obtained in a blurred state on the imaging plane of the image sensor 160. Therefore, the blur information can be kept in advance, and the hyperspectral image 20 can be reconstructed by reflecting the blur information in the above-mentioned matrix H. Here, the blur information is represented by a point spread function (Point Spread Function: PSF). PSF is a function that specifies the degree of diffusion of a point image to surrounding pixels. For example, in the case where a point image equivalent to 1 pixel on an image is diffused to an area of ​​k×k pixels around the pixel by blurring, PSF can be specified as a coefficient group, i.e., a matrix, representing the influence on the pixel values ​​of each pixel in the area. By reflecting the influence of the blur based on the coding pattern of the PSF in the matrix H, the hyperspectral image 20 can be reconstructed. The position where the filter array 110 is configured is arbitrary, but a position where the coding pattern of the filter array 110 does not diffuse excessively and disappear can be selected.

[0100] Through the above-described processing, the hyperspectral image 20 can be restored from the compressed image 10 acquired by the image sensor 160 .

[0101] [3. Optimization of parameters for restoration processing]

[0102] By using the camera system as described above, it is possible to construct a system for inspecting or analyzing an object transported by a conveying device based on a hyperspectral image. In such a system, in order to realize real-time inspection or analysis, it is required to perform the restoration processing for generating a restored image from a compressed image in a short time. However, in the existing algorithms based on compressed sensing, although the restoration performance is high, the amount of calculation is large, and there are cases where the processing cannot be completed within the requested short time. On the other hand, among the existing algorithms, there are also algorithms that can perform high-speed processing but have many parameters to be set, and the optimization of parameters is difficult or complicated.

[0103] Here, the process of generating a restored image based on a compressed image is not limited to the algorithm based on the compressed sensing of the above-mentioned formula (2), and can be performed using an algorithm that utilizes machine learning. Among the algorithms that utilize machine learning, there are algorithms that, for example, apply a learned model trained by deep learning to a compressed image and generate a restored image. By utilizing such an algorithm, the time required for the restoration process can be shortened. However, in order to achieve high-precision restoration, parameter optimization is required. Therefore, the parameters in the machine learning algorithm can be optimized based on the restored target image, which is generated using an algorithm based on compressed sensing. In this way, parameter optimization can be performed efficiently.

[0104] Figure 5 It is a block diagram showing a configuration example of the processing device 200 . Figure 5 The processing device 200 shown is used in combination with the imaging device 100 and the terminal device 500. The imaging device 100 may be the hyperspectral imaging device described above. The terminal device 500 is a computer for executing various operations related to the process of parameter optimization for generating a restored image. The terminal device 500 includes an input device 510 and a display device 520.

[0105] The processing device 200 includes one or more processors 210 such as a CPU or a GPU and one or more memories 250. The memory 250 stores computer programs executed by the processor 210 and various data generated by the processor 210. The computer program enables the processor 210 to execute Figure 1 That is, the processor 210 executes the following processing by executing the computer program.

[0106] · Obtaining a compressed image generated by the imaging device 100 .

[0107] Obtain the first parameter group and the restoration matrix (equivalent to the above-mentioned matrix H) from the memory 250 .

[0108] Generate a restored target image based on the compressed image, the first parameter group, and the restoration matrix.

[0109] Obtain the second parameter group from the memory 250 .

[0110] Generate a restored image based on the compressed image, the second parameter group, and the restoration matrix.

[0111] Based on the restoration target image and the restored image, the second parameter group is modified.

[0112] exist Figure 5 In the example shown, the processor 210 has the functions of a restored target image generation unit 212, an image restoration unit 214, and a parameter optimization unit 216. These functions can be implemented by software processing. In addition, in the present embodiment, one processor 210 performs all the processing of the restored target image generation unit 212, the image restoration unit 214, and the parameter optimization unit 216, but this is just an example. These processes can also be performed by multiple hardware (such as circuits or computers). For example, the processing of restoring the target image generation, image restoration, and parameter optimization can also be performed by multiple computers interconnected via a network.

[0113] The restored target image generation unit 212 performs a first restoration process based on the compressed image generated by the imaging device 100 and the first parameter group and the restoration matrix stored in the memory 250 to generate a restored target image. The first algorithm is used in the first restoration process. The generated restored target image is output to the display device 520 and the restoration parameter optimization unit 216.

[0114] The image restoration unit 214 performs a second restoration process based on the compressed image, the second parameter group, and the restoration matrix to generate a restored image. The second algorithm is used in the second restoration process. The generated restored image is output to the display device 520 and the parameter optimization unit 216.

[0115] The parameter optimization unit 216 modifies the second parameter group based on the restored target image and the restored image, and outputs the modified second parameter group to the image restoration unit 214. The restoration parameter optimization unit 216 modifies the second parameter group, for example, so that the difference between the restored image and the restored target image becomes smaller. The process of modifying the second parameter group based on the restored target image and the restored image and the process of generating the restored image using the modified second parameter group are repeated a predetermined number of times to determine the final value of the second parameter group. In this way, the second parameter group is optimized.

[0116] In the case of restoring a hyperspectral image from a compressed image, the restored target image and the restored image each include a plurality of images corresponding to a plurality of wavelength bands. The plurality of wavelength bands may include, for example, four or more bands having relatively narrow bandwidths, such as a band of wavelengths of 400 to 410 nm and a band of wavelengths of 410 to 420 nm. Correcting the second parameter group so that the restored image is close to the restored target image may include correcting one or more values ​​corresponding to one or more parameters included in the second parameter group so that the images of each band in the restored image are close to the images of the corresponding bands in the restored target image. For example, the second parameter group may be corrected so that the image of the wavelengths of 400 to 410 nm in the restored image is close to the image of the wavelengths of 400 to 410 nm in the restored target image, and the image of the wavelengths of 410 to 420 nm in the restored image is close to the image of the wavelengths of 410 to 420 nm in the restored target image.

[0117] The size of the difference between the restored image and the restored target image can be evaluated based on the error evaluation value. The error evaluation value can be calculated, for example, by calculating the error between the restored image and the restored target image in the image of the wavelength band corresponding to each other, and by summing or averaging these errors. That is, minimizing the error evaluation value can include minimizing the sum or average of the errors for each of the plurality of wavelength bands.

[0118] The display device 520 displays the restoration target image and the restored image generated in the process of optimizing the second parameter group.

[0119] The input device 510 may include a keyboard, a mouse, or other devices used by a user to set various setting items such as the first parameter group.

[0120] Figure 6 : is a conceptual diagram showing an example of a first restoration process and a second restoration process. The first restoration process in this example uses a first algorithm based on compressed sensing to generate a restored target image through the recursive iterative operation shown in the above-mentioned formula (2). The first parameter group in the first algorithm may include several parameters such as the number of iterations and the regularization coefficient τ. The initial value of the first parameter group is pre-stored in the memory 250. The first parameter group can also be set by the user using the input device 510. The first restoration process includes an iterative operation that minimizes the evaluation function based on the compressed image and the restoration matrix, that is, the function in the brackets on the right side of the formula (2). In addition, an evaluation function that represents the inverse of the value of the function, for example, can be used instead of the function in the brackets on the right side of the formula (2). In this case, the first restoration process includes an iterative operation that maximizes the evaluation function based on the compressed image and the restoration matrix.

[0121] Figure 6The second restoration process in the example uses a second algorithm based on machine learning such as deep learning to generate a restored image. On this basis, the second parameter group is corrected to minimize the error evaluation value between the restored image and the restored target image. The error evaluation value is, for example, the sum or average of the squares or absolute values ​​of the differences in the pixel values ​​in the area of ​​the object of interest between the restored image and the restored target image. In deep learning, a plurality of nodes are respectively provided in the input layer, a plurality of hidden layers, and an output layer, and each node has an inherent weight. The second parameter group may include the weights and hyperparameters of each of the multiple nodes in the deep learning algorithm. The hyperparameters may, for example, include the number of learning times, the learning rate, and the parameters for determining the learning algorithm. The second parameter group may, for example, include several hundred or more parameters. When correcting the second parameter group, the algorithm for determining the weights, the number of learning times, the learning rate, and / or the learning algorithm of each of the multiple nodes may also be corrected.

[0122] exist Figure 6 In the example of , the restored target image generated in the first restoration process is used as learning data, and training in the second restoration process, that is, optimization of the second parameter group, is performed. Thus, the second parameter group can be optimized efficiently.

[0123] The second restoration process does not necessarily need to be performed by a machine learning algorithm. For example, the second restoration process may also be performed by an algorithm based on compressed sensing.

[0124] Figure 7 2 is a conceptual diagram showing another example of the first restoration process and the second restoration process. Figure 7 In the example of , both the first restoration process and the second restoration process are performed using an algorithm based on compressed sensing. The first restoration process can be performed, for example, by a first algorithm having a high restoration performance but a high computational load. On the other hand, the second restoration process can be performed by a second algorithm having a lower computational load than the first algorithm but requiring more parameter settings.

[0125] exist Figure 7 In the first restoration process shown in FIG. 1 , a first restoration process is performed based on n first parameters p1, ..., p n On the other hand, in the second restoration process, the iterative calculation function Ψ1 is performed based on the m second parameters q1, ..., q greater than n. m Iterative operation of the function Ψ2. Figure 7 In the above example, the restored target image generated by the kth iteration operation is represented as f1 k , the restored image generated by the k-th iterative operation is expressed as f2 k . The first parameter p1, ..., p n For example, it can be set by the user. Since m is larger than n, the second parameters q1, ..., q mThe adjustment ratio of the first parameter p1, ..., p n In this embodiment, the second parameters q1, ..., q m The optimization is performed so that the restored image generated in the second restoration process is close to the restored target image generated in the first restoration process. Thus, the second parameter group can be optimized efficiently.

[0126] As Figure 6 The first algorithm in the example and Figure 7 The first algorithm and the second algorithm in the example may use any compressed sensing algorithm such as iterative shrinkage / thresholding, two-step iterative shrinkage / thresholding, or generalized alternating projection-total variation. The second algorithm is selected from algorithms different from the first algorithm. The second algorithm may be selected from algorithms having an advantage over the first algorithm, such as short operation time, high memory efficiency, or high restoration accuracy.

[0127] Figure 8 This is a flowchart showing an example of a method for adjusting the second parameter group. Figure 8 The illustrated method is executed by the processor 210 of the processing device 200 .

[0128] In step S101, the processor 210 obtains a compressed image generated by the imaging device 100. The processor 210 may obtain the compressed image directly from the imaging device 100 or obtain the compressed image via a storage medium such as the memory 250.

[0129] In step S102, processor 210 obtains a restoration matrix from memory 250. The restoration matrix is ​​generated in advance and stored in memory 250. Step S102 may be performed before step S101 or simultaneously with step S101.

[0130] In step S103, the processor 210 obtains the value of the first parameter group from the memory 250. When the user sets the first parameter group using the input device 510, the processor 210 obtains the set value of the first parameter group.

[0131] In step S104, the processor 210 generates a restored target image based on the compressed image, the restoration matrix and the first parameter group. This process is equivalent to the first restoration process described above and is performed using the first algorithm. When the first algorithm is an algorithm that performs iterative operations based on compressed sensing, the restored target image is generated by performing a predetermined number of iterative operations.

[0132] In step S105, it is determined whether the restored target image is a desired image. This determination may be performed based on an operation performed by the user using the input device 510. For example, the processor 210 may determine to display the generated restored target image on the display device 520 and to cause the user to confirm whether the restored target image may also be adopted, and determine that the restored target image is a desired image when the user confirms the adoption of the restored target image.

[0133] Fig. 9 FIG. 1 is a diagram showing an example of a GUI for allowing a user to confirm whether or not a restored target image can be adopted. Fig. 9 In the example, the display device 520 displays the generated restored target image 521, the number of iterative operations and the regularization coefficient as the first parameter, and the GUI522. The GUI522 includes buttons of "OK" and "Reset parameters". If the user presses the "OK" button, the displayed restored target image is adopted, and the processing proceeds to step S106. On the other hand, if the user presses the "Reset parameters" button, the user can reset the first parameter. In this case, the processing returns to step S103, and the processor 210 generates the restored target image again using the reset first parameter. Thereafter, the processing of steps S103 to S105 is repeated until it is determined in step S105 that the desired image is obtained.

[0134] Through the above processing, the generation of the restoration target image by the first restoration processing is completed. Next, the generation of the restoration image by the second restoration processing is performed.

[0135] In step S106 , the processor 210 sets the second parameter group to the initial values ​​stored in the memory 250 .

[0136] In step S107, the processor 210 generates a restored image based on the compressed image, the restoration matrix and the second parameter group. This process is equivalent to the second restoration process described above and is performed using the second algorithm. When the second algorithm is an algorithm that performs iterative operations based on compressed sensing, the restored image is generated by performing a predetermined number of iterative operations.

[0137] In step S108, the processor 210 compares the restored image with the restored target image and evaluates the error. For example, the processor 210 uses an error evaluation function that represents the size of the difference between the restored image and the restored target image to determine the error evaluation value. As the error evaluation function, for example, the mean square error (MSE) can be used. MSE is represented by the following formula (3).

[0138] [Formula 3]

[0139]

[0140] Here, n and m represent the number of pixels in the vertical and horizontal directions of the image, respectively. i，j Represents the pixel value of row i and column j of the correct image, I i，j Represents the pixel value of row i and column j of the inferred restored image. The error is not limited to MSE, and can also be represented by other error evaluation indicators such as RootMSE (RMSE, root mean square error), Peak-Signal-to-Noise-Ratio (PSNR, peak signal-to-noise ratio), MeanAbsoluteError (MAE, mean absolute error), StructuralSimilarity (SSMI, structural similarity index), or spectral angle.

[0141] In this embodiment, the restored target image and the restored image each include a plurality of images corresponding to a plurality of wavelength bands. For example, a plurality of images such as an image corresponding to a band of wavelengths of 400 to 410 nm and an image corresponding to a band of wavelengths of 410 to 420 nm may be restored in the first restoration process and the second restoration process, respectively. In such a case, an error evaluation function such as MSE may be calculated for each corresponding band between the restored image and the restored target image. The error evaluation value may be determined by summing or averaging the values ​​of the error evaluation function calculated for each band.

[0142] The plurality of restoration target images may include a first restoration target image corresponding to the wavelength band W1, a second restoration target image corresponding to the wavelength band W2, ..., a second restoration target image corresponding to the wavelength band W3, ..., a third restoration target image corresponding to the wavelength band W4, ..., a fourth restoration target image corresponding to the wavelength band W5, ..., a fifth restoration target image corresponding to the wavelength band W6, ..., a fifth restoration target image corresponding to the wavelength band W7. N The corresponding Nth restored target image.

[0143] The plurality of restored images may include a first restored image corresponding to the wavelength band W1, a second restored image corresponding to the wavelength band W2, ..., a second restored image corresponding to the wavelength band W N The corresponding Nth restored image.

[0144] The error evaluation values ​​between multiple restored target images and multiple restored images can also be determined based on the first error evaluation value between the first restored target image and the first restored image, the second error evaluation value between the second restored target image and the second restored image,..., the Nth error evaluation value between the Nth restored target image and the Nth restored image.

[0145] Alternatively, (error evaluation values ​​between a plurality of restoration target images and a plurality of restoration images)={(first error evaluation value)+(second error evaluation value)+…+(Nth error evaluation value)}.

[0146] Alternatively, (error evaluation values ​​between a plurality of restoration target images and a plurality of restoration images)={(first error evaluation value)+(second error evaluation value)+…+(Nth error evaluation value)} / N may be used.

[0147] The first error evaluation value may be the MSE between the first restored target image and the first restored image, the second error evaluation value may be the MSE between the second restored target image and the second restored image, ..., and the Nth error evaluation value may be the MSE between the Nth restored target image and the Nth restored image.

[0148] In step S109, the processor 210 updates the second parameter group to reduce the error between the restored image and the restored target image. For example, the second parameter group may be updated using a gradient descent method or a Bayesian optimization method to reduce the error evaluation value.

[0149] In step S110, the processor 210 determines whether a pre-set loop end condition is satisfied. The loop end condition may be, for example, that the optimization loop from step S107 to S109 is repeated a predetermined number of times, or that the error evaluation value between the restored image and the restored target image becomes smaller than a threshold value. If the loop end condition is not satisfied, the process returns to step S107. If the loop end condition is satisfied, the process ends.

[0150] When executing the processes from step S107 to step S109, the processor 210 may cause the display device 520 to display the restored image in the process of being generated together with the restored target image. Thus, the user can compare the restored image with the restored target image and confirm whether the optimization of the second parameter group is performed without any problem.

[0151] Fig.10 5 is a diagram showing an example of a restored target image 521 and a restored image 523 displayed by the display device 520. In this example, the restored target image 521 and the restored image 523 after the second parameter group is optimized are displayed side by side. The first parameter for generating the restored target image 521 and the end condition of the optimization loop for generating the restored image 523 are also displayed. The first parameter and the end condition can also be set by the user through the operation of the input device 510. Fig.10 In the example of , the first parameter includes the number of iterations and the regularization coefficient, and the end condition of the optimization loop includes the maximum number of loops and the allowable value of the error. The user views the displayed restored target image 521 and the restored image 523, and can change the parameters or the end condition of the optimization loop as needed to make the processor 210 perform the restoration process again.

[0152] Fig.11 Yes means Figure 8 Flowchart of a variation of the method shown. Fig.11 The flowchart shown in FIG. 1 is similar to the flowchart shown in FIG. 1 in that step S206 is added between step S105 and step S106, step S108 is replaced by step S208, and steps S211 and S212 are executed when the determination result in step S110 is "yes". Figure 8 The following is a different flowchart from the Figure 8 The examples are different.

[0153] exist Fig.11 In the example of FIG. 1 , after determining “yes” in step S105, the process proceeds to step S206. In step S206, the processor 210 extracts one or more regions from the restored target image. The processor 210 may be configured to extract one or more regions specified by the user. The user may specify, for example, a region where an important subject to be inspected or analyzed is shown. Fig.12 206 is a diagram showing an example of a region in which a user specifies a portion of the region. In this example, region A where a specific subject is shown is specified. Region B that is not specified is treated as a background. Processor 210 may also extract a region where an important subject is estimated to exist from the restoration target image by image processing regardless of the user's operation. After step S206, steps S106 and S107 are executed to generate a restored image.

[0154] In the next step S208, the processor 210 compares the restored image with the restored target image and evaluates the error. At this time, the processor 210 weights the evaluation value of each area so that the restoration accuracy of the designated area is improved. Fig.12 In the example, the weight of the evaluation value of the designated area is relatively large, and the weight of the evaluation value of the undesignated area is relatively small. For example, the error evaluation value can be expressed by the following formula with a and b as coefficients (a>b).

[0155] Evaluation value = a × evaluation value of area A + b × evaluation value of area B

[0156] Thus, the error in area B has little effect on the evaluation value, while the error in area A has a greater effect on the evaluation value. The coefficient a can be set to a value larger than the coefficient b, for example, 1.5 times, 2 times, or more of the coefficient b. By such processing, restoration is performed with priority given to restoration accuracy in areas where important subjects exist.

[0157] exist Fig.11 In the example of , after it is determined in step S110 that the loop end condition is satisfied, it is determined whether the error evaluation value between the restored image and the restored target image is greater than a threshold value (step S211). If the determination is "yes", the processor 210 causes the display device 520 to display a warning urging the user to reconstruct the target image (e.g., re-input the first parameter group) or change the optimization loop end condition (step S212).

[0158] Fig.13 : is a diagram showing an example of a displayed warning. In this example, a message "The error of the restored image exceeds the threshold. Please change the parameters." is displayed. If the user presses the OK button, the screen changes to a screen for changing the parameters.

[0159] Fig.14 : is a diagram showing an example of a GUI for changing parameters. In this example, in addition to the target image and the optimized restored image, an error image is also displayed. The error image is an image that shows the degree of consistency between the target image and the restored image, and may be, for example, a difference image, a square error image, or a spectral angle distribution image. In addition, when restoring a hyperspectral image, the target image and the restored image each include a plurality of images corresponding to a plurality of wavelength bands, but in Fig.14 Only an image corresponding to one wavelength band is illustrated. Images corresponding to a plurality of wavelength bands may also be displayed. The user can change the first parameter used to generate the target image and the optimization cycle end condition of the restored image while viewing the error image. Thus, the first parameter and the optimization cycle end condition can be changed to minimize the error in the region of the important subject.

[0160] Thus, in this variation, the processor 210 repeats the process of correcting the second parameter group based on the restored target image and the restored image, and generating the restored image using the corrected second parameter group, for a predetermined number of times, unless the termination condition is satisfied. The processor 210 calculates the error evaluation value between the restored image and the restored target image after the predetermined number of repetitions. When the error evaluation value is greater than the threshold, the processor 210 causes the display device 520 to display at least one of urging the user to re-enter the first parameter group, change the predetermined number of times, and change the termination condition. Thus, when the difference between the restored image and the restored target image is large, the restored target image can be regenerated with a more appropriate value of the first parameter group, or the predetermined number of times can be changed, or the termination condition can be changed, so as to generate a more appropriate restored image.

[0161] In addition, the processor 210 extracts a first region in the restored target image, extracts a second region corresponding to the first region in the restored image, and determines an error evaluation value based on a difference between the first region and the second region. The first region can be specified by a user, for example. Thus, for example, the second parameter group can be optimized to generate a restored image in which the error of a region with a higher importance is reduced.

[0162] In this embodiment, a hyperspectral image is restored from a compressed image, but the application scope of the technology disclosed in this disclosure is not limited to the use of restoring a hyperspectral image. For example, the technology disclosed in this disclosure can be applied to the use of restoring a restored image with a higher resolution from a low-resolution compressed image, the use of restoring an MRI image from a compressed image, or the use of restoring a three-dimensional image from a compressed image.

[0163] [5. Notes]

[0164] The following techniques are disclosed through the description of the above embodiments.

[0165] (Technology 1)

[0166] A signal processing method is a signal processing method executed using a computer, comprising: obtaining a first parameter group and a restoration matrix used in a first restoration process for generating a restored target image based on the compressed image; generating the restored target image based on the compressed image, the first parameter group and the restoration matrix; obtaining a second parameter group used in a second restoration process for generating a restored image based on the compressed image; generating the restored image based on the compressed image, the second parameter group and the restoration matrix; and correcting the second parameter group based on the restored target image and the restored image.

[0167] This configuration allows the second parameter group to be modified appropriately and efficiently. As a result, the quality of the restored image generated by the second restoration process can be improved.

[0168] (Technique 2)

[0169] In the signal processing method described in technique 1, the second parameter group includes more parameters than the first parameter group.

[0170] With this configuration, the second parameter group including more parameters than the first parameter group can be corrected appropriately and efficiently.

[0171] (Technique 3)

[0172] In the signal processing method described in technique 1 or 2, correcting the second parameter group includes correcting one or more values ​​corresponding to one or more parameters included in the second parameter group so that the restored image becomes close to the restored target image.

[0173] With this configuration, the second parameter group is appropriately corrected, and a restored image that is closer to the restoration target image can be generated.

[0174] (Technique 4)

[0175] As in the signal processing method described in Technology 3, correcting the above-mentioned second parameter group includes: calculating the error evaluation value between the above-mentioned target restored image and the above-mentioned restored image; and correcting one or more values ​​corresponding to one or more parameters included in the above-mentioned second parameter group to minimize the above-mentioned error evaluation value.

[0176] With this configuration, the second parameter group can be corrected more appropriately, and a restored image with a smaller error can be generated.

[0177] (Technique 5)

[0178] In the signal processing method described in any one of techniques 1 to 4, the first restoration process is performed by a first algorithm; the second restoration process is performed by a second algorithm; and the second algorithm has a smaller computational load than the first algorithm.

[0179] With this configuration, it is possible to generate a restored image in a shorter time using the second algorithm having a smaller calculation load than the first algorithm.

[0180] (Technique 6)

[0181] In the signal processing method described in Technology 5, the first restoration processing does not include processing based on a learned model trained through machine learning; and the second restoration processing includes processing based on a learned model trained through machine learning.

[0182] With this configuration, it is possible to generate a restored image in a shorter time by the second restoration process including the process based on the learned model.

[0183] (Technique 7)

[0184] In the signal processing method described in technique 6, the first restoration process includes an iterative operation of minimizing or maximizing an evaluation function based on the compressed image and the restoration matrix.

[0185] With this configuration, a more accurate restored target image can be generated, so the second parameter group can be corrected to more appropriate values.

[0186] (Technology 8)

[0187] In the signal processing method described in any one of techniques 1 to 7, the compressed image is an image that encodes spectral information of the object; and the restored target image and the restored image respectively include information of multiple images corresponding to multiple wavelength bands.

[0188] With this configuration, it is possible to appropriately generate a restored image including information on a plurality of images corresponding to a plurality of wavelength bands.

[0189] (Technique 9)

[0190] The signal processing method as described in any one of techniques 1 to 8 includes: determining the final second parameter group by repeatedly performing the processing of correcting the second parameter group based on the restored target image and the restored image and generating the restored image using the corrected second parameter group.

[0191] With this configuration, the second parameter group can be corrected to more appropriate values, thereby improving the accuracy of the restored image.

[0192] (Technology 10)

[0193] The signal processing method as described in any one of techniques 1 to 9 also includes: causing a display device to display a graphical user interface (GUI) for a user to input the above-mentioned first parameter group.

[0194] With this configuration, the user can adjust the first parameter group, so a more appropriate restored target image can be generated. As a result, the second parameter group can be corrected to a more appropriate value, and the accuracy of the restored image can be improved.

[0195] (Technology 11)

[0196] The signal processing method as described in Technology 10 also includes: repeating the processing based on the above-mentioned restored target image and the processing of correcting the above-mentioned second parameter group based on the above-mentioned restored image and generating the above-mentioned restored image using the corrected above-mentioned second parameter group for a specified number of times as long as the termination condition is not satisfied; calculating the error evaluation value between the above-mentioned restored image and the above-mentioned restored target image after the above-mentioned specified number of repetitions; and when the above-mentioned error evaluation value is larger than a threshold value, causing the above-mentioned display device to display at least one GUI urging the above-mentioned user to re-enter the above-mentioned first parameter group, change the above-mentioned specified number of times, and change the above-mentioned termination condition.

[0197] This configuration makes it possible to correct the second parameter group to more appropriate values, thereby generating a more accurate restored image.

[0198] (Technology 12)

[0199] As described in the signal processing method of technology 11, calculating the above-mentioned error evaluation value includes: extracting the first area in the above-mentioned restored target image; extracting the second area corresponding to the above-mentioned first area in the above-mentioned restored image; and determining the above-mentioned error evaluation value based on the difference between the above-mentioned first area and the above-mentioned second area.

[0200] With this configuration, it is possible to reduce the restoration error regarding the subject included in the extracted region.

[0201] (Technology 13)

[0202] A signal processing device comprises: one or more processors; and a memory storing a computer program executed by the one or more processors; the computer program causes the one or more processors to execute: obtaining a compressed image containing compressed information of a subject; obtaining a first parameter group and a restoration matrix used in a first restoration process for generating a restored target image based on the compressed image; generating the restored target image based on the compressed image, the first parameter group and the restoration matrix; obtaining a second parameter group used in a second restoration process for generating a restored image based on the compressed image; generating the restored image based on the compressed image, the second parameter group and the restoration matrix; and correcting the second parameter group based on the restored target image and the restored image.

[0203] This configuration allows the second parameter group to be modified appropriately and efficiently. As a result, the quality of the restored image generated by the second restoration process can be improved.

[0204] (other)

[0205] Modifications of the embodiments of the present disclosure may be as follows.

[0206] The method according to item 1 is a method executed by a computer, wherein the computer

[0207] (a) receiving a first image of a subject captured through a filter array including four or more filters, wherein the four or more filters correspond one-to-one to four or more spectral spectra, and the four or more spectral spectra are different from each other;

[0208] (b) determining a plurality of first pixel values ​​of a first restored target image corresponding to a first wavelength domain, and a plurality of n-th pixel values ​​of an n-th restored target image corresponding to an n-th wavelength domain based on a restoration matrix corresponding to the filter array and a plurality of pixel values ​​of the first image;

[0209] (c) determining a plurality of weight values ​​of a neural network based on the plurality of pixel values ​​of the first image, the plurality of first pixel values, 1 to 2, and the plurality of n-th pixel values;

[0210] The plurality of weight values ​​respectively correspond to two directly connected units, and the neural network includes the two units;

[0211] The plurality of first pixel values, to the plurality of n-th pixel values ​​are used as training data corresponding to a plurality of output values ​​output by the neural network when the plurality of pixel values ​​of the first image are input to the neural network;

[0212] (d) generating an image corresponding to the first wavelength region of the second object or an image corresponding to the n-th wavelength region of the second object based on a plurality of output values ​​from the neural network for a plurality of input values ​​to the neural network;

[0213] The plurality of input values ​​are a plurality of pixel values ​​of an image of the second object captured via the filter array;

[0214] The neural network generates the plurality of output values ​​using the plurality of input values ​​and the plurality of determined weight values ​​without using the restoration matrix.

[0215] Industrial Applicability

[0216] The technology disclosed herein is applicable to cameras and measuring equipment that obtain multi-wavelength or high-resolution images, for example. The technology disclosed herein can also be applied to, for example, biological / medical / cosmetic sensing, food foreign matter / pesticide residue inspection systems, remote sensing systems, and vehicle-mounted sensing systems.

[0217] Description of symbols

[0218] 100 Camera Device

[0219] 110 filter array

[0220] 140 Optical System

[0221] 160 Image Sensor

[0222] 200 Signal processing device

[0223] 210 processor

[0224] 250 Memory

[0225] 300 Display device

[0226] 400 Conveyor

[0227] 500 Terminal Devices

[0228] 510 Input Device

[0229] 520 Display Device

[0230] 1000 Camera System

Claims

1. A signal processing method is a signal processing method executed by a computer, characterized in that: include: obtaining a compressed image including compressed information of the subject; Acquiring a first parameter group and a restoration matrix used in a first restoration process for generating a restored target image from the compressed image; Based on the compressed image, the first parameter group and the restoration matrix, generating the restored target image; acquiring a second parameter group used in a second restoration process for generating a restored image from the compressed image; generating the restored image based on the compressed image, the second parameter group and the restoration matrix; and Based on the restoration target image and the restoration image, the second parameter group is corrected.

2. The signal processing method according to claim 1, characterized in that: The second parameter group includes more parameters than the first parameter group.

3. The signal processing method according to claim 1 or 2, characterized in that: Correcting the second parameter group includes correcting one or more values ​​corresponding to one or more parameters included in the second parameter group so that the restored image becomes close to the restored target image.

4. The signal processing method according to claim 3, characterized in that: Modifications to the above second parameter group include: Calculating an error evaluation value between the target restored image and the restored image; and One or more values ​​corresponding to one or more parameters included in the second parameter group are corrected so as to minimize the error evaluation value.

5. The signal processing method according to claim 1 or 2, characterized in that: The first restoration process is performed by a first algorithm; The second restoration process is performed by a second algorithm; The second algorithm has a smaller calculation load than the first algorithm.

6. The signal processing method according to claim 5, characterized in that: The first restoration process does not include a process based on a learned model trained by machine learning; The second restoration process includes a process based on a learned model trained through machine learning.

7. The signal processing method according to claim 6, characterized in that: The first restoration process includes an iterative calculation for minimizing or maximizing an evaluation function based on the compressed image and the restoration matrix.

8. The signal processing method according to claim 1 or 2, characterized in that: The compressed image is an image that encodes the spectral information of the object; The restoration target image and the restored image each include information of a plurality of images corresponding to a plurality of wavelength bands.

9. The signal processing method according to claim 1 or 2, characterized in that: include: The final second parameter group is determined by repeatedly performing a process of correcting the second parameter group based on the restoration target image and the restoration image and a process of generating the restoration image using the corrected second parameter group.

10. The signal processing method according to claim 1 or 2, characterized in that: Also includes: The display device is caused to display a GUI for a user to input the first parameter group, wherein the GUI is a graphical user interface.

11. The signal processing method according to claim 10, characterized in that: Also includes: As long as the termination condition is not satisfied, the process of correcting the second parameter group based on the restored target image and the restored image and the process of generating the restored image using the corrected second parameter group are repeated a predetermined number of times; calculating an error evaluation value between the restored image after the predetermined number of repetitions and the restored target image; and When the error evaluation value is larger than a threshold value, the display device is caused to display a GUI that urges the user to re-input the first parameter group, change the predetermined number of times, and change the end condition.

12. The signal processing method according to claim 11, characterized in that: Calculating the above error evaluation value includes: Extracting the first region in the restored target image; extracting a second region corresponding to the first region in the restored image; and The error evaluation value is determined based on the difference between the first area and the second area.

13. A signal processing device, characterized in that: have: More than 1 processor; and A memory storing a computer program executed by the one or more processors; The computer program causes the one or more processors to execute: obtaining a compressed image including compressed information of the subject; Acquiring a first parameter group and a restoration matrix used in a first restoration process for generating a restored target image from the compressed image; Based on the compressed image, the first parameter group and the restoration matrix, generating the restored target image; acquiring a second parameter group used in a second restoration process for generating a restored image from the compressed image; generating the restored image based on the compressed image, the second parameter group and the restoration matrix; and Based on the restoration target image and the restoration image, the second parameter group is corrected.

Images