Apparatus and method for processing spectral data of an image sensor

Abstract

A method and apparatus for processing spectral data of an image sensor are provided. The method includes obtaining spectral response signals corresponding to channels of spectral data of light, the spectral data being obtained from an object by an image sensor; determining a set of bases corresponding to the obtained spectral response signals; performing a basis transformation on at least one basis included in the determined set of bases based on the determined set of bases; and generating reconstructed spectral data from the spectral response signals on which the basis transformation has been performed by using a pseudo-inverse.

Description

[0001] Cross-reference to related applications

[0002] This application is based on and claims priority to Korean Patent Application No. 10-2021-0141643, filed with the Korean Intellectual Property Office on October 22, 2021, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] One or more exemplary embodiments of this disclosure relate to an apparatus and method for processing spectral data from an image sensor. Background Technology

[0004] Hyperspectral imaging combines spectral analysis and image processing to simultaneously acquire spatial and spectral information about the object under test. Hyperspectral imaging devices can identify the state, configuration, characteristics, and changes of an object using data obtained from hyperspectral image sensors, thereby identifying materials and / or measuring the degree of defects in products. However, in hyperspectral imaging systems, the resolution of the image sensor (e.g., spatial and / or spectral resolution) can decrease due to crosstalk between pixels as the pixel size of the image sensor decreases. Therefore, various image processing methods have been explored to compensate for crosstalk in order to enable image sensors to obtain high-resolution image data. Summary of the Invention

[0005] An apparatus and method for processing spectral data from an image sensor are provided. The purpose of this disclosure is not limited to the contents described herein, and other technical objectives can be inferred from the following embodiments.

[0006] Additional aspects will be set forth in part in the description which follows, and will also be apparent in part from the description, or may be learned by practicing the embodiments presented in this disclosure.

[0007] According to one aspect of an example embodiment, a method for processing spectral data is provided, the method comprising: obtaining a spectral response signal corresponding to a channel of spectral data of light, the spectral data being obtained from an object by an image sensor; determining a set of bases corresponding to the obtained spectral response signal; performing a basis transformation on at least one basis included in the determined set of bases based on the determined set of bases; and generating reconstructed spectral data from the spectral response signal for which the basis transformation has been performed by using a pseudo-inverse.

[0008] Determining a set of bases can include obtaining the principal components of the spectral response signal for each channel by using a principal component analysis (PCA) algorithm.

[0009] This set of bases may include bases whose number is less than or equal to the number of channels.

[0010] Performing a basis transformation may include: determining at least one basis with singular values ​​less than a threshold among the basis corresponding to the obtained spectral response signal; and replacing the singular values ​​of the determined at least one basis with specific singular values.

[0011] A particular singular value can be a threshold, or the minimum of the singular values ​​of other bases in the basis corresponding to the obtained spectral response signal, wherein each other basis has a singular value greater than or equal to the threshold.

[0012] The method may also include generating a hyperspectral image based on the generated reconstructed spectral data.

[0013] Generating hyperspectral images can include generating RGB images, which are obtained by color conversion from reconstructed spectral data using a color matching function.

[0014] Spectral data can have three or more channels.

[0015] The threshold can be determined based on the nth singular value among the singular values ​​of the basis corresponding to the obtained spectral response signal, arranged in ascending order, where n is a natural number greater than 0.

[0016] According to one aspect of a preferred embodiment, a spectral data processing apparatus is provided, the apparatus comprising: a light source configured to illuminate an object with light; an image sensor configured to acquire a spectral response signal corresponding to a channel of spectral data of the light, the spectral data being acquired from the object by the image sensor; and a processor configured to: determine a set of bases corresponding to the acquired spectral response signal; perform a basis transformation on at least one basis included in the determined set of bases based on the determined set of bases; and generate reconstructed spectral data from the spectral response signal for which the basis transformation has been performed by using a pseudo-inverse.

[0017] A set of basis components can be determined by using principal component analysis (PCA) to obtain the principal components of the spectral response signal for each channel.

[0018] This set of bases may include bases whose number is less than or equal to the number of channels.

[0019] The processor may be further configured to: determine at least one basis having singular values ​​less than a threshold among the basis corresponding to the obtained spectral response signal, and replace the singular values ​​of the determined at least one basis with specific singular values.

[0020] A particular singular value can be a threshold, or the minimum of the singular values ​​of other bases in the basis corresponding to the obtained spectral response signal, wherein each other basis has a singular value greater than or equal to the threshold.

[0021] The processor can be further configured to generate hyperspectral images based on the generated reconstructed spectral data.

[0022] The processor can also be configured to generate hyperspectral images by generating RGB images, which are obtained by color conversion from reconstructed spectral data using a color matching function.

[0023] Image sensors may include multispectral image sensors configured to acquire spectral data from three or more channels.

[0024] The threshold can be determined based on the nth singular value among the singular values ​​of the basis corresponding to the obtained spectral response signal, arranged in ascending order, where n is a natural number greater than 0.

[0025] According to one aspect of an example embodiment, a non-transitory computer-readable recording medium is provided, on which is recorded a program for performing a method of processing spectral data, the method comprising: obtaining a spectral response signal corresponding to a channel of spectral data of light, the spectral data being obtained from an object by an image sensor; determining a set of bases corresponding to the obtained spectral response signal; performing a basis transformation on at least one basis included in the determined set of bases based on the determined set of bases; and generating reconstructed spectral data from the spectral response signal on which the basis transformation has been performed by using a pseudo-inverse. Attached Figure Description

[0026] This patent or application document contains at least one color drawing. A copy of this patent or application disclosure containing color drawings will be provided by the official authority upon request and payment of the necessary fees.

[0027] The above and other aspects, features, and advantages of some embodiments of this disclosure will become clearer from the following description taken in conjunction with the accompanying drawings, in which:

[0028] Figure 1 This is a block diagram illustrating an example of an imaging device according to some example embodiments;

[0029] Figure 2 These are conceptual diagrams of an imaging device based on some example embodiments;

[0030] Figure 3 yes Figure 2 A conceptual diagram of an example light source;

[0031] Figure 4 yes Figure 2 A conceptual diagram of an example image sensor;

[0032] Figure 5 It is shown Figure 2 A conceptual diagram of an example image sensor;

[0033] Figure 6 This is a schematic diagram illustrating the operation of an imaging device according to some example embodiments;

[0034] Figure 7 This is a flowchart of a method for processing spectral data from an image sensor to generate reconstructed spectral data, according to some example embodiments.

[0035] Figure 8 This is a flowchart of a method for processing spectral data from an image sensor to generate reconstructed spectral data according to some example embodiments;

[0036] Figure 9 This is a graph illustrating the simulation results of spectral data reconstructed according to some example embodiments of the application;

[0037] Figure 10 This is a graph illustrating the effect of the number of bases on the performance of the reconstructed spectral data according to some example embodiments;

[0038] Figure 11 It is a graph illustrating the effect of singular values ​​of a basis according to some example embodiments; and

[0039] Figure 12 This is a diagram illustrating the RGB conversion of reconstructed spectral data according to some example embodiments. Detailed Implementation

[0040] Referring now to the embodiments, examples of which are illustrated in the accompanying drawings, wherein similar reference numerals throughout the drawings denote similar elements. In this respect, the exemplary embodiments may take different forms and should not be construed as limited to the description set forth herein. Therefore, the exemplary embodiments are described below only with reference to the accompanying drawings to explain various aspects of this disclosure. As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items. Expressions such as “at least one of…” modify the entire list of elements when following it, rather than individual elements within the list.

[0041] Although terms have been selected from commonly used terms widely in use by taking into account their function in this disclosure, these terms may differ depending on the intent of a person skilled in the art, precedent, or the emergence of new technologies. Furthermore, in certain cases, terms are chosen at the discretion of the applicant of this disclosure, in which case the meaning of these terms will be described in detail in the corresponding sections of the detailed description. Therefore, the terms used in this disclosure are not merely names of terms, but terms defined according to their meaning throughout the terminology and content of this disclosure.

[0042] In the description of the embodiments, it should be understood that when an element is referred to as being "connected to" another element, it can be "directly connected to" the other element or "electrically connected" to the other element via an intermediate element. The singular expression also includes the plural meaning, provided it does not contradict the context. When an element is referred to as "comprising" a component, the element can additionally include other components rather than exclude them, unless specifically stated otherwise.

[0043] Terms such as “comprising” or “including” as used herein should not be construed as including all the various elements or operations described herein, and it should be understood that some elements or operations may be omitted or additional elements or operations may be provided.

[0044] Furthermore, although this document may use terms such as “first” or “second” to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another.

[0045] It should be understood that the scope of the exemplary embodiments is not limited to the description of the following specific embodiments, and matters that can be readily conceived by those skilled in the art fall within the scope of the exemplary embodiments. In the following, exemplary embodiments will be described in detail with reference to the accompanying drawings.

[0046] Figure 1 This is a block diagram illustrating an example of an imaging device according to some example embodiments.

[0047] Reference Figure 1 The imaging device 10 can be any device capable of analyzing the characteristics of an object or identifying an object. For example, the imaging device 10 can correspond to an imaging system using hyperspectral or multispectral imaging techniques, but is not limited thereto. The object can include people, objects, terrain, plants, etc., and can be varied without limitation depending on the field where the imaging device 10 can be used.

[0048] Reference Figure 1 The imaging device 10 may include at least one light source 110, at least one image sensor 120, and at least one processor 130. However, in Figure 1 The imaging device 10 only shows components relevant to some example embodiments. Therefore, it will be apparent to those skilled in the art that the imaging device 10, in addition to… Figure 1 In addition to the components shown, other general-purpose components may also be included. For example, the imaging device 10 may also include hardware components such as a light source and a memory.

[0049] Furthermore, even if only including Figure 1Some of the components shown may correspond to imaging device 10, but imaging devices capable of achieving the purposes of this disclosure may also be used. For example, imaging device 10 may include only at least one image sensor 120 and at least one processor 130, and at least one light source 110 may be external to imaging device 10.

[0050] The memory that may be included in the imaging device 10 can be hardware used to store various data processed by the imaging device 10, and the memory may, for example, store data that has already been processed by the imaging device and data that will be processed by the imaging device. In addition, the memory may store applications, drivers, etc. that will be executed by the imaging device 10.

[0051] The memory may include random access memory (RAM), such as dynamic RAM (DRAM) or static SRAM, read-only memory (ROM), electrically erasable programmable ROM (EEPROM), compact disc-ROM (CD-ROM), Blu-ray or other optical disc storage, hard disk drive (HDD), solid-state drive (SSD) or flash memory, and may also include other external storage devices accessible by the imaging device 10.

[0052] At least one light source 110 can refer to a device for illuminating an object with light. At least one light source 110 can illuminate the object with light of multiple different wavelengths. For example, at least one light source 110 can selectively illuminate the object with light of a first wavelength (e.g., the range of 400 nm to 700 nm) and light of a second wavelength (e.g., the range of 250 nm to 400 nm), which is different from the first wavelength. However, the wavelengths of the light illuminated by at least one light source 110 can be set differently.

[0053] At least one light source 110 may be a light-emitting diode (LED) or a fluorescent lamp emitting broad-spectrum light in the visible light band, or a laser diode emitting high-intensity short-wavelength light. However, this disclosure is not limited thereto. At least one light source 110 may emit light in a wavelength band suitable for obtaining information about the object.

[0054] Furthermore, at least one light source 110 may be a single light source capable of selectively irradiating an object with light of multiple different wavelengths. However, this disclosure is not limited thereto, and at least one light source 110 may include multiple light sources, each irradiating light of one wavelength. Furthermore, at least one light source 110 may include multiple light sources capable of selectively irradiating an object with light of multiple different wavelengths.

[0055] At least one image sensor 120 can refer to a device for obtaining spectral data from light scattered, emitted, or reflected by or absorbed by an object. However, this disclosure is not limited thereto; at least one image sensor 120 can obtain spectral data from light transmitted through or refracted by an object. In one example, when light generated by light source 110 or natural light shines on an object, the object may absorb, scatter, emit, or reflect the light, and the image sensor 120 can measure the spectral data of the light absorbed, scattered, emitted, or reflected by the object. Because the spectral data may vary depending on the type of material constituting the object, the type of material constituting the object can be estimated by analyzing the measured spectral data.

[0056] The spectral data acquired by the image sensor 120 may include spectral data (or spectral signals) from multiple channels corresponding to a given wavelength. Here, the bandwidth and number of channels for each wavelength can be set differently. For example, the image sensor 120 may be a multispectral image sensor capable of acquiring spectral data from at least three channels.

[0057] According to some example embodiments, the spectral data obtained by the image sensor 120 may include at least one of a variety of spectra, such as Raman spectroscopy, visible spectroscopy, fluorescence spectroscopy, microwave spectroscopy, infrared spectroscopy, X-ray spectroscopy, etc., but this disclosure is not limited thereto. Here, Raman spectroscopy may refer to the spectrum obtained when a high-intensity, short-wavelength light is irradiated onto an object using a light source such as a laser diode, and then the light scattered or emitted from the object is measured at a band different from the band of the light source.

[0058] although Figure 1 The illustration shows an imaging device 10 comprising only one image sensor 120, but the imaging device 10 may include two or more image sensors. In the case where the imaging device 10 includes two or more image sensors, the types of spectral data obtained by the respective image sensors may differ from each other.

[0059] Image sensor 120 can acquire spectral data by using at least one of a grating and a filter array. A grating can correspond to a device that performs spectroscopy by using the refraction, reflection, or diffraction of light, while a filter array can correspond to a device that performs spectroscopy by using filters that selectively transmit or block light of a specific wavelength or wavelength range.

[0060] Image sensor 120 can acquire a visible or hyperspectral image of an object by measuring light in, for example, the wavelength range of 300 nm to 700 nm. However, this disclosure is not limited thereto, and at least one image sensor 120 can measure light in any wavelength range suitable for obtaining information about the object.

[0061] Image sensor 120 may include a photodiode array, a charge-coupled device (CCD) sensor, or a complementary metal-oxide-semiconductor (CMOS) sensor, capable of acquiring visible images including information about appearance, such as the shape and color of an object. Furthermore, image sensor 120 may acquire hyperspectral or multispectral images including information related to fluorescence emitted from the object. When light source 110 emits light toward the object, certain indicator materials of the object emit fluorescence, and parameters indicating the amount of indicator materials can be obtained from the hyperspectral or multispectral images. Although examples of hyperspectral imaging will be described below, this disclosure is not limited thereto, and example embodiments may be implemented, for example, by using multispectral and hyperspectral imaging techniques.

[0062] At least one processor 130 is used to perform overall functions for controlling the imaging device 10. For example, at least one processor 130 can control the operation of at least one light source 110 and at least one image sensor 120. At least one processor 130 can be implemented as an array of multiple logic gates, or it can be implemented as a combination of a general-purpose microprocessor and a memory storing programs that can be executed by the microprocessor.

[0063] At least one processor 130 can obtain first information about an object based on a visible image obtained by at least one image sensor 120, and obtain second information about the object based on the obtained first information and a hyperspectral image obtained by the image sensor 120. As described above, the processor 130 can improve the accuracy of the second information about the object by comprehensively considering the information obtained from analyzing the visible image instead of using only the information obtained from analyzing the hyperspectral image to obtain the second information.

[0064] For example, the first information may include the object's appearance information, and the second information may include the object's material parameters. However, this disclosure is not limited thereto.

[0065] Figure 2 This is a conceptual diagram of an imaging device 10 based on some example embodiments. Figure 3 yes Figure 2 A conceptual diagram of the light source 110. Figure 4 yes Figure 2 A conceptual diagram of an example image sensor 120. Figure 5 It is shown Figure 2 A conceptual diagram of an example image sensor 120.

[0066] Reference Figures 2 to 5 The imaging device 10 may include a light source 110, an image sensor 120, and a processor 130, and can perform imaging of the object OBJ.

[0067] Light source 110 can be directed towards object OBJ. Light source 110 can emit inspection rays ILa and ILb towards object OBJ. For example... Figure 3 As shown, the light source 110 may include a first light source array 1101, a second light source array 1102, and a transmission window 1103. The first light source array 1101 and the second light source array 1102 may be spaced apart from each other, with the transmission window 1103 located between them. Each of the first light source array 1101 and the second light source array 1102 may include a plurality of light sources 1100 arranged along one direction. For example, the plurality of light sources 1100 may be arranged along the transmission window 1103. Although Figure 3 The illustration shows multiple light sources 1100 in each of the first light source array 1101 and the second light source array 1102 arranged in two columns, but this disclosure is not limited thereto. For example, the multiple light sources 1100 may include LEDs.

[0068] An object OBJ exposed to inspection light ILa and ILb can emit fluorescence OL. The fluorescence OL can pass through transmission window 1103 and reach the interior of imaging device 10. Transmission window 1103 may include a transparent material. For example, transmission window 1103 may include transparent plastic or glass. In this example, transmission window 1103 may include a material with high durability at low temperatures. The fluorescence OL can be provided to image sensor 120. For example, the optical path of the fluorescence OL can be adjusted by optical path adjustment element 1140, thereby providing the fluorescence OL to image sensor 120. Image sensor 120 may include a hyperspectral camera.

[0069] In the example, image sensor 120 may include, for example, Figure 4 The dispersive element 230 is shown. Image sensor 200a (see...) Figure 4 The system may include a slit element 210, a collimating lens 220, a dispersive element 230, a condenser lens 240, and a sensor 250. The slit element 210 can be used to extract a desired portion of the fluorescent OL. In this example, the fluorescent OL that has passed through the slit element 210 can be dispersed. The collimating lens 220 can adjust the size of the fluorescent OL to convert it into parallel or converging light. For example, the collimating lens 220 may include a convex lens. The dispersive element 230 can separate the fluorescent OL provided from the collimating lens 220. Although... Figure 4The dispersive element 230 is shown as a grating, but this disclosure is not limited thereto. In another example embodiment, the dispersive element 230 may be a prism. The separated fluorescence OL may pass through a condenser lens 240 and then be provided to the sensor 250. For example, the condenser lens 240 may include a convex lens. The separated fluorescence OL may be provided to different positions of the sensor 250 according to wavelength. The sensor 250 may measure the separated fluorescence OL provided from the dispersive element 230. The sensor 250 may generate a spectral signal of the fluorescence OL. The sensor 250 may provide the spectral signal to the processor 130.

[0070] In the example, such as Figure 5 As shown, an image sensor 200b can be provided, including a spectral filter 232. The image sensor 200b may include a slit element 210, a collimating lens 220, a spectral filter 232, a condenser lens 240, and a sensor 250. The slit element 210, collimating lens 220, condenser lens 240, and sensor 250 can be connected to a reference... Figure 4 The descriptions are essentially the same. Spectral filter 232 can be a set of filters that allow light of different wavelengths to pass through separately. Spectral filter 232 can filter the fluorescence OL provided from collimating lens 220 to have spatially different wavelengths. That is, portions of the fluorescence OL that have passed through different regions of spectral filter 232 can have different wavelengths. The fluorescence OL separated by spectral filter 232 can pass through condenser lens 240 and then be provided to sensor 250. Sensor 250 can generate a spectral signal of the fluorescence OL. Sensor 250 can provide the spectral signal to processor 130.

[0071] Processor 130 can generate a hyperspectral image of object OBJ based on the spectral signal. Processor 130 can determine the state of object OBJ by using the hyperspectral image of object OBJ.

[0072] Figure 6 This is a schematic diagram illustrating the operation of an imaging device according to some example embodiments.

[0073] Reference Figure 6 The image sensor 120 can acquire the spectral data (or spectral signal) of an object by using a filter array.

[0074] Spectral data obtained by image sensor 120 may include, for example, data at wavelengths λ at N+1 (where N is an integer equal to or greater than 1). 10 ,...,λ 1NInformation related to the intensity of light at a given location. The processor 130 of the imaging device 10 can perform image processing 620 on the acquired spectral data. Here, image processing 620 can refer to processing for obtaining a hyperspectral image 630 of the object. Furthermore, image processing 620 can include a spectral reconstruction process for generating reconstructed spectral data from the input spectral data.

[0075] Image sensor 120 may include multiple pixels. These multiple pixels may each correspond to multiple wavelength bands, or may include multiple sub-pixels each corresponding to multiple wavelength bands. For example, a first sub-pixel may measure light in a first wavelength band, and a second sub-pixel may measure light in a second wavelength band different from the first wavelength band.

[0076] To increase the resolution (e.g., spatial or spectral resolution) of the image sensor 120, it may be necessary to achieve a small pixel size. However, as the pixel size of the image sensor 120 decreases, crosstalk may occur between adjacent pixels. Therefore, to image at an appropriate resolution, crosstalk effects need to be compensated for. In particular, the spatial and spectral resolutions of the image sensor 120 used for hyperspectral imaging are inversely proportional to each other. Therefore, to enable the image sensor 120 to perform hyperspectral imaging with appropriate spatial and spectral resolutions, methods for compensating for crosstalk effects can be used.

[0077] Figure 7 This is a flowchart of a method for processing spectral data from an image sensor to generate reconstructed spectral data, according to some example embodiments. Figure 7 The method for processing spectral data can be performed by the imaging device 10 described above.

[0078] In operation 701, processor 130 obtains a spectral response signal (or spectral response function) corresponding to each channel of the spectral data obtained by image sensor 120.

[0079] The spectral data obtained by the image sensor 120 may include spectral data (or spectral signals) of multiple channels corresponding to the corresponding wavelengths, and the spectral data may include spectral response signals indicating the spectral characteristics corresponding to the corresponding multiple channels.

[0080] Specifically, the spectral response signal of each channel can be represented by Equation 1.

[0081] [Equation 1]

[0082]

[0083] In Equation 1, f(λ), I(λ), and D represent the spectrum of the filter, the spectrum of the incident light, and the intensity of the detected light, respectively. Additionally, L represents the number of channels for the spectral data.

[0084] Equation 1 regarding the spectral response signal of each channel can be represented in matrix form by Equation 2.

[0085] [Equation 2]

[0086]

[0087] In operation 702, processor 130 determines the basis set corresponding to the spectral response signal. Here, processor 130 can use a principal component analysis (PCA) algorithm. Specifically, processor 130 can obtain the principal components of the spectral response signal for each channel by performing PCA, and determine the basis set for the channels based on the obtained principal components.

[0088] The determined basis set can include a number of bases equal to the number of channels in the spectrum. As another example, the determined basis set can include a number of bases fewer than the number of channels in the spectrum. As the number of bases increases, the spectral resolution can increase, but the image sensor 120 may become more sensitive to noise. Conversely, as the number of bases decreases, the image sensor 120 may become relatively less sensitive to noise, but the spectral resolution may decrease. Therefore, the performance of the reconstructed spectral data can be tuned by optimizing the number of bases to be included in the basis set, depending on the characteristics of the imaging device.

[0089] In operation 703, processor 130 performs a basis transformation on the spectral response signals of the channels based on the determined basis set. Since a basis transformation is performed on the spectral response signal of each channel, the spectral response signals of each channel can be orthogonal. Therefore, crosstalk between channels can be reduced, thereby reducing errors in spectral reconstruction using pseudo-inverses.

[0090] In operation 704, processor 130 generates reconstructed spectral data from the spectral response signal that has already undergone a basis transformation by using a pseudo-inverse. That is, processor 130 can generate reconstructed spectral data compensated for crosstalk effects from the input spectral data by performing a basis transformation on the spectral data obtained by image sensor 120 and using a pseudo-inverse.

[0091] Figure 8 This is a flowchart of a method for processing spectral data from an image sensor to generate reconstructed spectral data, according to some example embodiments. Figure 8 The method can also be applied to Figure 7 Operations 702 to 704.

[0092] In operation 801, processor 130 obtains the principal components of the spectral response signal for each channel by using the PCA algorithm, and obtains the basis corresponding to the spectral response signal of the channel based on the obtained principal components.

[0093] In operation 802, processor 130 may perform the following processing scheme to select only the meaningful (or optimal) basis from the obtained basis and perform basis transformation.

[0094] Specifically, in order to select only meaningful bases, matrix F can be represented as singular value decomposition (SVD), as shown in Equation 3.

[0095] [Equation 3]

[0096] F=U∑V T

[0097] In Equation 3, ∑ represents an N×L dimensional diagonal matrix, U, V T Let F represent an N×N, L×L unitary matrix, and SVD is feasible for any matrix F.

[0098] To determine a unique SVD, the diagonal elements of the matrix ∑ are typically arranged in descending order of size, and the values ​​of the diagonal elements can be defined as singular values.

[0099] In the base transformed into unitary matrices U and V, the spectral response signal can be represented by Equation 4.

[0100] [Equation 4]

[0101]

[0102]

[0103] When using the inverse of the above matrix, small singular values ​​can amplify noise. Therefore, to eliminate small singular values, it may be desirable to use only a few singular values ​​that are greater than or equal to a preset size. Thus, a truncation matrix can be used to remove small singular values ​​that may cause noise.

[0104] As shown in Equation 5, the traditional PCA method is a dimensionality reduction method that replaces all small singular values ​​with 0 and removes the singular eigenvectors corresponding to the small singular values.

[0105] [Equation 5]

[0106]

[0107] However, in the example embodiment, instead of treating small singular values ​​as 0, a method is used to replace the previous minimum value equally by using a truncation matrix of Equation 6, which is modified from conventional PCA (e.g., Equation 5), thus avoiding dimensionality reduction. Therefore, the method of the example embodiment can minimize the information lost due to dimensionality reduction compared to conventional methods (e.g., Equation 5).

[0108] [Equation 6]

[0109]

[0110] The processor 130 can execute the above processing method to determine the number of bases with singular values ​​of a preset threshold or greater among the bases obtained in operation 801 and the singular values ​​of the bases, thereby determining the basis set.

[0111] As another example, when there exists a basis with singular values ​​less than or equal to a threshold, the processor 130 can determine the basis set by replacing the singular values ​​of the basis with the threshold or by replacing them with the minimum of the singular values ​​greater than or equal to the threshold.

[0112] That is, the processor 130 can determine at least one basis with singular values ​​less than a threshold among the basis corresponding to the obtained spectral response signal, and replace the singular values ​​of the determined at least one basis with a specific singular value. Therefore, the basis set can include at least one basis whose singular values ​​are replaced with a specific singular value. The specific singular value can be the threshold, or the minimum of the singular values ​​of other basis bases having singular values ​​greater than or equal to the threshold.

[0113] In operation 803, processor 130 performs a basis transformation on the spectral response signal of the channel based on the determined basis set, and generates reconstructed spectral data from the spectral response signal that has already undergone the basis transformation by using a pseudo-inverse. That is, processor 130 can generate reconstructed spectral data compensated for crosstalk effects from the input spectral data by performing a basis transformation on the spectral data obtained by image sensor 120 and using a pseudo-inverse.

[0114] The processor 130 can obtain the spectrum I(λ) of the incident light by using a pseudo-inverse as shown in Equation 7.

[0115] [Equation 7]

[0116] I = (F T F) -1 F T D

[0117] This can be achieved by using the pseudo-inverse of the elements truncated by the truncated matrix described in reference equation 6. To obtain the reconstructed spectrum D.

[0118] In operation 804, processor 130 can generate hyperspectral image data based on the reconstructed spectral data. For example, the reconstructed spectral data D can be converted to values ​​in the XYZ color space using Equation 8, where Equation 8 is the XYZ color matching function of the CIE color space.

[0119] [Equation 8]

[0120] X=∫dλS(λ)R(λ)x(λ)=∫dλI(λ)x(λ)

[0121] Y=∫dλS(λ)R(λ)y(λ)=∫dλI(λ)y(λ)

[0122] Z=∫dλS(λ)R(λ)z(λ)=∫dλI(λ)z(λ)

[0123] Alternatively, values ​​in the XYZ color space can be converted to values ​​in the RGB color space using Equation 9.

[0124] [Equation 9]

[0125]

[0126] The processor 130 can obtain image pixel data constituting a hyperspectral image by converting the reconstructed spectral data into values ​​in a specific color space.

[0127] That is, the processor 130 can perform imaging processing with improved resolution (e.g., spatial resolution and / or spectral resolution) by generating hyperspectral image data with reduced crosstalk effects, based on reconstructed spectral data instead of the input spectral data obtained by the image sensor 120.

[0128] Figure 9 This is a graph showing the results of simulations of spectral data reconstructed according to some example embodiments of the application.

[0129] Figure 9 A graph is shown comparing simulation results for spectral data reconstructed according to an example embodiment with simulation results for spectral data without applying the reconstruction scheme according to the example embodiment.

[0130] Graphs 901, 902, and 903 represent reconstructed spectral data from the blue, green, and red patches of a Macbeth colorimetric chart measured under fluorescent light, according to an example embodiment. The measured spectra represent the signals that have passed through 15 filters relative to the center wavelength of each filter.

[0131] Graphs 911, 912, and 913 represent spectral data obtained by measuring the blue, green, and red patches of the Macbeth colorimetric chart under fluorescent light. That is, graphs 911, 912, and 913 are based on spectral data without applying the reconstruction scheme according to the example embodiment. Similarly, the measured spectra, relative to the center wavelength of each filter, represent the signal that has passed through 15 filters.

[0132] Comparing curves 901, 902, and 903 with curves 911, 912, and 913 respectively, the curves based on the reconstructed spectral data according to the example embodiment are measured to be more similar to the reference spectrum compared to the spectral data without the application of the reconstruction scheme of the example embodiment. Therefore, it can be seen that the crosstalk effect is reduced.

[0133] Figure 10 This is a graph illustrating the effect of the number of bases on the performance of the reconstructed spectral data according to some example embodiments.

[0134] Reference Figure 10 Graphs 1001 to 1006 represent simulation results of spectral data reconstructed using different numbers of basis vectors, arranged in ascending order of their respective reference numbers. Graphs 1001 to 1006 show that as the number of basis vectors increases, spectral resolution increases, but the data becomes more sensitive to noise. Conversely, graphs 1001 to 1006 show that as the number of basis vectors decreases, the data becomes less sensitive to noise, but spectral resolution decreases. Therefore, the performance of the reconstructed spectral data can be tuned by optimizing the number of basis vectors to be included in the basis set, depending on the characteristics of the imaging device.

[0135] Figure 11 This is a graph illustrating the effects of singular values ​​of a basis according to some example embodiments.

[0136] Reference Figure 11 The graphs 1101, 1102, 1103 and 1104 show the results of spectral reconstruction performed by setting the threshold to the singular values ​​of the first, fifth, ninth and thirteenth bases, respectively, out of a total of 14 bases (i.e., bases of sizes arranged in ascending order).

[0137] As the number of bases preserving their original singular values ​​increases, spectral resolution can remain high, but the data becomes sensitive to noise. On the other hand, as the number of bases preserving their original singular values ​​decreases, spectral resolution decreases, but distortion due to noise also decreases.

[0138] Figure 12 This is a diagram illustrating the RGB conversion of reconstructed spectral data according to some example embodiments.

[0139] Figure 12The illustrations show an original image 1200 according to an example embodiment, an RGB image 1201 obtained by color conversion from reconstructed spectral data relative to the original image 1200 according to an example embodiment, and an RGB image 1202 represented using channels corresponding to the R, G, and B wavelengths without reconstruction relative to the original image 1200. Comparing the RGB image 1201 and the RGB image 1202, it can be seen that the RGB image 1201 obtained by color conversion from reconstructed spectral data according to the example embodiment can have more improvements in color representation than the RGB image 1202.

[0140] The above-described method can be provided using a non-transitory computer-readable recording medium on which one or more programs, including instructions for performing the method, are recorded. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, or magnetic tapes; optical media such as CD-ROMs or digital video discs (DVDs); magneto-optical media such as floppy disks; and hardware devices such as ROMs, RAMs, and flash memory, specifically designed for storing and executing program instructions. Examples of program instructions include not only machine code, such as code generated by a compiler, but also high-level language code executable by a computer using an interpreter or similar tool.

[0141] According to the example embodiments, at least one of the components, elements, modules, or units (collectively referred to as "components" in this paragraph) indicated by the boxes in the accompanying drawings can be embodied in various numbers of hardware, software, and / or firmware structures that perform the corresponding functions described above. According to the example embodiments, at least one of these components can use direct circuit structures, such as memory, processors, logic circuits, lookup tables, etc., which can perform the corresponding functions under the control of one or more microprocessors or other control devices. Furthermore, at least one of these components can be implemented by a module, program, or portion of code containing one or more executable instructions for performing a specific logical function and executed by one or more microprocessors or other control devices. Furthermore, at least one of these components can include or can be implemented by a processor, such as a central processing unit (CPU), microprocessor, etc., that performs the corresponding functions. Two or more of these components can be combined into a single component that performs all the operations or functions of the combined two or more components. Furthermore, at least a portion of the function of at least one of these components can be performed by another of these components. The functional scheme of the above example embodiments can be implemented as an algorithm executed on one or more processors. Furthermore, the components or processing steps indicated by the boxes can utilize any number of related techniques for electronic configuration, signal processing and / or control, data processing, etc.

[0142] It should be understood that the embodiments described herein should be considered in a descriptive sense and not for limiting purposes only. The description of features or aspects in each embodiment should generally be considered as applicable to other similar features or aspects in other embodiments. Although one or more embodiments have been described with reference to the accompanying drawings, those skilled in the art will understand that various changes in form and detail may be made without departing from the spirit and scope defined by the appended claims and their equivalents.

Claims

1. A method for processing spectral data, the method comprising: A spectral response signal corresponding to a channel of spectral data of light is obtained, the spectral data being acquired from an object by an image sensor; Determine a set of bases corresponding to the obtained spectral response signal; Based on the determined set of bases, perform a basis transformation on at least one basis included in the determined set of bases; as well as Reconstructed spectral data is generated from the spectral response signal that has undergone the basis transformation by using a pseudo-inverse. The basis transformation includes: Determine at least one basis with singular values ​​less than a threshold among the basis corresponding to the obtained spectral response signal; and Replace the singular values ​​of at least one of the identified bases with specific singular values. Wherein, the specific singular value is the threshold, or the minimum of the singular values ​​of other bases in the basis corresponding to the obtained spectral response signal, wherein each of the other bases has a singular value greater than or equal to the threshold, and The threshold is determined based on the nth singular value among the singular values ​​of the basis corresponding to the obtained spectral response signal, arranged in ascending order, where n is a natural number greater than 0.

2. The method according to claim 1, wherein, Determining the set of bases involves obtaining the principal components of the spectral response signal for each channel using a principal component analysis (PCA) algorithm.

3. The method according to claim 1, wherein, The number of bases included in the set of bases is less than or equal to the number of channels.

4. The method of claim 1 further includes generating a hyperspectral image based on the generated reconstructed spectral data.

5. The method according to claim 4, wherein, Generating the hyperspectral image includes generating an RGB image, which is obtained by color conversion from the reconstructed spectral data using a color matching function.

6. The method according to claim 1, wherein, The spectral data includes three or more channels.

7. An apparatus for processing spectral data, the apparatus comprising: The light source is configured to shine light onto the object; An image sensor is configured to acquire a spectral response signal corresponding to a channel of spectral data of light, said spectral data being acquired by the image sensor from the object; as well as The processor is configured as follows: Determine a set of bases corresponding to the obtained spectral response signal; Based on the determined set of bases, perform a basis transformation on at least one basis included in the determined set of bases; as well as Reconstructed spectral data is generated from the spectral response signal that has undergone the basis transformation by using a pseudo-inverse. The processor is further configured to: determine at least one basis with singular values ​​less than a threshold among the basis corresponding to the obtained spectral response signal, and replace the singular values ​​of the determined at least one basis with specific singular values. Wherein, the specific singular value is the threshold, or the minimum of the singular values ​​of other bases in the basis corresponding to the obtained spectral response signal, wherein each of the other bases has a singular value greater than or equal to the threshold, and The threshold is determined based on the nth singular value among the singular values ​​of the basis corresponding to the obtained spectral response signal, arranged in ascending order, where n is a natural number greater than 0.

8. The device according to claim 7, wherein, The processor is further configured to determine the set of bases by using a principal component analysis (PCA) algorithm to obtain the principal components of the spectral response signal for each channel.

9. The device according to claim 7, wherein, The number of bases included in the set of bases is less than or equal to the number of channels.

10. The device according to claim 7, wherein, The processor is further configured to generate hyperspectral images based on the generated reconstructed spectral data.

11. The device according to claim 10, wherein, The processor is further configured to generate the hyperspectral image by generating an RGB image, which is obtained by color conversion from the reconstructed spectral data using a color matching function.

12. The device according to claim 7, wherein, The image sensor includes a multispectral image sensor configured to acquire spectral data from three or more channels.

13. A non-transitory computer-readable recording medium having a program recorded thereon for performing a method of processing spectral data, the method comprising: A spectral response signal corresponding to a channel of spectral data of light is obtained, the spectral data being acquired from an object by an image sensor; Determine a set of bases corresponding to the obtained spectral response signal; Based on the determined set of bases, perform a basis transformation on at least one basis included in the determined set of bases; as well as Reconstructed spectral data is generated from the spectral response signal that has undergone the basis transformation by using a pseudo-inverse. The basis transformation includes: Determine at least one basis with singular values ​​less than a threshold among the basis corresponding to the obtained spectral response signal; and Replace the singular values ​​of at least one of the identified bases with specific singular values. Wherein, the specific singular value is the threshold, or the minimum of the singular values ​​of other bases in the basis corresponding to the obtained spectral response signal, wherein each of the other bases has a singular value greater than or equal to the threshold, and The threshold is determined based on the nth singular value among the singular values ​​of the basis corresponding to the obtained spectral response signal, arranged in ascending order, where n is a natural number greater than 0.

Images