Multi-spectral reconstruction method and system based on low-dimensional color space camera data stream
By learning the dictionary transformation matrix library for sparse spectral representation and performing mixed optimization during the imaging process, the problem that the existing technology is difficult to accurately reconstruct multispectral performance under changing lighting conditions is solved, and a low-cost and high-efficiency multispectral reconstruction effect is achieved.
Patent Information
- Application Number
- CN202411875003.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-18
- Publication Date
- 2025-04-11
AI Technical Summary
Existing RGB-based multispectral reconstruction methods are difficult to accurately reconstruct the multispectral performance of objects under changing lighting conditions, and require high-cost equipment and complex calibration processes.
By collecting multiple sets of spectral sparse representation data under different illumination conditions, a preset sparse representation dictionary conversion matrix library is learned using a preset sparse representation optimization algorithm. During the imaging process, a hybrid conversion matrix is obtained to reconstruct the multispectral camera data stream.
The multispectral response can be accurately reconstructed under changing lighting conditions without repeated calibration, improving the efficiency and flexibility of reconstruction and reducing equipment costs and operational complexity.
Smart Images
Figure CN119814939A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of spectral reconstruction, and in particular relates to a multi-spectral reconstruction method and system based on low-dimensional color space camera data stream. Background Art
[0002] Multispectral imaging technology has shown great application value in many fields because of its higher spectral resolution than RGB color imaging. This technology can obtain image data of the target at different wavelengths, provide rich spectral information, and become a powerful tool for in-depth understanding of the surface characteristics and composition of objects. It is widely used in agriculture, geological exploration, biomedicine, materials science, remote sensing monitoring and other fields. However, multispectral imaging requires precise optical components, complex sensor arrays, high-performance data processing systems and professional operating software, resulting in high purchase and maintenance costs, which has become a major obstacle to its popularization and limited its application scope and market penetration.
[0003] In order to overcome these limitations, researchers have been actively exploring new ways to use popular low-dimensional color space cameras and simulate multispectral imaging effects through algorithms in recent years. Low-dimensional color space cameras are mainly RGB cameras, which are low-cost, have high public penetration, and have rich data accumulation. Researchers have tried to extract hidden multispectral information from RGB images through advanced image processing algorithms to achieve low-cost and high-efficiency multispectral imaging. However, these methods still face challenges in practical applications. Since RGB cameras only capture information about objects in the three visible spectrum segments of red, green, and blue, compared with multispectral cameras, RGB image data has a lower dimension and serious aliasing of signals from different channels. Therefore, there are pathological problems in directly reconstructing multispectral reflectance signals from RGB signals, which is prone to large reconstruction errors.
[0004] The multispectral reconstruction method based on sparse representation considers that the multispectral response and RGB response are respectively the mapping of spectral reflectance on different bases, which can be regarded as different sparse representations of the same signal and have an inherent fixed relationship, thereby avoiding the risk of pathological reconstruction. The conventional method is to train the multispectral response and RGB response of the sample separately, obtain the multispectral and RGB sparse dictionaries and the mapping matrix between the two dictionaries, and then reconstruct the multispectral image from RGB based on the two dictionaries and the mapping relationship. Although this method alleviates the morbidity problem of reconstruction, the reconstruction effect depends on the similarity of the reflection signal of the sample set during the test and training process, that is, the product of reflectivity and illumination.
[0005] At present, most RGB-based multispectral reconstruction methods rely on data learning methods in a single environment, attempting to reconstruct multispectral images by learning the mapping relationship between RGB response and multispectral response under fixed lighting conditions. However, the lighting conditions in actual imaging scenes are complex and changeable. For example, the lighting in RGB camera imaging often changes, requiring repeated calibration under changing lighting conditions to obtain a conversion matrix that can accurately reconstruct multispectral information. This requires a high-dimensional color space camera or a standard color card with high cost, and the calibration process is complex and difficult to operate. Therefore, most existing RGB-based multispectral reconstruction methods are difficult to accurately reconstruct the multispectral performance of objects under different lighting conditions, and are not suitable for use in general scenes. Summary of the invention
[0006] In order to solve the above-mentioned deficiencies of the prior art, the main purpose of the present invention is to provide a multispectral reconstruction method and system based on a low-dimensional color space camera data stream, which can accurately reconstruct the multispectral response under changing lighting conditions without repeated calibration.
[0007] To achieve the above object, the present invention adopts the following technical solutions:
[0008] <Option 1>
[0009] The present invention provides a multispectral reconstruction method based on a low-dimensional color space camera data stream, which has the following characteristics and comprises the following steps:
[0010] Step S1, using imaging sensors with different sensor response characteristics to collect multiple sets of spectral sparse representation data of the training target under different lighting conditions, to obtain a spectral sparse representation dictionary of the training target; the spectral sparse representation data includes a low-dimensional color space response and a corresponding multi-spectral response;
[0011] Step S2, using a preset sparse representation optimization algorithm to perform deep learning on the spectral sparse representation dictionary to calculate a conversion matrix for converting each low-dimensional color space response to a corresponding multi-spectral response, and obtain a spectral sparse representation dictionary conversion matrix library;
[0012] Step S3, extracting the lighting condition and sensor response characteristics in the current imaging scene based on the prior knowledge of the imaging condition as the current lighting condition and the current sensor response characteristics, respectively, where the current lighting condition includes the number of the current light sources;
[0013] Step S4, judging whether the number of current light sources is greater than 1, if the judgment result is yes, proceeding to step S5; otherwise, proceeding to step S8;
[0014] Step S5, searching the spectral sparse representation dictionary conversion matrix library according to the current lighting condition and the current sensor response characteristics, and obtaining a plurality of conversion matrices that are closest to the current lighting condition and the current sensor response characteristics;
[0015] Step S6, performing hybrid optimization on the obtained multiple conversion matrices to obtain a hybrid conversion matrix;
[0016] Step S7, using the hybrid transformation matrix to reconstruct the input low-dimensional color space camera data stream to obtain a multi-spectral camera data stream,
[0017] Step S8, searching the spectral sparse representation dictionary conversion matrix library according to the current lighting condition and the current sensor response characteristics, and obtaining a conversion matrix that is closest to the current lighting condition and the current sensor response characteristics;
[0018] Step S9: Reconstruct the input low-dimensional color space camera data stream using the obtained conversion matrix to obtain a multi-spectral camera data stream.
[0019] The multispectral reconstruction method based on low-dimensional color space camera data stream provided by the present invention may also have the following features: wherein, in step S1, the imaging sensor includes a smart phone, a personal computer and a digital camera.
[0020] In the multispectral reconstruction method based on low-dimensional color space camera data stream provided by the present invention, it can also have the following characteristics: wherein, in step S1, the lighting conditions include a light source and a corresponding color temperature, and the light source includes daylight, incandescent lamp, fluorescent lamp, halogen lamp and LED lamp.
[0021] In the multispectral reconstruction method based on low-dimensional color space camera data stream provided by the present invention, it can also have the following characteristics: wherein, in step S2, the preset sparse representation optimization algorithm is a convolutional neural network, a least squares method, a ridge regression, a Lasso regression, a support vector machine regression or an iterative closest point algorithm.
[0022] In the multispectral reconstruction method based on low-dimensional color space camera data stream provided by the present invention, it can also have the following characteristics: wherein, in step S6, the mixed transformation matrix is obtained by averaging all the obtained transformation matrices based on the similarity between the lighting conditions and sensor response characteristics of each obtained transformation matrix and the current lighting conditions and the current sensor response characteristics.
[0023] <Option 2>
[0024] The present invention also provides a multispectral reconstruction system based on a low-dimensional color space camera data stream, which has the following characteristics, including: a training data acquisition module, which is used to use imaging sensors with different sensor response characteristics to acquire multiple groups of spectral sparse representation data of training targets under different lighting conditions, and obtain a spectral sparse representation dictionary of the training targets; the spectral sparse representation data includes a low-dimensional color space response and a corresponding multi-spectral response; a conversion matrix learning module, which is used to use a preset sparse representation optimization algorithm to perform deep learning on the spectral sparse representation dictionary to calculate the conversion matrix of each low-dimensional color space response to the corresponding multi-spectral response, and obtain a spectral sparse representation dictionary conversion matrix library; an imaging condition extraction module, which is used to extract the lighting conditions and sensor response characteristics under the current imaging scene based on the prior knowledge of the imaging conditions as the current lighting conditions and the current sensor response characteristics, respectively, and the current lighting conditions include the number of current light sources; a judgment module, which is used to judge when Whether the number of front light sources is greater than 1; a retrieval acquisition module, which is used to retrieve the spectral sparse representation dictionary conversion matrix library according to the current lighting conditions and the current sensor response characteristics when the current number of light sources is greater than 1, and obtain multiple conversion matrices that are closest to the current lighting conditions and the current sensor response characteristics; it is also used to retrieve the spectral sparse representation dictionary conversion matrix library according to the current lighting conditions and the current sensor response characteristics when the current number of light sources is equal to 1, and obtain a conversion matrix that is closest to the current lighting conditions and the current sensor response characteristics; a hybrid optimization module, which is used to perform hybrid optimization on the obtained multiple conversion matrices to obtain a hybrid conversion matrix; and a reconstruction module, which is used to reconstruct the input low-dimensional color space camera data stream using the hybrid conversion matrix to obtain a multi-spectral camera data stream; it is also used to reconstruct the input low-dimensional color space camera data stream using the obtained conversion matrix to obtain a multi-spectral camera data stream.
[0025] <Option 3>
[0026] The present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program and has the following characteristics: when the processor executes the computer program, the steps of the multispectral reconstruction method based on low-dimensional color space camera data stream of <Solution 1> are implemented.
[0027] <Option 4>
[0028] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which has the following characteristics: Specifically, when the computer program is executed by a processor, the steps of the multispectral reconstruction method based on low-dimensional color space camera data stream of <Solution 1> are implemented.
[0029] Compared with the prior art, the present invention obtains a conversion matrix library between spectral sparse dictionaries based on common light source types and light source states in general scenes; in the imaging process, through a priori judgment of the light source characteristics, several matrices in the conversion matrix library are called to form a matrix basis to perform hybrid optimization to obtain a hybrid conversion matrix, so that a multispectral camera data stream can be accurately reconstructed from an ordinary low-dimensional color space camera data stream under changing lighting conditions without repeated calibration. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] Figure 1 is a flow chart of a multispectral reconstruction method based on a low-dimensional color space camera data stream in an embodiment of the present invention; and
[0031] Figure 2 4 is a structural block diagram of a multi-spectral reconstruction system based on a low-dimensional color space camera data stream in an embodiment of the present invention. DETAILED DESCRIPTION
[0032] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention.
[0033] <Example>
[0034] refer to Figure 1 The multispectral reconstruction method based on low-dimensional color space camera data stream provided by an embodiment of the present invention specifically includes the following steps S1-S9.
[0035] Step S1, using imaging sensors with different sensor response characteristics to collect multiple sets of spectral sparse representation data of training targets under different lighting conditions, to obtain a spectral sparse representation dictionary of the training targets, wherein the spectral sparse representation data includes low-dimensional color space responses, such as RGB values, and corresponding multi-spectral responses.
[0036] Specifically, the lighting conditions include multiple light sources and the type and color temperature corresponding to each light source. The light sources include daylight, incandescent lamps, fluorescent lamps, halogen lamps, and LED lamps. During the training process, the color temperature settings of various light sources are slightly different: the color temperature classification of daylight comes from different time periods of different weather conditions (including cloudy, cloudy, sunny, and rainy days), from sunrise to sunset at one-hour intervals; the color temperature of incandescent lamps ranges from 2400K to 3400K, with intervals of 500K; the color temperature of fluorescent lamps ranges from 2700K to 6500K, with intervals of 1000K; the color temperature of LED lamps ranges from 3000K to 7000K, with intervals of 1000K.
[0037] Imaging sensors include smartphones, personal computers, and digital cameras with different spectral response characteristics, and the top five brands and models in market sales rate are selected for each.
[0038] In this embodiment, the Xrite ColorChecker Digital SG is used as a training target, and the responses of 96 standard color blocks in the low-dimensional color space camera are collected.
[0039] Step S2, using a preset sparse representation optimization algorithm to perform deep learning on the spectral sparse representation dictionary to calculate the conversion matrix of each low-dimensional color space response to the corresponding multi-spectral response, and obtain a spectral sparse representation dictionary conversion matrix library.
[0040] Specifically, the preset sparse representation optimization algorithm adopts a convolutional neural network. In the sparse representation framework, the spectral signal can be approximately represented by a few non-zero coefficients in color spaces of different dimensions. Low-dimensional color space responses such as RGB can be regarded as sparse sampling of the original spectral signal in low dimensions, and multi-spectral responses can be regarded as sparse sampling of the original spectral signal in higher dimensions. The convolutional neural network uses the gradient descent method to minimize the mean square error between the reconstructed multi-spectral response and the original multi-spectral response to obtain the response conversion matrix. Since the standard color card will produce different low-dimensional color space responses under different lighting conditions and sensors during the training process, each conversion matrix corresponds to a specific lighting condition and sensor spectral response characteristic.
[0041] In this embodiment, when learning the spectral sparse dictionary conversion matrix, large labels such as the light source emission spectrum curve and the spectral response curve of the imaging sensor are taken into consideration, so that the training results do not depend on small labels such as the light source and the model of the low-dimensional color space imaging sensor, thereby making full use of the homogeneity of the light source and camera spectral information in common scenes, and improving the versatility and flexibility of the reconstruction method.
[0042] Deep learning technology is used to replace the traditional Wiener estimation method to calculate the conversion relationship from low-dimensional color response to multi-spectral response, thereby improving the accuracy and robustness of multi-spectral reconstruction.
[0043] By collecting training data under a variety of light source types, color temperatures, and sensor conditions, a comprehensive dictionary library and mapping matrix library are constructed, covering common lighting conditions and sensor response characteristics, providing strong data support for practical applications.
[0044] Step S3, extracting the lighting condition and sensor response characteristics in the current imaging scene based on the prior knowledge of the imaging condition as the current lighting condition and the current sensor response characteristics respectively.
[0045] Specifically, during the imaging process, based on the prior knowledge of the imaging conditions, the imaging features of the current scene are extracted, and the imaging features include the current lighting conditions and the current sensor response features, where the current lighting conditions include the number of current light sources and the type and color temperature corresponding to each light source. The brand and model of the sensor, as well as the type of light source of the lighting conditions can be input by the user. The color temperature of daylight is determined according to the weather and time period, and the color temperature of other light sources can be checked in the color temperature settings. If it is not known, it can be measured using smartphone applications such as LightSpectrum Pro.
[0046] Step S4, judging whether the number of current light sources is greater than 1, if the judgment result is yes, proceeding to step S5; otherwise, proceeding to step S8.
[0047] Step S5, searching the spectral sparse representation dictionary conversion matrix library according to the current lighting condition and the current sensor response characteristic, and obtaining a plurality of conversion matrices that are closest to the current lighting condition and the current sensor response characteristic.
[0048] Step S6, performing hybrid optimization on the obtained multiple conversion matrices to obtain a hybrid conversion matrix.
[0049] In this embodiment, the hybrid conversion matrix is obtained by averaging all the obtained conversion matrices according to the similarity between the illumination conditions and sensor response characteristics of each obtained conversion matrix and the current illumination conditions and the current sensor response characteristics. The hybrid conversion matrix comprehensively considers the influence of various illumination conditions and can better adapt to complex and changeable real-world general scenarios.
[0050] Step S7, using the hybrid transformation matrix to reconstruct the input low-dimensional color space camera data stream to obtain a multi-spectral camera data stream.
[0051] Specifically, the input low-dimensional color space image is reconstructed into a multispectral domain using a hybrid optimized hybrid transformation matrix. That is, the spectral information in the low-dimensional color space image is mapped to the multispectral domain through matrix transformation to obtain a high spectral resolution image containing rich spectral information.
[0052] In this embodiment, by collecting training data in advance under different lighting conditions and sensor response characteristics, multiple spectral sparse representation dictionary transformation matrices are learned, each dictionary matrix corresponding to a lighting condition. Under multi-light source lighting conditions, a hybrid optimization method is used to calculate a new transformation matrix based on multiple transformation matrices, which solves the limitation of relying on a single environmental data in traditional methods and improves the robustness of the reconstruction results.
[0053] Step S8, searching the spectral sparse representation dictionary conversion matrix library according to the current lighting condition and the current sensor response characteristics, and obtaining a conversion matrix that is closest to the current lighting condition and the current sensor response characteristics.
[0054] Step S9: Reconstruct the input low-dimensional color space camera data stream using the obtained conversion matrix to obtain a multi-spectral camera data stream.
[0055] refer to Figure 2 The embodiment of the present invention also provides a multispectral reconstruction system 10 based on low-dimensional color space camera data stream. The multispectral reconstruction system 10 based on low-dimensional color space camera data stream includes a training data acquisition module 11, a conversion matrix learning module 12, an imaging condition extraction module 13, a judgment module 14, a retrieval module 15, a hybrid optimization module 16 and a reconstruction module 17.
[0056] The training data acquisition module 11 is used to use imaging sensors with different sensor response characteristics to collect multiple sets of spectral sparse representation data of the training target under different lighting conditions to obtain a spectral sparse representation dictionary of the training target; wherein the spectral sparse representation data includes a low-dimensional color space response and a corresponding multi-spectral response.
[0057] The conversion matrix learning module 12 is used to perform deep learning on the spectral sparse representation dictionary using a preset sparse representation optimization algorithm to calculate the conversion matrix of each low-dimensional color space response converted to the corresponding multi-spectral response, and obtain a spectral sparse representation dictionary conversion matrix library.
[0058] The imaging condition extraction module 13 is used to extract the lighting condition and the sensor response feature in the current imaging scene based on the prior knowledge of the imaging condition as the current lighting condition and the current sensor response feature respectively.
[0059] The judging module 14 is used to judge whether the number of current light sources is greater than one.
[0060] The retrieval acquisition module 15 is used to search the spectral sparse representation dictionary conversion matrix library according to the current lighting conditions and the current sensor response characteristics when the current number of light sources is greater than 1, and obtain multiple conversion matrices that are closest to the current lighting conditions and the current sensor response characteristics; it is also used to search the spectral sparse representation dictionary conversion matrix library according to the current lighting conditions and the current sensor response characteristics when the current number of light sources is equal to 1, and obtain a conversion matrix that is closest to the current lighting conditions and the current sensor response characteristics.
[0061] The hybrid optimization module 16 is used to perform hybrid optimization on the obtained multiple conversion matrices to obtain a hybrid conversion matrix.
[0062] The reconstruction module 17 is used to reconstruct the input low-dimensional color space camera data stream using a mixed conversion matrix to obtain a multi-spectral camera data stream; it is also used to reconstruct the input low-dimensional color space camera data stream using an obtained conversion matrix to obtain a multi-spectral camera data stream.
[0063] An embodiment of the present invention also provides a computer device, including a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the steps of the multispectral reconstruction method based on the low-dimensional color space camera data stream as described above.
[0064] The computer device may be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected through a system bus. The processor is used to provide computing and control capabilities; the memory includes a non-volatile storage medium and an internal memory, the non-volatile storage medium stores an operating system and a computer program, and the internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium; the communication interface is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies; the display screen may be a liquid crystal display or an electronic ink display screen; the input device may be a touch layer covering the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.
[0065] An embodiment of the present invention further provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the multispectral reconstruction method based on a low-dimensional color space camera data stream as described above.
[0066] The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.
[0067] In summary, first, the embodiment of the present invention collects low-dimensional color space responses and corresponding multi-spectral response data under various lighting conditions, uses a sparse representation optimization algorithm to learn multiple sets of sparse representation dictionary mapping matrices, and stores these mapping matrices in a database. In the actual imaging process, it is only necessary to call the corresponding mapping matrix from the database for multi-spectral reconstruction based on the prior estimation of the lighting conditions, without the need for repeated on-site calibration. It has the following beneficial effects: (1) Improved efficiency, pre-calibration and establishment of a database, so that the mapping matrix can be quickly called in actual applications, greatly saving the time of on-site calibration; (2) Enhanced flexibility, no longer relying on on-site calibration, so that the present invention can be more flexibly adapted to various practical application scenarios; (3) Reduced costs, reduced dependence on high-dimensional color space cameras or standard color cards, and reduced equipment costs and complexity of use.
[0068] Secondly, the embodiment of the present invention collects training data under different lighting conditions, learns multiple groups of sparse representation dictionary mapping matrices, and constructs a comprehensive mapping matrix library; during the imaging process, according to the prior estimation of the lighting conditions, the mapping matrix closest to the current lighting conditions is selected for multi-spectral reconstruction. In addition, a hybrid optimization strategy is adopted to adapt to complex and changeable lighting environments by weighted averaging multiple dictionary mapping matrices. It has the following beneficial effects: (1) Enhanced robustness, able to adapt to a variety of lighting conditions, and maintaining a high reconstruction quality even in complex and changeable environments; (2) Expanded scope of application, no longer limited to stable lighting conditions, so that multi-spectral reconstruction technology can be applied to more practical scenarios; (3) Improved reconstruction accuracy, through the hybrid optimization strategy, further improve the accuracy and reliability of the reconstruction results.
[0069] Thirdly, the embodiment of the present invention abandons the dependence on standard color cards or multispectral cameras, and performs multispectral reconstruction through the sparse representation dictionary mapping matrix library obtained by pre-learning. In practical applications, it is only necessary to select a suitable mapping matrix according to the lighting conditions of the imaging environment, without any additional hardware equipment. It has the following beneficial effects: (1) Reduce costs, without the need to purchase and maintain expensive equipment such as standard color cards or multispectral cameras; (2) Simplify operation, users do not need to have complex calibration skills or equipment operation experience to perform multispectral reconstruction; (3) Improve popularity, because no additional equipment is required and the operation is simple and convenient, making the present invention easier to popularize and promote to a wider range of application fields.
[0070] Finally, the embodiment of the present invention uses a convolutional neural network to directly learn the conversion matrix from the low-dimensional color space response to the multi-spectral response. Through a large amount of training data, the deep learning model can learn a more accurate and robust mapping matrix. It has the following beneficial effects: (1) Improve the reconstruction accuracy. The deep learning model has a strong learning ability and generalization ability, and can achieve more accurate multi-spectral reconstruction under complex and changeable lighting conditions. (2) Enhanced robustness. Compared with the Wiener estimate, the deep learning model has better robustness to the noise and outliers of the training data and can reconstruct more stably.
[0071] The above-mentioned embodiments are preferred examples of the present invention and are not intended to limit the protection scope of the present invention.
[0072] For example, in the above-mentioned embodiment, the preset sparse representation optimization algorithm adopts a convolutional neural network to minimize the minimum mean square error between the reconstructed multispectral signal and the original multispectral signal to obtain a conversion matrix. Of course, in the present invention, the preset sparse representation optimization algorithm can also adopt the least square method, ridge regression, Lasso regression, support vector machine regression or iterative nearest point algorithm to perform deep learning on the spectral sparse representation dictionary to obtain the conversion matrix.
[0073] The least squares method is a mathematical optimization technique that finds the best function match for a set of data by minimizing the sum of squared errors. In the multispectral reconstruction problem, the best transformation matrix can be found by solving a system of linear equations so that the sum of squared errors between the reconstructed multispectral signal and the original signal is minimized.
[0074] Ridge regression is a biased estimation regression method specifically used for collinear data analysis. It is essentially a modified least squares estimation method. It abandons the unbiasedness of the least squares method and obtains a more practical and reliable regression coefficient at the cost of losing some information and reducing precision. It fits pathological data better than the least squares method. In multispectral reconstruction, the transformation matrix can be solved by ridge regression to avoid the estimation distortion problem that occurs when the feature correlation is high.
[0075] Lasso regression is an extension of linear regression. It constrains the complexity of the model by adding a penalty term to the absolute value of the regression coefficient, thereby avoiding overfitting and improving the generalization ability of the model. In multispectral reconstruction, Lasso regression can be used to select important variables and obtain the transformation matrix, which is particularly effective for processing high-dimensional data.
[0076] Although SVM is mainly used for classification problems, its regression version (SVR) can also be used for regression problems. SVR handles regression problems by introducing an insensitive loss function and seeks to minimize structural risk. In multispectral reconstruction, you can try to use SVR to fit the relationship between RGB response and multispectral response to obtain the transformation matrix.
[0077] Although the iterative closest point algorithm (ICP) is commonly used for point cloud registration problems, the optimization idea behind it (minimizing the distance between corresponding points) can also inspire us to design other optimization algorithms to minimize reconstruction errors. Although the ICP algorithm itself may not be directly used to solve the transformation matrix in multispectral reconstruction, its optimization strategy can be used as a reference when designing new algorithms.
Claims
1. A multispectral reconstruction method based on low-dimensional color space camera data stream, characterized in that: The following steps are involved: Step S1, using imaging sensors with different sensor response characteristics to collect multiple sets of spectral sparse representation data of the training target under different lighting conditions, to obtain a spectral sparse representation dictionary of the training target; The spectral sparse representation data includes a low-dimensional color space response and a corresponding multi-spectral response; Step S2, using a preset sparse representation optimization algorithm to perform deep learning on the spectral sparse representation dictionary to calculate a conversion matrix for converting each of the low-dimensional color space responses to the corresponding multi-spectral responses, and obtain a spectral sparse representation dictionary conversion matrix library; Step S3, extracting the lighting condition and sensor response characteristics in the current imaging scene based on the prior knowledge of the imaging condition as the current lighting condition and the current sensor response characteristics, respectively, wherein the current lighting condition includes the number of the current light sources; Step S4, judging whether the number of the current light sources is greater than 1, and when the judgment result is yes, proceeding to step S5; Otherwise, proceed to step S8; Step S5, searching the spectral sparse representation dictionary conversion matrix library according to the current lighting condition and the current sensor response characteristic to obtain a plurality of conversion matrices that are closest to the current lighting condition and the current sensor response characteristic; Step S6, performing hybrid optimization on the obtained multiple conversion matrices to obtain a hybrid conversion matrix; Step S7, using the hybrid transformation matrix to reconstruct the input low-dimensional color space camera data stream to obtain a multi-spectral camera data stream, Step S8, searching the spectral sparse representation dictionary conversion matrix library according to the current lighting condition and the current sensor response characteristic to obtain a conversion matrix that is closest to the current lighting condition and the current sensor response characteristic; Step S9: Reconstruct the input low-dimensional color space camera data stream using the obtained conversion matrix to obtain a multi-spectral camera data stream.
2. The multispectral reconstruction method based on low-dimensional color space camera data stream according to claim 1, characterized in that: in, In the step S1, the imaging sensor includes a smart phone, a personal computer and a digital camera.
3. The multispectral reconstruction method based on low-dimensional color space camera data stream according to claim 1, characterized in that: in, In the step S1, the lighting conditions include light sources and corresponding color temperatures, and the light sources include daylight, incandescent lamps, fluorescent lamps, halogen lamps, and LED lamps.
4. The multispectral reconstruction method based on low-dimensional color space camera data stream according to claim 1, characterized in that: in, In step S2, the preset sparse representation optimization algorithm is a convolutional neural network, a least squares method, a ridge regression, a Lasso regression, a support vector machine regression or an iterative closest point algorithm.
5. The multispectral reconstruction method based on low-dimensional color space camera data stream according to claim 1, characterized in that: in, In step S6, the hybrid conversion matrix is obtained by averaging all the conversion matrices obtained according to the similarity between the lighting condition and the sensor response characteristic of each conversion matrix obtained and the current lighting condition and the current sensor response characteristic.
6. A multispectral reconstruction system based on low-dimensional color space camera data stream, characterized in that: include: A training data acquisition module, used to acquire multiple sets of spectral sparse representation data of training targets under different lighting conditions using imaging sensors with different sensor response characteristics, and obtain a spectral sparse representation dictionary of the training targets; The spectral sparse representation data includes a low-dimensional color space response and a corresponding multi-spectral response; A conversion matrix learning module is used to perform deep learning on the spectral sparse representation dictionary using a preset sparse representation optimization algorithm to calculate a conversion matrix for converting each low-dimensional color space response to the corresponding multi-spectral response, and obtain a spectral sparse representation dictionary conversion matrix library; An imaging condition extraction module, used to extract the lighting condition and sensor response characteristics in the current imaging scene based on prior knowledge of the imaging condition as the current lighting condition and the current sensor response characteristics, respectively, wherein the current lighting condition includes the number of current light sources; A judging module, used to judge whether the number of the current light sources is greater than 1; A retrieval acquisition module, used for searching the spectral sparse representation dictionary conversion matrix library according to the current lighting condition and the current sensor response characteristic when the number of the current light sources is greater than 1, and obtaining a plurality of conversion matrices that are closest to the current lighting condition and the current sensor response characteristic; and also used for searching the spectral sparse representation dictionary conversion matrix library according to the current lighting condition and the current sensor response characteristic when the number of the current light sources is equal to 1, and obtaining a conversion matrix that is closest to the current lighting condition and the current sensor response characteristic; A hybrid optimization module, used for performing hybrid optimization on the obtained multiple transformation matrices to obtain a hybrid transformation matrix; as well as A reconstruction module is used to reconstruct the input low-dimensional color space camera data stream using the mixed conversion matrix to obtain a multi-spectral camera data stream; and is also used to reconstruct the input low-dimensional color space camera data stream using an obtained conversion matrix to obtain a multi-spectral camera data stream.
7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the multispectral reconstruction method based on low-dimensional color space camera data stream according to any one of claims 1 to 5 are implemented.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the multispectral reconstruction method based on low-dimensional color space camera data stream according to any one of claims 1 to 5 are implemented.