A portable spectral imaging material identification method and system
By deploying spectral reconstruction network models and light field cameras on mobile devices, combined with deep learning technology, portable, lossless and fast material identification is achieved, solving the time and operational complexity of traditional methods, and improving the ability to obtain material information.
Patent Information
- Application Number
- CN202410475301.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-19
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-04-19
AI Technical Summary
Traditional material identification methods require the destruction of samples, which takes a long time and are difficult to meet the needs of fast and lossless identification. In addition, light field imaging and spectral imaging technologies require large equipment and professional operators, making it difficult to apply to real-time detection.
By downloading the spectral dataset, constructing the initial spectral reconstruction network model, training and compressing the model, deploying it to a mobile device, collecting the light field image and reconstructing the spectral image, using the MobileNetV3 network model for material type identification, realizing portable and easy-to-use Material identification.
It realizes lossless, fast and accurate material identification, reduces equipment costs, improves material information acquisition capabilities, and solves the time and operational complexity problems of traditional methods.
Smart Images

Figure CN118443593B_ABST
Abstract
Description
Technical Field
[0001] The present invention mainly relates to the technical field of material hyperspectral detection, and in particular to a material identification method and system using portable spectral imaging. Background Art
[0002] With the continuous development of science and technology, the requirements for the performance and composition of materials in many industries are becoming higher and higher, and the accurate identification of materials is becoming more and more important. Therefore, material identification has become a vital technology in many fields.
[0003] Traditional material identification methods mainly include chemical analysis and physical property testing, etc. However, these methods usually require the destruction of samples and are time-consuming, making it difficult to meet the needs of rapid and non-destructive identification.
[0004] In recent years, with the continuous development of light field imaging and spectral imaging technology, all-optical functions are used to calculate light waves in nature and represent the information carried by light. However, light field imaging and spectral imaging technologies require large equipment and professional operators, and are difficult to apply in real-time detection of material identification. Therefore, a portable and easy-to-operate material identification method is urgently needed. Summary of the invention
[0005] The technical problem to be solved by the present invention is to provide a portable spectral imaging material identification method and system in view of the deficiencies in the prior art.
[0006] The technical solution of the present invention to solve the above technical problems is as follows:
[0007] A portable spectral imaging material identification method comprises the following steps:
[0008] Downloading a spectral data set, constructing an initial spectral reconstruction network model, training the initial spectral reconstruction network model through the spectral data set to obtain a trained spectral reconstruction network model, compressing the trained spectral reconstruction network model, and converting the format of the compressed spectral reconstruction network model to obtain a spectral reconstruction network model;
[0009] Replacing the camera of the mobile device with a pre-assembled light field camera, deploying the spectral reconstruction network model in the mobile device, and obtaining a light field spectral imaging device;
[0010] The light field spectral imaging device is used to collect a plurality of light field images corresponding to a plurality of materials at different angles and in different environments, and a plurality of corresponding spectral images are reconstructed, a plurality of corresponding spectral curve graphs are generated according to each of the spectral images, and each material category and the corresponding plurality of spectral curve graphs are constructed into a material data set;
[0011] The pre-constructed initial MobileNetV3 network model is trained by using the material data set, and the format of the trained MobileNetV3 network model is converted to obtain a MobileNetV3 network model;
[0012] The MobileNetV3 network model is deployed in the light field spectral imaging device, and the type of the target material is identified by the deployed light field spectral imaging device to obtain a material type result, and the material type result is displayed in a visualization interface of the deployed light field spectral imaging device.
[0013] Another technical solution of the present invention to solve the above technical problems is as follows:
[0014] A portable spectral imaging material identification system, comprising:
[0015] A first network generation module is used to download a spectral data set, construct an initial spectral reconstruction network model, train the initial spectral reconstruction network model through the spectral data set to obtain a trained spectral reconstruction network model, compress the trained spectral reconstruction network model, and convert the format of the compressed spectral reconstruction network model to obtain a spectral reconstruction network model;
[0016] A device generation module, used to replace the camera of the mobile device with a pre-assembled light field camera, deploy the spectral reconstruction network model into the mobile device, and obtain a light field spectral imaging device;
[0017] A data construction module, for collecting a plurality of light field images corresponding to a plurality of materials at different angles and in different environments through the light field spectrum imaging device, and reconstructing a plurality of corresponding spectrum images, generating a plurality of corresponding spectrum curve graphs according to each of the spectrum images, and constructing each material category and the corresponding plurality of spectrum curve graphs into a material data set;
[0018] A second network generation module is used to train the pre-constructed initial MobileNetV3 network model through the material data set, and convert the format of the trained MobileNetV3 network model to obtain a MobileNetV3 network model;
[0019] The application module is used to deploy the MobileNetV3 network model to the light field spectral imaging device, identify the type of the target material through the deployed light field spectral imaging device, obtain the material type result, and display the material type result in the visualization interface of the deployed light field spectral imaging device.
[0020] The beneficial effects of the present invention are as follows: the initial spectral reconstruction network model constructed by training a pair of real light field images with light field information and spectral information and a real spectral image with spectral information enables the initial spectral reconstruction network model to decouple the spectral information in the light field image. The spectral reconstruction network model is compressed and trained to reduce storage space and computing resources, and the compressed spectral reconstruction network model is format converted to improve the compatibility of the model deployed in mobile devices and improve the performance of the model. The spectral reconstruction network model is deployed in a mobile device whose camera has been replaced, so that the mobile device becomes a portable light field spectral imaging device. The light field images of various materials are collected by the light field spectral imaging device, and the corresponding spectral images are generated, and a material type data set is constructed to train the MobileNetV3 network model for material category classification, and finally the trained MobileNetV3 network model is format converted and deployed to the light field spectral imaging device to realize material identification anytime and anywhere. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] Figure 1 A flow chart of a portable spectral imaging material identification method provided by an embodiment of the present invention;
[0022] Figure 2 A structural diagram of an initial spectrum reconstruction network model provided by an embodiment of the present invention;
[0023] Figure 3 A schematic diagram of a channel-by-channel convolution operation designed for model update provided by an embodiment of the present invention;
[0024] Figure 4 A schematic diagram of a point-by-point convolution operation designed for model update provided by an embodiment of the present invention;
[0025] Figure 5 A schematic diagram of model sparsification provided by an embodiment of the present invention;
[0026] Figure 6 A schematic diagram of the propagation of imaging light for a light field camera provided in an embodiment of the present invention;
[0027] Figure 7 A module block diagram of a portable spectral imaging material identification system provided in an embodiment of the present invention.
[0028] In the accompanying drawings, the names of the components represented by the various marks are as follows:
[0029] 1. Image sensor; 2. Liquid crystal microlens array; 3. Polarizer; 4. Main lens; 5. Target object. DETAILED DESCRIPTION
[0030] The principles and features of the present invention are described below in conjunction with the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.
[0031] Traditional material identification methods mainly include chemical analysis and physical property testing, but these methods usually require sample destruction and are time-consuming, making it difficult to meet the needs of rapid, non-destructive identification. Therefore, non-destructive material identification technology has become a hot topic and difficulty in research.
[0032] However, with the continuous development of light field imaging and spectral imaging technology, when collecting light field information and spectral information of target objects, the full light function can be used to calculate the light waves in nature and express the information carried by the light, including spatial information (x, y, z), angle information (θ, φ), spectral information λ and time information t. Therefore, the information carried by the light can express the characteristics of the target object and can be applied in the field of material identification technology. In addition, light field imaging and spectral imaging technology have the advantages of non-destructive, fast and accurate, so they have broad application prospects in the field of material identification.
[0033] However, light field imaging and spectral imaging technologies usually require large equipment and professional operators, and are difficult to apply for real-time detection at the identification site. Therefore, developing a portable and easy-to-operate material identification technology is an important research direction at present.
[0034] With the widespread application of deep learning and the maturity of the optimized deployment model framework, light field imagers and spectral imagers are made portable, and light field imaging and spectral imaging technologies are combined with material identification technology to achieve portable, efficient and accurate identification of target materials.
[0035] like Figure 1 As shown, a portable spectral imaging material identification method provided by an embodiment of the present invention includes the following steps:
[0036] Downloading a spectral data set, constructing an initial spectral reconstruction network model, training the initial spectral reconstruction network model through the spectral data set to obtain a trained spectral reconstruction network model, compressing the trained spectral reconstruction network model, and converting the format of the compressed spectral reconstruction network model to obtain a spectral reconstruction network model;
[0037] Replacing the camera of the mobile device with a pre-assembled light field camera, deploying the spectral reconstruction network model in the mobile device, and obtaining a light field spectral imaging device;
[0038] The light field spectral imaging device is used to collect a plurality of light field images corresponding to a plurality of materials at different angles and in different environments, and a plurality of corresponding spectral images are reconstructed, a plurality of corresponding spectral curve graphs are generated according to each of the spectral images, and each material category and the corresponding plurality of spectral curve graphs are constructed into a material data set;
[0039] The pre-constructed initial MobileNetV3 network model is trained by using the material data set, and the format of the trained MobileNetV3 network model is converted to obtain a MobileNetV3 network model;
[0040] The MobileNetV3 network model is deployed in the light field spectral imaging device, and the type of the target material is identified by the deployed light field spectral imaging device to obtain a material type result, and the material type result is displayed in a visualization interface of the deployed light field spectral imaging device.
[0041] It should be understood that the spectral data set includes light field images and spectral images, the light field images include light field information and spectral information, and the spectral images include spectral information. Therefore, the initial spectral reconstruction network model is trained by light field images and spectral images, so that the trained spectral reconstruction network model decouples the spectral information in the light field images.
[0042] The process of training the initial spectral reconstruction network model, model compression and format conversion, and the process of training the initial MobileNetV3 network model are all performed on a computer.
[0043] In an embodiment of the present invention, an initial spectral reconstruction network model constructed by real light field images with light field information and spectral information and real spectral images with spectral information is trained, so that the initial spectral reconstruction network model can decouple the spectral information in the light field image. The initial spectral reconstruction network model after training is compressed to reduce storage space and computing resources, and the compressed spectral reconstruction network model is format-converted to improve the compatibility of the model deployed in mobile devices and improve the performance of the model. The spectral reconstruction network model is deployed to a mobile device whose camera has been replaced, so that the mobile device becomes a portable light field spectral imaging device. The light field images of various materials are collected by the light field spectral imaging device, and the corresponding spectral images are generated. A material type data set is constructed to train the MobileNetV3 network model for material category classification, and finally the trained MobileNetV3 network model is format-converted and deployed to the light field spectral imaging device to realize material identification anytime and anywhere.
[0044] Preferably, downloading the spectral dataset includes: downloading the public CAVE dataset and ICVL dataset from a designated website.
[0045] Specifically, the CAVE dataset and the ICVL dataset include light field images and corresponding spectral images of many landscapes, house images, plant images, real and fake fruits, etc. The CAVE dataset and the ICVL dataset are preprocessed respectively, including: splitting the spectral images and light field images in the CAVE dataset and the ICVL dataset into multiple small blocks of 64 pixels × 64 pixels, and changing the file format of the spectral image to a mat format file.
[0046] In the embodiment of the present invention, preprocessing the data set can increase data diversity, improve model robustness, accelerate the training process, and reduce the risk of overfitting. Training the model with the CAVE data set and the ICVL data set can provide the model with more light field information and spectral information to reconstruct the spectrum of the target object, thereby increasing the generalization ability of the model.
[0047] Preferably, the method further comprises dividing the spectral dataset into a spectral training set and a spectral verification set in a ratio of 8:2.
[0048] It should be understood that the spectral training set is used to train the constructed spectral reconstruction network model, and the spectral verification set is used to verify the trained spectral reconstruction network model.
[0049] Preferably, if Figure 2 As shown, the construction of the initial spectrum reconstruction network model is specifically as follows:
[0050] The initial spectral reconstruction network model is constructed through an activation function layer (i.e., an activation layer), a pooling layer, a residual block, three ordinary convolutional layers (i.e., a 3×3 convolutional layer), and multiple attention blocks (i.e., eight attention blocks), including:
[0051] Connect the input port to the input of the first ordinary convolution layer, connect the output of the first ordinary convolution layer to the input of the first attention block, connect multiple attention blocks one by one, connect the output of the last attention block to the input of the second ordinary convolution layer, respectively connect the output of the first ordinary convolution layer and the output of the second ordinary convolution layer to the input of the residual block, connect the output of the residual block to the input of the activation function layer, connect the output of the activation function layer to the input of the pooling layer, connect the output of the pooling layer to the input of the third ordinary convolution layer, connect the output of the third ordinary convolution layer to the output port, and complete the construction of the initial spectral reconstruction network model.
[0052] Specifically, the three common convolutional layers are composed of 3×3 convolution kernels, and the activation function layer adopts the PReLU activation function;
[0053] Among them, each attention block is composed of an attention convolution layer (i.e., ordinary convolution layer) with a 3×3 convolution kernel, an attention activation function layer (i.e., activation function layer, using PReLU activation function), and an attention residual block (i.e., residual block); the output of the first ordinary convolution layer is connected to the input of the first attention convolution layer of the first attention block and the input of the first attention residual block, the output of the first attention convolution layer is connected to the input of the first attention activation function layer, the output of the first attention activation function layer is connected to the input of the first attention residual block, the output of the first convolution layer is connected to the output of the first attention activation function layer and the input of the first attention residual block (i.e., the output of the first convolution layer is added to the output of the first attention activation function layer through the first attention residual block), and the output of the first attention residual block is connected to the input of the attention convolution layer of the next attention block (i.e., the second attention convolution layer of the second attention block), and so on, and finally the output of the eighth attention residual block of the eighth attention block is connected to the input of the second ordinary convolution layer.
[0054] In the embodiment of the present invention, multiple residual blocks are used to form a global residual connection to eliminate the gradient vanishing and exploding problems in the network model.
[0055] Preferably, the spectral data set includes a plurality of real light field images and a plurality of corresponding real spectral images;
[0056] The initial spectral reconstruction network model is trained by the spectral data set (i.e., spectral training set) to obtain a trained spectral reconstruction network model, specifically:
[0057] Performing shallow feature extraction on a plurality of real light field images and a plurality of corresponding real spectrum images through the first convolution layer to obtain a plurality of first shallow feature maps;
[0058] Performing deep feature extraction on the plurality of the first shallow feature maps through the first attention block, performing deep feature extraction on the plurality of feature maps extracted by the previous attention block through the remaining attention blocks, until the last attention block performs deep feature extraction on the plurality of feature maps extracted by the previous attention block, to obtain a plurality of deep feature maps;
[0059] Performing shallow feature extraction on the plurality of deep feature maps through the second convolutional layer to obtain a plurality of second shallow feature maps;
[0060] Summing the plurality of first shallow feature maps and the plurality of corresponding second shallow feature maps through the residual block;
[0061] Performing nonlinear transformation on the multiple feature maps obtained by summing up through the activation function layer to obtain multiple nonlinear feature maps;
[0062] Performing a dimensionality reduction operation on the plurality of the nonlinear feature maps through the pooling layer to obtain a plurality of reduced-dimensionality feature maps;
[0063] The third convolutional layer is used to perform shallow feature extraction on the plurality of dimension reduction feature maps to obtain a plurality of training reconstructed spectral images, complete the training of the initial spectral reconstruction network model, and obtain the training spectral reconstruction network model.
[0064] It should be understood that a real light field image and a corresponding real spectrum image are a set of spectrum data. Multiple sets of spectrum data constitute a spectrum training set. The model is trained by the spectrum training set. After training, the model outputs a training reconstructed spectrum image corresponding to each set of spectrum data. The training reconstructed spectrum image generated by the model is a hyperspectral image (HSI).
[0065] Specifically, the convolution operation for feature extraction through the convolution layer (i.e., the ordinary convolution layer or the attention convolution layer) can be understood as a sliding window (i.e., the convolution kernel) performing sliding filtering on the feature map output by the previous layer. Specifically, the convolution kernel is multiplied and summed with the local pixels at the corresponding position, and then the bias is superimposed to obtain the feature map output by this layer. Each convolution kernel corresponds to a weight coefficient and a bias parameter, which are fixed during the training process, that is, the weights are shared. Since different convolution kernels can detect different features of the image, the convolution layer of the 3×3 convolution kernel will extract features from the image, which helps the neural network to better understand and recognize the image content. Eight attention blocks are stacked to form a deep network for deep feature extraction, in which a global residual connection is used to eliminate the gradient vanishing and exploding problems in the deep network. Each attention block can make full use of the rich low-frequency information of the original RGB image (i.e., the real light field image in the spectral dataset), thereby enhancing the learning features with great correlation. The parameter rectified linear unit PReLU is selected as the activation function to introduce more nonlinearity and accelerate convergence.
[0066] In the embodiment of the present invention, the model is trained by using a spectral training set to enable the model to learn the rules of the training data, thereby adjusting the parameters of the model to enhance the data processing capability and generalization capability of the model.
[0067] Preferably, the step of training the initial spectral reconstruction network model using the spectral data set (i.e., spectral training set) further includes:
[0068] After the initial spectral reconstruction network model is trained, a plurality of the training reconstructed spectral images and the corresponding real spectral images are respectively calculated by a pre-constructed loss function to obtain a plurality of reconstruction losses. The loss function is:
[0069]
[0070] Where L represents the reconstruction loss, represents the j-th pixel value of the real spectrum image (j=1,...,N), represents the jth pixel value of the training reconstructed spectral image (j=1,...,N), N represents the total number of pixels in the image, ε represents the zero point of the real spectral image, and ε=1×10 -3 ;
[0071] A weighted calculation is performed on the multiple reconstruction losses to obtain a total loss, and the initial spectral reconstruction network model is iteratively optimized using the total loss to obtain a training spectral reconstruction network model.
[0072] It should be understood that the multiple spectral images output by the model are compared with the corresponding real spectral images to obtain multiple reconstruction losses. The loss function value of the model reconstructing the training set is calculated from the multiple reconstruction losses. The loss function value of the model on the training set is used as a comprehensive indicator to measure the reconstruction performance of the model.
[0073] In the embodiment of the present invention, the loss function is used as the objective function of the optimization algorithm. The model is iteratively optimized according to the reconstruction loss to minimize the loss function, and then the parameters of the model are adjusted so that the model can better approximate the real result.
[0074] Preferably, the model compression of the trained spectrum reconstruction network model is specifically:
[0075] Performing model updating on the trained spectral reconstruction network model to obtain an updated spectral reconstruction network model;
[0076] Performing model sparsification on the updated spectral reconstruction network model to obtain a pruned spectral reconstruction network model;
[0077] The pruned spectral reconstruction network model is quantized to obtain a compressed spectral reconstruction network model.
[0078] In the embodiment of the present invention, the compressed model occupies a small storage space while maintaining accurate and efficient computing and processing, which is conducive to being deployed in mobile devices for real-time use.
[0079] Preferably, the method further includes evaluating the compressed spectral reconstruction network model through evaluation indicators, and optimizing model parameters of the compressed spectral reconstruction network model according to the evaluation.
[0080] Specifically, the evaluation indicators include PSNR and MRAE. PSNR is used to evaluate the mean square error of the model, and MRAE is used to evaluate the mean relative absolute error of the model. The compressed spectral reconstruction network model is verified through the spectral verification set to obtain multiple verification spectral images reconstructed by the compressed spectral reconstruction network model;
[0081] Performing mean square error calculation on multiple verification spectral images reconstructed by the compressed spectral reconstruction network model to obtain a first PSNR index, performing mean relative absolute error calculation on multiple verification spectral images reconstructed by the compressed spectral reconstruction network model to obtain a first MRAE index, and performing weighted calculation on the first PSNR index and the first MRAE index to obtain a first evaluation index;
[0082] Performing mean square error calculation on multiple training reconstructed spectral images reconstructed by the training spectral reconstruction network model to obtain a second PSNR index, performing mean relative absolute error calculation on multiple training reconstructed spectral images reconstructed by the training spectral reconstruction network model to obtain a second MRAE index, and performing weighted calculation on the second PSNR index and the second MRAE index to obtain a second evaluation index;
[0083] The first evaluation index is compared with the second evaluation index. If the comparison result (ie, the index error) is within 10%, the model is practical.
[0084] Preferably, the training spectrum reconstruction network model is updated to obtain an updated spectrum reconstruction network model, specifically:
[0085] The first common convolution layer, the second common convolution layer and the third common convolution layer of the trained spectral reconstruction network model are all replaced with depthwise separable convolution layers to obtain an updated spectral reconstruction network model.
[0086] Specifically, all convolutional layers (i.e., the first common convolutional layer, the second common convolutional layer, the third common convolutional layer, and the attention convolutional layer in each attention block) of the original network (i.e., the initial spectral reconstruction network model after training) are replaced with a depthwise separable convolution layer composed of 3×3 convolution kernels. The depthwise separable convolution layer extracts features from the input image through depthwise separable convolution, which is a special convolution method designed to reduce the amount of calculation and increase the calculation speed, especially on mobile and embedded devices.
[0087] The depth-wise separable convolution layer performs depth-wise convolution operations (i.e. channel-by-channel convolution operations) and point-wise convolution operations (i.e. point-by-point convolution operations) on the input image.
[0088] Among them, Figure 3 As shown in the figure, the Depth-wise convolution operation is the first step of the depth-wise separable convolution. Each convolution kernel is responsible for only one input channel (channel input), that is, one channel is convolved by only one convolution kernel (3×3 convolution kernel Filters). Therefore, the number of channels of the generated image is equal to the number of channels of the input layer, that is, the number of convolution kernels. This method cannot expand the number of feature information (feature map) of feature maps. Each channel is convolved independently, so it may not be possible to effectively utilize the feature information of different channels at the same spatial position, so point-wise convolution operation is performed.
[0089] like Figure 4 As shown in the figure, the Point-wise convolution operation is the second step of the depth-wise separable convolution. Since the Depth-wise convolution operation cannot expand the number of feature information, the Point-wise convolution operation is used to expand the dimension of these feature information and combine them to generate new feature information. The convolution operation in this step is similar to the ordinary convolution operation. It combines the feature information obtained in the previous step in the depth direction by weighting to generate new feature information of the feature map, thereby expanding the number of channels.
[0090] In the embodiment of the present invention, channel-by-channel convolution performs convolution on each input channel separately, while point-by-point convolution combines the results of channel-by-channel convolution and increases the number of channels to the required size. This decomposition method greatly reduces the number of parameters, because the convolution of each point no longer calculates all input channels, and less computing resources can be used for training and reasoning, thereby improving operating efficiency.
[0091] Preferably, if Figure 5 As shown, the updated spectrum reconstruction network model includes a plurality of neurons;
[0092] The updating spectrum reconstruction network model is subjected to model sparsification to obtain a pruned spectrum reconstruction network model, specifically:
[0093] Verifying the updated spectral reconstruction network model through the spectral data set (i.e., spectral verification set) to obtain a verified spectral reconstruction network model;
[0094] The multiple neurons of the verification spectrum reconstruction network model are calculated by the APoZ indicator expression to obtain multiple activation sparse indicators. The APoZ indicator expression is:
[0095]
[0096] in, represents the activation sparse index of the cth channel in the i-th layer, represents the output of the jth position of the cth channel in the i-th layer, N represents the number of image samples used for verification, and M represents Output feature map size, k represents the index of the neuron;
[0097] The plurality of activation sparsity indices are respectively compared with a preset index threshold, and neurons corresponding to activation sparsity indices less than the threshold are pruned (i.e., neurons are pruned to reduce the number of neurons) to obtain a pruned spectral reconstruction network model.
[0098] It also includes weight initialization of the pruned spectral reconstruction network model, specifically:
[0099] The network weights of all neurons in the updated spectral reconstruction network model are copied to the weight positions of neurons at corresponding positions in the pruned spectral reconstruction network model to complete the weight initialization (i.e., reinitialization) of the pruned spectral reconstruction network model, and the initialized pruned spectral reconstruction network model is retrained through the spectral training set to obtain the pruned spectral reconstruction network model.
[0100] It also includes three processes: cyclic model pruning (i.e., sparse model), model initialization and model retraining (i.e., retraining the model after weight initialization through spectral training set) until the model reaches the expected size and accuracy, and the model compression is completed.
[0101] It should be understood that neurons represent the basic units in deep learning networks, and the number, hierarchy, and connection mode of neurons are usually determined by defining the network structure. For the weights corresponding to the neurons pruned during pruning, their values are copied to the corresponding positions in the pruned network model (i.e., the pruned spectral reconstruction network model) to complete the initialization of the pruned model; copying the weights of the neurons can maintain the continuity of the network before and after pruning, which helps to maintain the performance of the network. The initialized pruned spectral reconstruction network model is retrained with the spectral training set to adjust the values of the remaining weights to restore and improve the model performance; and this retraining (i.e., retraining) process only requires a smaller learning rate and a smaller number of iterations, avoiding the destruction of the effective computational features that the network has learned before pruning.
[0102] In an embodiment of the present invention, the model is pruned to remove neurons that have a smaller impact on the model calculation process to compress the model, and the pruned model is weight initialized and retrained to avoid a decrease in the calculation accuracy of the model.
[0103] Preferably, the pruned spectral reconstruction network model is quantized to obtain a compressed spectral reconstruction network model, specifically:
[0104] The pruned spectral reconstruction network model is subjected to 8-bit quantization processing to obtain a compressed spectral reconstruction network model.
[0105] Specifically, the present invention uses 8-bit quantization processing (i.e., model quantization) for the pruned spectral reconstruction network model, including: converting the weights of the parameters in the model from 32-bit floating point numbers to low-precision representations (such as 8-bit integers) to reduce the consumption of storage and computing resources. The numerical quantization processing can also be other numerical quantizations, such as 1 bit, 16 bits, and 8 bits are generally used.
[0106] It should be understood that, usually during model inference (i.e., when performing calculations using a quantized model), pseudo-quantization technology can be used to temporarily convert low-precision parameters back to 32-bit floating-point numbers for calculation to reduce precision loss.
[0107] In the embodiment of the present invention, 8-bit quantization is used for the model, which compresses the model while maintaining the model accuracy and placing lower requirements on the model reasoning stage.
[0108] Preferably, if Figure 6 As shown, the method of replacing the camera of the mobile device with a pre-assembled light field camera is specifically as follows:
[0109] The image sensor 1, the liquid crystal microlens array 2, the polarizer 3 and the main lens 4 are placed in advance in the order of the image sensor 1, the liquid crystal microlens array 2, the polarizer 3 and the main lens 4, and assembled into a light field camera;
[0110] The camera of the mobile device is replaced with the light field camera.
[0111] Specifically, the CMOS sensor of the mobile device is retained, the original camera on the mobile device is removed, and the light field camera is integrated into the mobile device and connected to the CMOS sensor of the mobile device. The light field camera is used to collect the light field image of the material object 5, specifically including: controlling the single chip microcomputer of the mobile device to adjust the voltage of the liquid crystal micro lens array through the Bluetooth function of the mobile device, obtaining the effective focal length, so as to collect a clear light field image.
[0112] In the embodiment of the present invention, the liquid crystal microlens array can encode the direction information of the incident light into a two-dimensional image. By analyzing the image under each liquid crystal microlens, the directional distribution of the light can be obtained, and then the depth information of the scene can be reconstructed. Therefore, the liquid crystal microlens array is assembled into a light field camera with other optical devices, which can be integrated into a mobile device to form a portable light field spectral imager, which can collect the light field image of the target object at any time and then reconstruct the spectral image.
[0113] Preferably, light field images of a plurality of different material objects are collected respectively by a light field spectrum imaging device, and the collected light field images are reconstructed by the light field spectrum imaging device to obtain a plurality of corresponding spectrum images;
[0114] A plurality of corresponding spectral curve graphs are generated according to each spectral image, and each material category and the corresponding plurality of spectral curve graphs are constructed into a material data set.
[0115] Among them, the spectral image is composed of multiple sub-band spectral images;
[0116] Generate multiple corresponding spectral curve graphs according to each spectral image, specifically:
[0117] Pixel values at the same pixel points in multiple sub-band spectral images are extracted, and a spectral curve graph is constructed according to the multiple pixel values, and so on, until the construction of multiple spectral curve graphs corresponding to each material is completed, thereby obtaining multiple spectral curve graphs.
[0118] Specifically, 100 light field images of 10 kinds of material objects are collected respectively by a light field spectrum imaging device, and the 100 light field images of the 10 kinds of material objects are reconstructed respectively to obtain 100 spectrum images corresponding to the 10 kinds of material objects;
[0119] Among them, the collected light field images are in the visible light band of 400nm to 700nm (nanometers). Each light field image is divided into 31 sub-band light field images with a step size of 10nm (nanometers), so each corresponding spectral image is also composed of 31 sub-band spectral images.
[0120] A pixel point is randomly selected on each spectral image of the 10 material objects, and a spectral curve graph corresponding to each spectral image is generated according to the pixel value of the pixel point, and the categories of the 10 materials and the corresponding multiple spectral curve graphs are constructed into a material data set;
[0121] Among them, a pixel point is randomly selected on each spectral image, and a spectral curve graph corresponding to each spectral image is generated according to the pixel value of the pixel point, specifically:
[0122] At the same pixel point of the 31 sub-band spectral images of the spectral image, the corresponding pixel values are extracted respectively, and the 31 pixel values are plotted into a line to obtain a spectral curve graph. Similarly, a spectral curve graph corresponding to each spectral image is generated.
[0123] It should be understood that 100 groups of band spectrum images of light field and spectral information of various materials are collected by light field spectrum imaging equipment, and each group is 31 images of 400nm-700nm band divided into 31 images at intervals of 10nm. The material categories are: iron, copper, aluminum, zinc, plastic, rubber, wool, goose feathers, red gouache and red color powder, with a total of 1000 groups of light field images and 1000 groups of spectrum images. A pixel point is randomly selected on each group of spectrum images of 10 materials, and 1000 spectrum curve graphs of 10 materials are drawn; wherein, the drawing of the spectrum curve graph is to extract the intensity information (i.e., pixel value) of the pixel at the same position on the 31 sub-band spectrum images and draw it into a line. Therefore, 10,000 spectrum curve graphs are generated.
[0124] It also includes constructing a material data set by combining 10,000 spectral curves with corresponding material categories, and dividing the material data set into a material training set and a material test set in a ratio of 8:2.
[0125] Generally, light field images of various materials can also be collected through a light field camera, the light field image format can be unified into jpg or png format, the collected light field images can be reconstructed through a spectral reconstruction network to obtain multiple corresponding spectral images, and the multiple light field images and multiple spectral images can be preprocessed, and the file formats of the preprocessed light field images and spectral images can be changed to files in mat format.
[0126] Preferably, the pre-constructed initial MobileNetV3 network model is trained with the material training set to obtain a trained MobileNetV3 network model, and the material test set is predicted with the trained MobileNetV3 network model to obtain a MobileNetV3 network model.
[0127] When training the model or predicting the data set through the model, the model compares the spectral curve of the target material with the spectral curves of various materials, and outputs the material category corresponding to the spectral curve with the highest overlap as the result.
[0128] It should be understood that when training a model, it includes optimizing the model through a classification loss function. MobileNetV3 is a lightweight deep convolutional neural network, an existing, mature network designed for mobile devices and embedded systems, with efficient inference speed and low model size, and has achieved excellent performance in tasks such as image classification, object detection, and semantic segmentation on mobile terminals.
[0129] In the embodiment of the present invention, MobileNetV3 is used as a material category classification model, and the material category is identified based on the spectral curve graph to achieve high-precision category identification.
[0130] Preferably, the collecting of a plurality of light field images corresponding to a plurality of materials at different angles and environments by the light field spectrum imaging device and reconstructing a plurality of corresponding spectrum images further includes:
[0131] The light field spectrum imaging device is used to collect multiple light field images corresponding to multiple materials at different angles and environments, and the parameter type of the spectrum reconstruction network model is restored through a pseudo-quantization expression. The pseudo-quantization expression is:
[0132]
[0133] Wherein, Q(x, scale, zero_point) represents a quantization function, round() represents rounding operation of network parameters, x represents network parameters, scale represents stretching ratio, and zero_point represents the data type corresponding to zero value (floating point number is selected in the present invention);
[0134] The plurality of light field images are reconstructed by a spectral reconstruction network model that restores network parameters to obtain a plurality of corresponding spectral images.
[0135] It should be understood that pseudo-quantization refers to quantization using low precision during storage, and restoration to normal high precision during reasoning. Because some operators in the framework layer only support floating-point operations, and the accuracy of high-precision reasoning is relatively high, it is restored to normal high precision during reasoning. The parameter type of a general deep learning model is usually 32-bit floating point, so the present invention uses 32-bit floating point numbers (i.e., 32-bit floating point numbers).
[0136] In an embodiment of the present invention, during model inference (i.e., when performing calculation processing using a quantized model), pseudo-quantization is used to temporarily restore the parameters of the network model to the model parameters before quantization, thereby reducing the loss of model calculation accuracy.
[0137] Preferably, the compressed spectral reconstruction network model is format-converted to obtain the spectral reconstruction network model, and the trained MobileNetV3 network model is format-converted to obtain the MobileNetV3 network model, both of which are:
[0138] The PTH model is converted into an ONNX model and deployed to the mobile device.
[0139] Specifically, the compressed spectral reconstruction network model and the trained MobileNetV3 network model are both pth models. The network model is converted into an onnx model using the torch.onnx.export() function in the PyTorch framework. During the conversion process, the input name and output name are set, and the batch dimension is usually set to dynamic for other subsequent operations. After the conversion is completed, an onnx model with equivalent functions to the original pth model is obtained.
[0140] In the embodiment of the present invention, after the PTH model is converted into the ONNX model, it has a high degree of cross-platform compatibility and is easy to deploy the model to other devices.
[0141] like Figure 7 As shown, an embodiment of the present invention provides a portable spectral imaging material identification system, comprising:
[0142] A first network generation module is used to download a spectral data set, construct an initial spectral reconstruction network model, train the initial spectral reconstruction network model through the spectral data set to obtain a trained spectral reconstruction network model, compress the trained spectral reconstruction network model, and convert the format of the compressed spectral reconstruction network model to obtain a spectral reconstruction network model;
[0143] A device generation module, used to replace the camera of the mobile device with a pre-assembled light field camera, deploy the spectral reconstruction network model into the mobile device, and obtain a light field spectral imaging device;
[0144] A data construction module, for collecting a plurality of light field images corresponding to a plurality of materials at different angles and in different environments through the light field spectrum imaging device, and reconstructing a plurality of corresponding spectrum images, generating a plurality of corresponding spectrum curve graphs according to each of the spectrum images, and constructing each material category and the corresponding plurality of spectrum curve graphs into a material data set;
[0145] A second network generation module is used to train the pre-constructed initial MobileNetV3 network model through the material data set, and convert the format of the trained MobileNetV3 network model to obtain a MobileNetV3 network model;
[0146] The application module is used to deploy the MobileNetV3 network model to the light field spectral imaging device, identify the type of the target material through the deployed light field spectral imaging device, obtain the material type result, and display the material type result in the visualization interface of the deployed light field spectral imaging device.
[0147] In the present invention, since the plenoptic function calculates the light waves in nature, the information carried by the light is expressed; the present invention decouples the light field information and the spectral information of the light field image through a deep neural network (i.e., a spectral reconstruction network model) to obtain spectral information, thereby characterizing the high-dimensional plenoptic function, which can better promote the research on scene information reconstruction in the field of computational photography; through optimized deployment, the spectral reconstruction network model is transplanted into a mobile device, the light field camera is integrated into the mobile device, and the light field camera integrated with a liquid crystal microlens array and other optical devices is used to collect images of the object to be identified, and the optical characteristics of the liquid crystal microlens array of the optical device are used to obtain a light field image with light field information and spectral information at the same time point, so that the spectral information in the light field image is decoupled through the spectral reconstruction network model deployed in the mobile device; a spectral curve graph is generated according to the spectral information, and through optimized deployment, the MobileNetV3 network model used for material category classification is transplanted into the mobile device, so as to achieve the purpose of portable, accurate and efficient material identification, and solve the problem that material identification can only be performed by large-scale equipment and professional operators. At the same time, the present invention greatly improves the ability of mobile devices to obtain material information in a scene, and greatly reduces the cost of imaging devices (i.e., light field cameras of mobile devices), and can be handheld and mobile, and the imaging device has a simple and portable structure.
[0148] The above-mentioned portable spectral imaging material identification system can refer to the implementation content and beneficial effects of the portable spectral imaging material identification method specifically described above, which will not be repeated here.
[0149] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.
[0150] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0151] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A portable spectral imaging material identification method, It is characterized in that The steps include: Downloading a spectral data set, constructing an initial spectral reconstruction network model, training the initial spectral reconstruction network model through the spectral data set to obtain a trained spectral reconstruction network model, compressing the trained spectral reconstruction network model, and converting the format of the compressed spectral reconstruction network model to obtain a spectral reconstruction network model; Replacing the camera of the mobile device with a pre-assembled light field camera, deploying the spectral reconstruction network model in the mobile device, and obtaining a light field spectral imaging device; The light field spectral imaging device is used to collect a plurality of light field images corresponding to a plurality of materials at different angles and in different environments, and a plurality of corresponding spectral images are reconstructed, a plurality of corresponding spectral curve graphs are generated according to each of the spectral images, and each material category and the corresponding plurality of spectral curve graphs are constructed into a material data set; The pre-constructed initial MobileNetV3 network model is trained by using the material data set, and the format of the trained MobileNetV3 network model is converted to obtain a MobileNetV3 network model; Deploy the MobileNetV3 network model to the light field spectral imaging device, perform type identification on the target material through the deployed light field spectral imaging device to obtain a material type result, and display the material type result in a visualization interface of the deployed light field spectral imaging device; The construction of the initial spectrum reconstruction network model is specifically as follows: The initial spectral reconstruction network model is constructed through activation function layers, pooling layers, residual blocks, three ordinary convolutional layers and multiple attention blocks, including: Connect the input port to the input of the first ordinary convolution layer, connect the output of the first ordinary convolution layer to the input of the first attention block, connect multiple attention blocks one by one, connect the output of the last attention block to the input of the second ordinary convolution layer, respectively connect the output of the first ordinary convolution layer and the output of the second ordinary convolution layer to the input of the residual block, connect the output of the residual block to the input of the activation function layer, connect the output of the activation function layer to the input of the pooling layer, connect the output of the pooling layer to the input of the third ordinary convolution layer, connect the output of the third ordinary convolution layer to the output port, and complete the construction of the initial spectral reconstruction network model.
2. The material identification method according to claim 1, It is characterized in that The model compression of the training spectrum reconstruction network model is specifically: Performing model updating on the trained spectral reconstruction network model to obtain an updated spectral reconstruction network model; Performing model sparsification on the updated spectral reconstruction network model to obtain a pruned spectral reconstruction network model; The pruned spectral reconstruction network model is quantized to obtain a compressed spectral reconstruction network model.
3. The material identification method according to claim 2, It is characterized in that The updating of the trained spectral reconstruction network model to obtain the updated spectral reconstruction network model is specifically as follows: The first common convolution layer, the second common convolution layer and the third common convolution layer of the trained spectral reconstruction network model are all replaced with depthwise separable convolution layers to obtain an updated spectral reconstruction network model.
4. The material identification method according to claim 2, It is characterized in that The updated spectrum reconstruction network model includes a plurality of neurons; The updating spectrum reconstruction network model is subjected to model sparsification to obtain a pruned spectrum reconstruction network model, specifically: Verifying the updated spectral reconstruction network model through the spectral data set to obtain a verified spectral reconstruction network model; The multiple neurons of the verification spectrum reconstruction network model are calculated by the APoZ indicator expression to obtain multiple activation sparse indicators. The APoZ indicator expression is: in, represents the activation sparse index of the cth channel in the i-th layer, represents the output of the jth position of the cth channel in the i-th layer, N represents the number of image samples used for verification, and M represents Output feature map size, k represents the index of the neuron; The plurality of activation sparsity indicators are respectively compared with a preset indicator threshold, and neurons corresponding to activation sparsity indicators less than the threshold are pruned to obtain a pruned spectral reconstruction network model.
5. The material identification method according to claim 1, It is characterized in that The spectral dataset includes a plurality of real light field images and a plurality of corresponding real spectral images; The initial spectrum reconstruction network model is trained by the spectrum data set to obtain a trained spectrum reconstruction network model, specifically: The initial spectral reconstruction network model is trained by using a plurality of real light field images and a plurality of corresponding real spectral images to obtain a plurality of training reconstructed spectral images; The plurality of training reconstructed spectral images and the corresponding real spectral images are calculated respectively by using a pre-constructed loss function to obtain a plurality of reconstruction losses, wherein the loss function is: Where L represents the reconstruction loss, represents the jth pixel value of the real spectral image, represents the jth pixel value of the training reconstructed spectral image, N represents the total number of pixels in the image, and ε represents the zero point of the real spectral image; A weighted calculation is performed on the multiple reconstruction losses to obtain a total loss, and the initial spectral reconstruction network model is iteratively optimized using the total loss to obtain a training spectral reconstruction network model.
6. The material identification method according to claim 1, It is characterized in that The method of replacing the camera of the mobile device with the pre-assembled light field camera is specifically as follows: The image sensor, the liquid crystal microlens array, the polarizer, and the main lens are placed in advance in the order of the image sensor, the liquid crystal microlens array, the polarizer, and the main lens, and assembled into a light field camera; The camera of the mobile device is replaced with the light field camera.
7. The material identification method according to claim 1, It is characterized in that The method of collecting a plurality of light field images corresponding to a plurality of materials under different angles and environments by the light field spectrum imaging device and reconstructing a plurality of corresponding spectrum images also includes: The light field spectrum imaging device is used to collect multiple light field images corresponding to multiple materials at different angles and environments, and the parameter type of the spectrum reconstruction network model is restored through a pseudo-quantization expression. The pseudo-quantization expression is: Among them, Q(x, scale, zero_point) represents the quantization function, round() represents the rounding operation of the network parameters, x represents the network parameters, scale represents the stretching ratio, and zero_point represents the data type corresponding to the zero value; The plurality of light field images are reconstructed by a spectral reconstruction network model that restores network parameters to obtain a plurality of corresponding spectral images.
8. The material identification method according to claim 1, It is characterized in that The spectral image is composed of a plurality of sub-band spectral images; The generating of a plurality of corresponding spectral curve graphs according to each of the spectral images is specifically as follows: Pixel values at the same pixel points in multiple sub-band spectral images are extracted, and a spectral curve graph is constructed according to the multiple pixel values, and so on, until the construction of multiple spectral curve graphs corresponding to each material is completed, thereby obtaining multiple spectral curve graphs.
9. A portable spectral imaging material identification system, It is characterized in that include: A first network generation module is used to download a spectral data set, construct an initial spectral reconstruction network model, train the initial spectral reconstruction network model through the spectral data set to obtain a trained spectral reconstruction network model, compress the trained spectral reconstruction network model, and convert the format of the compressed spectral reconstruction network model to obtain a spectral reconstruction network model; A device generation module, used to replace the camera of the mobile device with a pre-assembled light field camera, deploy the spectral reconstruction network model into the mobile device, and obtain a light field spectral imaging device; A data construction module, for collecting a plurality of light field images corresponding to a plurality of materials at different angles and in different environments through the light field spectrum imaging device, and reconstructing a plurality of corresponding spectrum images, generating a plurality of corresponding spectrum curve graphs according to each of the spectrum images, and constructing each material category and the corresponding plurality of spectrum curve graphs into a material data set; A second network generation module is used to train the pre-constructed initial MobileNetV3 network model through the material data set, and convert the format of the trained MobileNetV3 network model to obtain a MobileNetV3 network model; An application module, used to deploy the MobileNetV3 network model to the light field spectral imaging device, perform type identification on the target material through the deployed light field spectral imaging device, obtain a material type result, and display the material type result in a visualization interface of the deployed light field spectral imaging device; The construction of the initial spectrum reconstruction network model is specifically as follows: The initial spectral reconstruction network model is constructed through activation function layers, pooling layers, residual blocks, three ordinary convolutional layers and multiple attention blocks, including: Connect the input port to the input of the first ordinary convolution layer, connect the output of the first ordinary convolution layer to the input of the first attention block, connect multiple attention blocks one by one, connect the output of the last attention block to the input of the second ordinary convolution layer, respectively connect the output of the first ordinary convolution layer and the output of the second ordinary convolution layer to the input of the residual block, connect the output of the residual block to the input of the activation function layer, connect the output of the activation function layer to the input of the pooling layer, connect the output of the pooling layer to the input of the third ordinary convolution layer, connect the output of the third ordinary convolution layer to the output port, and complete the construction of the initial spectral reconstruction network model.
