A rice grain RGB image multispectral reconstruction method based on characteristic wave band selection and deep network

Abstract

The application discloses a kind of based on feature band selection and deep network's rice grain RGB image multispectral reconstruction method, system and storage medium, belong to agricultural information perception and spectral reconstruction field.The method includes: obtaining rice grain RGB image and corresponding full spectrum hyperspectral image;Reflectivity pre-processing is carried out to hyperspectral image and image alignment is completed;Competitive self-adapting re-weighted sampling algorithm (CARS) is used to full spectrum band set rough screening, so that band number is reduced to M candidate bands from N, and further using step projection algorithm (SPA) is screened to obtain K optimal feature band;The training dataset with RGB image as input, with K channel feature band reflectivity as output label is constructed;Spectral reconstruction network containing feature extraction module, multistage global-spectrum direction-local enhancement module (GSL) and reconstruction head is trained;The RGB image to be measured is input into the model trained to output K channel reconstruction spectral image.The application reduces spectral dimension and redundant information while improving feature band reconstruction accuracy and model generalization ability, and can be used for rice grain variety identification and quality detection and other applications.

Description

Technical Field

[0001] This invention relates to the fields of spectral imaging, agricultural information acquisition and deep learning image processing, and in particular to a method for reconstructing RGB images of rice grains into multi-band spectral images using a deep learning model under controlled imaging conditions. Background Technology

[0002] Hyperspectral imaging can simultaneously acquire spatial information and continuous narrowband spectral information of a target, making it valuable for applications such as agricultural quality assessment, variety identification, and component analysis. However, traditional hyperspectral imaging equipment is typically expensive and bulky, and often employs a pushbroom-style line-by-line scanning method. The acquisition process is sensitive to mechanical synchronization and illumination stability, resulting in complex operation and low efficiency, making it difficult to promote in large-scale applications or low-cost scenarios.

[0003] In contrast, RGB cameras are low-cost, fast to image, and easy to deploy, but RGB only contains three channels and cannot directly provide fine-grained spectral distribution information. To compensate for this deficiency, existing research has attempted to replace part of the hyperspectral acquisition task with ordinary RGB imaging by establishing RGB-to-multi / hyperspectral mapping models. Although existing methods can recover spectral information to some extent, they still have the following shortcomings: First, some methods are highly dependent on camera spectral sensitivity or prior models, and lack generalization ability across devices and scenes; second, some methods lack explicit constraints on spatial details and spectral changes, which can easily lead to blurred edge details or spectral distortion; third, in the underdetermined mapping problem, metamerism may lead to unstable reconstruction results, resulting in insufficient reproducibility of the reconstructed spectrum under different samples and conditions.

[0004] Therefore, there is an urgent need for a multispectral reconstruction method for RGB images that is suitable for rice grain samples, can maintain low cost and high efficiency, take into account spatial details and spectral morphology, and has good stability and reproducibility. Summary of the Invention

[0005] The purpose of this invention is to provide a multispectral reconstruction method for rice grain RGB images based on RMSNet. By constructing a controlled acquisition of paired datasets and combining multi-scale context modeling, gradient attention mechanism and normalized attention mechanism, a high-precision reconstruction of the characteristic band multispectral image of rice grains from RGB images can be achieved, thereby reducing the dependence on hyperspectral hardware.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] S1. Sample preparation and controlled imaging: Place rice grain samples in a dark room or a light-proof environment, and simultaneously acquire RGB images and original hyperspectral images under fixed geometric structure and stable light source conditions; to ensure that the spatial position of the samples is consistent under dual cameras, a sliding rail or fixed support structure can be used to make the fields of view of the two cameras consistent.

[0008] S2. Hyperspectral radiometric correction: To eliminate the influence of dark current, environmental noise and light source inhomogeneity on the original hyperspectral image, a white reference image Rw and a dark reference image Rd are acquired, and the original image Rr is corrected to obtain the corrected reflectance image Rc; the correction relationship can be expressed as: Rc=(Rr−Rd) / (Rw−Rd).

[0009] S3. Feature Band Selection and Paired Dataset Construction: The corrected full-spectrum reflectance data obtained in step S2 is subjected to a competitive adaptive reweighted sampling algorithm (CARS) for coarse feature screening. CARS combines a partial least squares regression (PLSR) model, using adaptive reweighted sampling iterations to retain wavelength variables with large absolute regression coefficients and eliminate invalid bands with low weights. Monte Carlo cross-validation is used to determine the optimal size of the feature subset to avoid overfitting. After CARS screening, the number of full-spectrum bands is reduced from N to M candidate bands. After obtaining the candidate bands through coarse screening, a successive projection algorithm (SPA) is further used to refine the selection of the M candidate bands. Iterative projection analysis reduces inter-band collinearity and redundancy, obtaining K optimal feature bands while ensuring no loss of key spectral information. Here, N, M, and K are positive integers, and N > M > K. The band images corresponding to the K feature bands are extracted, and one-to-one training sample pairs are established with the synchronously acquired RGB images.

[0010] S4. Data Preprocessing: Register, crop, and normalize the RGB image and the target multispectral image to form training / validation / test data; and use lightweight augmentation methods such as random flipping to improve the robustness of the model.

[0011] S5. Constructing the RMSNet Spectral Reconstruction Network: The network includes an initial feature extraction module (SFE), a backbone network formed by stacking multiple global spectral local blocks (GSL), a normalized attention module (NAM), and a spectral reconstruction head (SRH). GSL employs a multi-scale convolutional structure to acquire local contextual information and adaptively emphasizes effective features through a gated fusion mechanism. Spatial gradient attention (SGA) and spectral gradient attention (PGA) are introduced within GSL to enhance spatial edge details and features sensitive to spectral changes. NAM uses the scaling factor γ of the normalized layer to estimate channel importance, achieving lightweight channel recalibration. SRH maps backbone features to spectral outputs of the target band number and combines cross-layer residual fusion to stabilize training and preserve low-frequency information.

[0012] S6. Model Training and Joint Loss Constraints: The network is trained using an end-to-end supervised learning approach. The loss function includes at least a spectral numerical error term and a spectral angle constraint term to simultaneously constrain the numerical error and spectral shape differences in the reconstructed spectrum, thereby improving reconstruction accuracy and result stability. During the prediction phase, the RGB image of the rice grain to be tested is input into the trained network, which outputs a K-channel target band multispectral reconstruction result.

[0013] Furthermore, in step S3, the number of optimal feature bands is K; the optimal feature bands are adaptively determined from the full spectrum data by a two-level screening process of CARS and SPA, and can be adjusted according to sample category, light source spectrum distribution and sensor response characteristics.

[0014] Furthermore, in step S5, the gated fusion mechanism can suppress redundant information by nonlinearly modulating the fused features after splicing the multi-branch features using 1×1 convolution and Sigmoid gating.

[0015] Further, in step S6, the joint loss function can be expressed as: L = α·L1(\hat{x},x) + β·SAM(\hat{x},x), where \hat{x} is the reconstructed spectrum, x is the true spectrum, and α and β are weighting coefficients. SAM stands for Spectral Angle Mapper, which can be expressed as: SAM(\hat{x},x) = arccos((\hat{x}·x) / (||\hat{x}||·||x||)).

[0016] Compared with the prior art, the present invention has at least the following beneficial effects:

[0017] (1) By reconstructing RGB to multispectral, the dependence on high-cost hyperspectral equipment is reduced, and the acquisition efficiency and deployment convenience are improved;

[0018] (2) Improve the consistency and reproducibility of spectral data and reduce the impact of illumination fluctuations and noise by using white / dark reference correction;

[0019] (3) By selecting characteristic bands, the spectral dimension and data redundancy are significantly reduced, while maintaining coverage of key response intervals;

[0020] (4) Improve the accuracy of edge detail and spectral shape reconstruction by combining multi-scale context modeling with spatial / spectral gradient attention;

[0021] (5) Lightweight channel recalibration is achieved through the normalized attention module, which improves model stability and reduces inference overhead;

[0022] (6) By constraining both spectral values ​​and spectral shape through joint loss, the reconstruction accuracy and result stability are improved. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the overall process of the method of the present invention;

[0024] Figure 2 This is a schematic diagram of the characteristic band screening process based on CARS and SPA;

[0025] Figure 3 This is a schematic diagram of the RMSNet network structure. Detailed Implementation

[0026] The specific embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the following embodiments are only used to illustrate the technical solutions of the present invention, and equivalent substitutions made by those skilled in the art to parameters, step order, and implementation methods without departing from the concept of the present invention are all within the protection scope of the present invention.

[0027] This embodiment constructs RGB-multispectral paired data using rice grains as the object. The sample may include multiple rice varieties, with each variety's grains packaged separately and stored in a dry, cool, and ventilated environment. Before collection, the surface is cleaned and spread out to reduce background interference.

[0028] (a) Controlled synchronous acquisition (corresponding to step S1)

[0029] The overall acquisition system is set up in a dark room or shaded environment, using a stable light source to illuminate the sample at a fixed angle to reduce shadow interference. The hyperspectral camera and RGB camera are fixed on the same structural base or the middle of the slide rail, and rice grains are placed on the loading platform; during the acquisition process, the hyperspectral and RGB cameras are aligned and acquired in the same spatial position by moving the platform or synchronously triggering the cameras, thereby obtaining a one-to-one corresponding RGB image and hyperspectral data cube.

[0030] (II) Hyperspectral correction and target band extraction (corresponding steps S2-S3)

[0031] To eliminate the effects of dark current and light source inhomogeneity, a white reference image Rw (e.g., using a PTFE reference plate with a nominal reflectance of approximately 0.99) and a dark reference image Rd (e.g., obtained by covering the lens) are acquired. The original hyperspectral image Rr is then corrected for reflectance using the formula Rc = (Rr − Rd) / (Rw − Rd) to obtain the corrected hyperspectral image Rc.

[0032] The corrected hyperspectral data can contain a large number of bands (denoted as N bands, where N is a positive integer). To reduce redundancy and improve training efficiency, a two-stage band selection strategy is adopted to determine the target output bands: First, the competitive adaptive reweighted sampling algorithm CARS combined with partial least squares regression (PLSR) is used to perform a coarse feature screening of the entire spectrum. Iterative adaptive reweighted sampling selects wavelength variables with larger absolute values ​​of regression coefficients and removes invalid bands with lower weights. Monte Carlo cross-validation is then used to determine the size of the candidate feature subset, resulting in M ​​candidate bands. Subsequently, the progressive projection algorithm SPA is used to refine the candidate bands, selecting the band subset with the lowest collinearity through forward progressive projection, ultimately obtaining K optimal feature bands; where M and K are positive integers and M > K. The band images corresponding to the K bands are extracted and paired with synchronized RGB images to establish samples for subsequent network training and validation, such as... Figure 2 As shown.

[0033] (III) Data preprocessing (corresponding to step S4)

[0034] The RGB image and the target multispectral image are size aligned and registered. Spatial alignment can be achieved using a calibration board or feature point matching. Subsequently, the image is cropped to obtain training sample blocks, and the pixel values ​​are linearly normalized to the range [0,1]. The samples are divided into training and test sets according to a preset ratio, and the training set can be augmented by random horizontal flipping, etc.

[0035] (iv) RMSNet network construction (corresponding to step S5, such as...) Figure 3 (As shown)

[0036] The network includes:

[0037] 1) Initial Feature Extraction Module (SFE): Consists of two 3×3 convolutional layers and a non-linear activation function, used to extract shallow texture and edge features;

[0038] 2) Backbone network: It is formed by stacking multiple Global Spectral Local Blocks (GSLs). Each GSL contains a dilated convolutional pyramid multi-branch structure. The branch features are concatenated in the channel dimension and then fused with 1×1 convolution and Sigmoid gate to obtain multi-scale context and suppress redundancy.

[0039] 3) Gradient attention mechanism: Spatial gradient attention (SGA) and spectral gradient attention (PGA) are introduced inside GSL. Spatial first-order difference and spectral difference information are used to generate attention weights to enhance edge details and spectral abrupt change regions.

[0040] 4) Normalized Attention Module (NAM): It uses the scaling factor γ of the normalization layer to estimate the channel importance, generates channel weights to recalibrate the feature response, and achieves lightweight attention enhancement.

[0041] 5) Spectral Reconstruction Head (SRH): Maps the backbone output to the spectral output of the target band number, and combines shallow baseline information and deep residual information through cross-layer residual fusion to improve convergence stability and reconstruction accuracy.

[0042] (v) Training and loss constraints (corresponding to step S6)

[0043] Supervised learning is used to train the network parameters. The loss function includes a spectral numerical error term (e.g., L1 or L2) and a spectral angle mapping (SAM) constraint term to constrain the spectral shape. During training, the Adam optimizer can be used, and parameters such as learning rate, batch size, and number of training epochs can be set; the model weights are saved when the validation metric reaches its optimum.

[0044] (vi) Reasoning output and application (corresponding to step S6)

[0045] During the inference phase, the RGB image of the rice grain to be tested is input into the trained model, which outputs a K-channel multispectral reconstruction result. This multispectral result can be used for subsequent variety identification, quality characterization, component correlation analysis, or in conjunction with other machine learning models.

[0046] It should be noted that the method described in this invention is not limited to the parameter settings in the above embodiments; without departing from the concept of this invention, the type of light source, acquisition structure, number of bands, number of network layers, loss weight, etc., can be adjusted, and these equivalent modifications should all be included within the protection scope of this invention.

Claims

1. A method for multispectral reconstruction of RGB images of rice grains based on feature band selection and deep networks, characterized in that, Includes the following steps: S1. Obtain the RGB image of the rice grain to be tested and the full-spectrum hyperspectral image corresponding to the RGB image; S2. Perform radiometric correction on the hyperspectral image to obtain reflectance data, and perform region extraction and spatial alignment on the RGB image and the hyperspectral image; S3. Feature screening of the full-spectrum bands corresponding to the reflectance data includes: firstly, using the competitive adaptive reweighted sampling algorithm CARS to coarsely screen the full-spectrum bands, reducing N bands to M candidate bands; then using the stepwise projection algorithm SPA to finely screen the M candidate bands, obtaining K optimal feature bands, where N > M > K; S4. Based on the RGB image and the K optimal feature band reflectance data, construct training samples to form a training dataset with RGB image as input and K channel feature band reflectance as output label; S5. Train the spectral reconstruction network using the training dataset to obtain the spectral reconstruction model; S6. Input the RGB image of the rice grain to be tested into the spectral reconstruction model, and output the K-channel characteristic band spectral reconstruction image.

2. The method according to claim 1, characterized in that: The CARS coarse screening in step S3 includes: A regression model was established based on partial least squares regression (PLSR), and the regression coefficients of each band variable were calculated. During multiple iterations, the band variables with larger absolute values ​​of regression coefficients were retained first, and the band variables with lower weights were removed, according to the adaptive reweighted sampling strategy. The size M of the candidate band subset was determined by Monte Carlo cross-validation.

3. The method according to claim 1, characterized in that: In step S3, the SPA fine screening uses a forward stepwise projection method to select variables for candidate bands, so as to reduce collinearity among the selected bands and retain bands with strong information independence, thereby obtaining K optimal feature bands.

4. The method according to claim 1, characterized in that: The spectral reconstruction network in step S5 includes: an initial feature extraction module (SFE), a feature enhancement backbone network composed of multiple global spectral local blocks (GSL), and a spectral reconstruction head (SRH); wherein, the RGB image is first processed by the SFE to extract initial features and then input into the GSL backbone network, and the SRH is used to map the backbone network output to a K-channel spectral reconstruction result.

5. The method according to claim 4, characterized in that: The GSL includes a parallel feature enhancement structure and an attention mechanism structure. The input features are fed into the Spectral Gradient Attention (PGA) module, the Hollow Spatial Pyramid Pooling (ASPP) module, and the Spatial Gradient Attention (SGA) module to form parallel branches. The outputs of the parallel branches are then fused to generate enhanced features.

6. The method according to claim 5, characterized in that, The GSL includes parallel feature enhancement and channel attention mechanisms, specifically: The input features are fed into the Spectral Gradient Attention (PGA), the Spatial Pyramid Pooling (ASPP), and the Spatial Gradient Attention (SGA) modules to form parallel branches. The outputs of the parallel branches are then fused and subjected to nonlinear activation and convolution transformation. Furthermore, the Channel Attention (NAM) module is introduced to adaptively weight the fused features; at the same time, residual connections are set to fuse the GSL input and GSL output to obtain the GSL output features.

7. The method according to claim 6, characterized in that, The ASPP includes: At least three parallel dilated convolution branches are used, each branch uses 3×3 convolution with different dilation rates; the outputs of each branch are concatenated and then channel-fused using 1×1 convolution to obtain the ASPP output.

8. The method according to claim 1, characterized in that: The loss function used in step S5 when training the spectral reconstruction network includes spectral reconstruction error loss and spectral angle constraint loss.

9. A spectral reconstruction system for characteristic bands of rice grains, characterized in that, include: The data acquisition module is used to acquire RGB images of rice grains and their corresponding full-spectrum hyperspectral images; The preprocessing module is used to preprocess the full-spectrum hyperspectral image and perform image alignment; The band selection module is used to reduce the number of full-spectrum bands from N to M candidate bands using CARS, and to select the K optimal characteristic bands from the M candidate bands using SPA. The model training module is used to build a training dataset and train the spectral reconstruction network to obtain a spectral reconstruction model. The spectral reconstruction module is used to input the RGB image into the spectral reconstruction model and output the K-channel feature bands to reconstruct the spectral image. The system is used to implement the method according to any one of claims 1 to 8.

10. An electronic device comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, characterized in that, When the computer program is executed by the processor, it implements the method according to any one of claims 1 to 8.

Images