A reference super-resolution method for wheat sample images based on an improved Transformer network

Abstract

The present invention relates to a wheat sample image reference super-resolution method based on an improved Transformer network, comprising: obtaining LR-HSI images, HR-HSI images, and HR-RGB images of wheat samples to form image pairs; preprocessing the obtained image pairs to obtain training samples, and dividing the training samples into a training set and a test set; building an improved Transformer network, and training the improved Transformer network using the training samples; importing the LR-HSI images and HR-RGB images of the wheat samples to be processed into the trained improved Transformer network, and outputting a reference super-resolution HR-HSI image. The present invention uses a dual-branch feature extraction module to process the training samples in a targeted manner, and uses a multi-level feature fusion module to supplement the diverse feature details of the training samples, so that the improved Transformer network has sharper edges and finer details in the wheat sample image reference super-resolution.

Description

Technical Field

[0001] The present invention relates to the technical field of hyperspectral image processing, and in particular to a wheat sample image reference super-resolution method based on an improved Transformer network. Background Art

[0002] Wheat is one of my country's most important staple crops. The development of the wheat industry is directly linked to national food security and social stability. Therefore, wheat quality analysis is crucial. Hyperspectral images are three-dimensional data cubes containing rich spectral and spatial information, containing information on hundreds of continuous spectral segments, greatly improving detection and identification capabilities. However, due to the mutual exclusivity between spectral and spatial resolutions, it is difficult to directly obtain high-spatial-resolution hyperspectral images of wheat. Using a low-resolution hyperspectral (LR-HSI) image of a wheat sample and a high-resolution RGB (HR-RGB) image as a reference to generate a high-resolution hyperspectral (HR-HSI) image can overcome the limitations of hyperspectral instrument hardware and is of great significance for the analysis of wheat quality using hyperspectral images.

[0003] Reference super-resolution involves converting LR-HSI images into high-quality HR-HSI images by compensating for the HR information in the reference image. Transformer networks excel in reference super-resolution tasks thanks to their superior feature extraction capabilities and self-attention mechanisms. However, due to the different properties of HSI and RGB images, the traditional Transformer network's indiscriminate feature extraction approach is irrational, potentially leading to a decrease in network super-resolution performance. As the number of network layers increases, the traditional Transformer network's single-scale feature fusion method is unable to incorporate diverse features, resulting in a loss of image resolution.

[0004] Therefore, in response to the above problems, how to design a reference super-resolution method that obtains different image attribute features and incorporates diverse feature details has become an urgent problem to be solved. Summary of the Invention

[0005] In order to overcome the defects of reference super-resolution image resolution loss caused by the indiscriminate feature extraction method and single-scale feature fusion method of the traditional Transformer network, the purpose of the present invention is to provide a wheat sample image reference super-resolution method based on an improved Transformer network, which can obtain different image attribute features and integrate diverse feature details.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a wheat sample image reference super-resolution method based on an improved Transformer network, the method comprising the following steps in sequence:

[0007] (1) Obtain the LR-HSI image, HR-HSI image, and HR-RGB image of the wheat sample to form an image pair;

[0008] (2) Preprocessing the acquired image pairs to obtain training samples, and dividing the training samples into a training set and a test set;

[0009] (3) Using a dual-branch feature extraction module and a multi-level feature fusion module to build an improved Transformer network, and using training samples to train the improved Transformer network;

[0010] (4) The LR-HSI image and HR-RGB image of the wheat sample to be processed are imported into the trained improved Transformer network, and the trained improved Transformer network outputs the reference super-resolution HR-HSI image.

[0011] In step (1), the wheat samples include healthy grains, moldy grains, grains with scab, underdeveloped grains, wheat straw and grain husks, the dimensions of the LR-HSI image are c×h×B, c and h are the length and width of the LR-HSI image, respectively, and B is the number of channels of the LR-HSI image; the dimensions of the HR-HSI image are kc×kh×B, k is a multiple of the reference super-resolution; the dimensions of the HR-RGB image are l×w×b, l and w are the length and width of the HR-RGB image, and b is the number of channels of the HR-RGB image.

[0012] The step (2) specifically includes the following steps:

[0013] (2a) performing threshold denoising on the image pair to remove the noise during acquisition;

[0014] (2b) Register the denoised images using the following steps:

[0015] (2b1) Convert the LR-HSI image and HR-HSI image with B channels into a pseudo-RGB image with 3 channels using the CIE 1931 color conversion function;

[0016] (2b2) ​​The pseudo-RGB image and the HR-RGB image are input into the image registration model, the registered image pairs are encapsulated to obtain training samples, and the training samples are divided into a training set and a test set.

[0017] The step (3) specifically refers to: the improved Transformer network includes a dual-branch feature extraction module and a multi-level feature fusion module:

[0018] The dual-branch feature extraction module includes an LR-HSI image feature extraction branch for extracting texture and spectral feature details of the LR-HSI image, and an HR-RGB image feature extraction branch for extracting texture and spectral feature details of the HR-RGB image. The LR-HSI image feature extraction branch and the HR-RGB image feature extraction branch are parallel structures, and both the LR-HSI image feature extraction branch and the HR-RGB image feature extraction branch are composed of a shallow feature extraction module and a deep feature extraction module.

[0019] The shallow feature extraction module adopts the Transformer encoder and decoder structure, and the HR-RGB image and the HR-RGB image after downsampling and upsampling operations are converted into feature vectors V through the HR-RGB image feature extraction branch. s , K s , the upsampling and downsampling operations refer to the bicubic interpolation function; the LR-HSI image is converted into a feature vector Q by the LR-HSI image feature extraction branch s , the shallow feature extraction function expression is:

[0020] (Q s ) shallow =MatMul((vector LR-HSI ),W Q )

[0021] (V s ,K s ) shallow =(MatMul((vector HR-RGB ),W v ),MatMul((vector HR-RGB↓↑ ),W k ))

[0022] Among them, MatMul() represents the matrix multiplication function, vector LR-HSI Represents the original image vector of the LR-HSI image, vector HR-RGB Represents the original image vector of HR-RGB image, vector HR-RGB↓↑ Represents the original image vector of the HR-RGB image after downsampling and upsampling operations, W Q , W k and W v Both represent the linear layer weight matrix, (Q s ) shallow ,(V s ,K s ) shallow Both represent feature vectors after shallow feature extraction, ↓↑ represent image downsampling and upsampling operations;

[0023] The deep feature extraction module processes the feature vector Q through stacked convolution layers, normalization layers, activation layers and pooling layers. s , K s 、V s Further processing is done to extract feature vectors Q of different depths before each pooling layer operation. i , K i 、V i , i = 1, 2, 3, the introduction of the multi-scale residual module retains the shallow features of the image pair; the multi-scale residual module uses 1*1, 3*3 and 5*5 convolutions in parallel to obtain the shallow feature mapping features. The feature vector space after the 1*1 convolution adjustment series is consistent with the original feature space. The training sample is converted into feature vectors Q, K, V through the dual-branch feature extraction module and the multi-scale residual module. The mathematical expression of the deep feature extraction block is as follows:

[0024] (Q s ) deep =ReLU(FC(MSA((Q s ) shallow )))

[0025] (V s ,K s ) deep =ReLU(FC(MSA((V s ,K s ) shallow )))

[0026] Among them, MSA() is the multi-head attention module function, FC() is the fully connected layer function, ReLU() is the activation layer function, (Q s ) deep Denotes the eigenvector (Q s ) shallow The feature vector after deep feature extraction by LR-HSI image feature extraction branch, (V s ,K s ) deep Represents the eigenvector (V s ,K s ) shallow Feature vector after deep feature extraction by HR-RGB image feature extraction branch;

[0027] The mathematical expression of the multi-scale residual module is as follows:

[0028]

[0029] Among them, Conv 1*1 is a 1*1 convolution, Represents the processing of shallow feature vectors by 1*1, 3*3 and 5*5 convolutions, and M is the output of the multi-scale residual module;

[0030] The function expression of the dual-branch feature extraction module is:

[0031] (Q)=((Q s ) deep )+M

[0032] (K, V) = ((K s ) deep ,(V s ) deep )+M

[0033] Among them, (Q s ) deep ,((K s ) deep ,(V s ) deep ) denotes the feature vector after deep feature extraction, and M is the output of the multi-scale residual module;

[0034] The multi-level feature fusion module combines the feature vectors Q, K, V and Q i , K i 、V i , to perform fusion at different levels, the multi-level feature fusion module first combines the feature vectors Q, K, V with each level of Q i , K i 、V i Concatenate to generate feature vector Q + , K + 、V + ; Then, the fusion weights are calculated by the dual attention module DAB and compared with the feature vectors Q at different levels. + , K + 、V + After multiplication, diverse features are obtained. The sizes of diverse features are adjusted to be consistent with the feature space through upsampling interpolation functions of different scales, 1*1 convolution and linear layers. The upsampling interpolation function is usually bicubic interpolation.

[0035] The calculation formula of the multi-level feature fusion module is as follows:

[0036] F i+1 =F i ·DAB(Q + , K + 、V + )

[0037] F=conv 1*1 (concat(F1,F2↑,F3↑↑))

[0038] Where i = 1, 2, 3, Q + , K + 、V + Represents the feature vectors Q, K, V and Q at each level respectively i , K i 、V i The concatenated feature vector, F i represents the feature vector obtained by multiplying the weight value obtained by the dual attention module DAB with the original feature vector, F represents the feature vector after the fusion of multiple features, ↑ represents 2x bicubic interpolation upsampling, and ↑↑ represents 4x bicubic interpolation upsampling.

[0039] In step (3), the improved Transformer network is trained using the L1 loss function, and the mathematical expression of the L1 loss function is:

[0040]

[0041] Among them, ∑ is the summation operation, To perform square root operation, is the true label distribution of the input data, is the predicted value of the improved Transformer network, the superscript k represents the kth sample in the training sample, and N is the total number of samples in each batch.

[0042] First, it can be seen from the above technical scheme that the beneficial effects of the present invention are: first, a dual-branch feature extraction module is constructed, and the texture and spectral feature details of the LR-HSI image and HR-RGB image of the wheat sample are extracted respectively through the parallel structure of the LR-HSI image feature extraction branch and the HR-RGB image feature extraction branch, and a multi-scale residual module is further introduced to retain rich shallow features; second, a multi-level feature fusion module is constructed, and features of different depths are extracted to obtain sufficient diverse feature details, and a dual attention block is introduced to balance the information interaction between global information and local representation, avoiding problems such as local sharpening of the image or blurring of local details; third, compared with the existing technology, the training samples are targeted processed by the dual-branch feature extraction module, and the diverse feature details of the training samples are supplemented by the multi-level feature fusion module, so that the improved Transformer network has sharper edges and finer details in the reference super-resolution of the wheat sample image. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a flow chart of the method of the present invention;

[0044] Figure 2 Schematic diagram of a dual-branch feature extraction module in the present invention;

[0045] Figure 3 Schematic diagram of the multi-scale feature fusion module in the present invention;

[0046] Figure 4 Schematic diagram of the structure of the multi-scale residual module in the present invention;

[0047] Figure 5 Schematic diagram of the structure of the dual attention module in the present invention;

[0048] Figure 6 Schematic diagram of the reference super-resolution results of six types of wheat sample images in the present invention. DETAILED DESCRIPTION

[0049] like Figure 1 As shown in FIG, a reference super-resolution method for wheat sample images based on an improved Transformer network comprises the following steps in sequence:

[0050] (1) Obtain the LR-HSI image, HR-HSI image, and HR-RGB image of the wheat sample to form an image pair;

[0051] (2) Preprocessing the acquired image pairs to obtain training samples, and dividing the training samples into a training set and a test set;

[0052] (3) Using a dual-branch feature extraction module and a multi-level feature fusion module to build an improved Transformer network, and using training samples to train the improved Transformer network;

[0053] (4) The LR-HSI image and HR-RGB image of the wheat sample to be processed are imported into the trained improved Transformer network, and the trained improved Transformer network outputs the reference super-resolution HR-HSI image.

[0054] In step (1), the wheat samples include healthy grains, moldy grains, grains with scab, underdeveloped grains, wheat straw and grain husks, the dimensions of the LR-HSI image are c×h×B, c and h are the length and width of the LR-HSI image, respectively, and B is the number of channels of the LR-HSI image; the dimensions of the HR-HSI image are kc×kh×B, k is a multiple of the reference super-resolution; the dimensions of the HR-RGB image are l×w×b, l and w are the length and width of the HR-RGB image, and b is the number of channels of the HR-RGB image.

[0055] The step (2) specifically includes the following steps:

[0056] (2a) performing threshold denoising on the image pair to remove the noise during acquisition;

[0057] (2b) Register the denoised images using the following steps:

[0058] (2b1) Convert the LR-HSI image and HR-HSI image with B channels into a pseudo-RGB image with 3 channels using the CIE 1931 color conversion function;

[0059] (2b2) ​​The pseudo-RGB image and the HR-RGB image are input into the image registration model, the registered image pairs are encapsulated to obtain training samples, and the training samples are divided into a training set and a test set.

[0060] The step (3) specifically refers to: the improved Transformer network includes a dual-branch feature extraction module and a multi-level feature fusion module:

[0061] like Figure 2 As shown, the dual-branch feature extraction module includes an LR-HSI image feature extraction branch for extracting texture and spectral feature details of the LR-HSI image, and an HR-RGB image feature extraction branch for extracting texture and spectral feature details of the HR-RGB image. The LR-HSI image feature extraction branch and the HR-RGB image feature extraction branch are parallel structures, and both the LR-HSI image feature extraction branch and the HR-RGB image feature extraction branch are composed of a shallow feature extraction module and a deep feature extraction module.

[0062] The shallow feature extraction module adopts the Transformer encoder and decoder structure, and the HR-RGB image and the HR-RGB image after downsampling and upsampling operations are converted into feature vectors V through the HR-RGB image feature extraction branch. s , K s , the upsampling and downsampling operations refer to the bicubic interpolation function; the LR-HSI image is converted into a feature vector Q by the LR-HSI image feature extraction branch s , the shallow feature extraction function expression is:

[0063] (Q s ) shallow =MatMul((vector LR-HSI ),W Q )

[0064] (V s ,K s ) shallow =(MatMul((vector HR-RGB ),W v ),MatMul((vector HR-RGB↓↑ ),W k ))

[0065] Among them, MatMul() represents the matrix multiplication function, vector LR-HSI Represents the original image vector of the LR-HSI image, vector HR-RGB Represents the original image vector of HR-RGB image, vector HR-RGB↓↑ Represents the original image vector of the HR-RGB image after downsampling and upsampling operations, W Q , W k and W v Both represent the linear layer weight matrix, (Q s ) shallow ,(V s ,K s ) shallow Both represent feature vectors after shallow feature extraction, ↓↑ represent image downsampling and upsampling operations;

[0066] The deep feature extraction module processes the feature vector Q through stacked convolution layers, normalization layers, activation layers and pooling layers. s , K s 、V s Further processing is done to extract feature vectors Q of different depths before each pooling layer operation. i , K i 、V i , i = 1, 2, 3, the introduction of the multi-scale residual module retains the shallow features of the image pair; the multi-scale residual module uses 1*1, 3*3 and 5*5 convolutions in parallel to obtain the shallow feature mapping features. The feature vector space after the 1*1 convolution adjustment series is consistent with the original feature space. The training sample is converted into feature vectors Q, K, V through the dual-branch feature extraction module and the multi-scale residual module. The mathematical expression of the deep feature extraction block is as follows:

[0067] ( Q s) deep =ReLU(FC(MSA((Q s ) shallow )))

[0068] (V s ,K s ) deep =ReLU(FC(MSA((V s ,K s ) shallow )))

[0069] Among them, MSA() is the multi-head attention module function, FC() is the fully connected layer function, ReLU() is the activation layer function, (Q s ) deep Denotes the eigenvector (Q s ) shallowThe feature vector after deep feature extraction by LR-HSI image feature extraction branch, (V s ,K s ) deep Represents the eigenvector (V s ,K s ) shallow Feature vector after deep feature extraction by HR-RGB image feature extraction branch;

[0070] like Figure 4 As shown in , the mathematical expression of the multi-scale residual module is as follows:

[0071]

[0072] Among them, Conv 1*1 is a 1*1 convolution, Represents the processing of shallow feature vectors by 1*1, 3*3 and 5*5 convolutions, and M is the output of the multi-scale residual module;

[0073] The function expression of the dual-branch feature extraction module is:

[0074] (Q)=((Q s ) deep )+M

[0075] (K, V) = ((K s ) deep ,(V s ) deep )+M

[0076] Among them, (Q s ) deep ,((K s ) deep ,(V s ) deep ) denotes the feature vector after deep feature extraction, and M is the output of the multi-scale residual module;

[0077] like Figure 3 As shown, the multi-level feature fusion module combines the feature vectors Q, K, V and Q i , K i 、V i , to perform fusion at different levels, the multi-level feature fusion module first combines the feature vectors Q, K, V with each level of Q i , K i 、V i Concatenate to generate feature vector Q + , K + 、V + ; Then, the fusion weights are calculated by the dual attention module DAB and compared with the feature vectors Q at different levels. + , K+ 、V + After multiplication, diverse features are obtained. The sizes of diverse features are adjusted to be consistent with the feature space through upsampling interpolation functions of different scales, 1*1 convolution and linear layers. The upsampling interpolation function is usually bicubic interpolation.

[0078] The calculation formula of the multi-level feature fusion module is as follows:

[0079] F i+1 =F i ·DAB(Q + , K + 、V + )

[0080] F=conv 1*1 (concat(F1,F2↑,F3↑↑))

[0081] Where i = 1, 2, 3, Q + , K + 、V + Represents the feature vectors Q, K, V and Q at each level respectively i , K i 、V i The concatenated feature vector, F i represents the feature vector obtained by multiplying the weight value obtained by the dual attention module DAB with the original feature vector, F represents the feature vector after the fusion of multiple features, ↑ represents 2x bicubic interpolation upsampling, and ↑↑ represents 4x bicubic interpolation upsampling.

[0082] In step (3), the improved Transformer network is trained using the L1 loss function, and the mathematical expression of the L1 loss function is:

[0083]

[0084] Among them, ∑ is the summation operation, To perform square root operation, is the true label distribution of the input data, is the predicted value of the improved Transformer network, the superscript k represents the kth sample in the training sample, and N is the total number of samples in each batch.

[0085] like Figure 5 As shown in the figure, the dual attention module (DAB) introduces traditional convolution into the Transformer's self-attention mechanism, adjusting the network to capture more local representations while acquiring global dependencies. The dual attention module (DAB) first stacks self-attention to form a multi-head attention, achieving an effect equivalent to channel attention. It then introduces traditional convolution, replacing the fully connected layer with a convolutional structure, using the convolutional layer to supplement local information feedback.

[0086] The effects of the present invention are further described below in conjunction with simulation experiments:

[0087] 1. Simulation experiment conditions:

[0088] The computer hardware environment of the experiment of the present invention is Intel Core i9-10980XE CPU and GTX3090 GPU, the software environment is Centos 7.6 operating system, the compilation environment is PyCharm, and the deep learning framework is PyTorch; all subsequent training and testing are based on this platform. The wheat image samples used in the simulation experiment of the present invention are hyperspectral imager (SOC710-E, Surface Optics Corporation, San Diego, USA) and MV-CE060-10UC (Hikrobot, Hikvision color industrial camera), and the data are all captured in the laboratory. Referring to Table 1, there are 6 groups of wheat sample images in this experiment, 300 samples of each type, and a total of 1800 samples.

[0089] Table 1 Comparison of wheat grain dataset categories and sample sizes

[0090]

[0091] 2. Super-resolution accuracy evaluation indicators:

[0092] In order to evaluate the super-resolution performance of this method, the present invention sets the root mean square error loss function (MSE) to measure the deviation between the reconstructed data and the real data, and uses the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) as image quality evaluation indicators.

[0093] Peak signal-to-noise ratio (PSNR) is a commonly used evaluation indicator to measure the difference between the output image and the true value image. It is generally defined by the mean square error (MSE). The mathematical expressions of the mean square error loss function (MSE) and peak signal-to-noise ratio (PSNR) are as follows:

[0094]

[0095] Where m and n are the image sizes; I(i,j) represents the pixel value at coordinate (i,j) of the ground truth image; K(i,j) represents the pixel value at coordinate (i,j) of the image generated by the improved Transformer network;

[0096]

[0097] Among them, MAX I 2is the maximum pixel value of the image. If each pixel is represented by 8 bits of binary, the maximum pixel value is 255. MSE is the mean square error.

[0098] Image structural similarity (SSIM) refers to the comparison of brightness, contrast, and structure between the generated image x and the ground truth image y. For hyperspectral images, it is necessary to calculate the structural similarity (SSIM) for different bands separately, and then take the average as the global structural similarity (SSIM). Its mathematical function expression is as follows:

[0099] SSIM(x,y)=[l(x,y) α c(x,y) β s(x,y) γ ]

[0100] in,

[0101] Where μ x 、μ y represents the mean of x and y, δ x 2 , δ y 2 represents the variance of x and y, δ xy Expressed as the covariance of x and y, c1=(k1L) 2 、c2=(k2L) 2 are two constants to avoid division by zero errors. k1 is generally set to 0.01, and k2 is generally set to 0.03. L is the dynamic range of grayscale, which is determined by the image data type. If the data type is uint8, then L = 255. c3 = c2 / 2. Setting α, β, and γ to 1, we can obtain:

[0102]

[0103] 3. Simulation experiment content and result analysis:

[0104] To verify the impact of dual-branch feature extraction and multi-level feature fusion in this method on reference super-resolution results, we compared the super-resolution results of wheat sample images using a simple interpolation function method, a Transformer network, and an improved Transformer network. To ensure consistency in the operating environment, 80% of the wheat samples from each category were randomly selected as training samples for the network model, with the remaining samples used as testing samples. The batch size was set to 8, the learning rate was 0.0001, and the L1 loss function and Adam optimization algorithm were used for 5000 iterations.

[0105] like Figure 6The diagram below shows the reference super-resolution results for six wheat sample types, which is a pseudo-RGB visualization of the wheat sample LR-HSI image, the reference super-resolution HR-HSI image, and the ground truth HR-HSI image pair. The visualization and the quantitative results in Table 2 show that the texture of the reference super-resolution HR-HSI image obtained by the improved Transformer network is closer to the ground truth HR-HSI image, resulting in higher-quality reference super-resolution results.

[0106] Table 2 Comparison of wheat sample super-resolution results

[0107]

[0108] In summary, the present invention builds a dual-branch feature extraction module, and through the parallel structure of the LR-HSI image feature extraction branch and the HR-RGB image feature extraction branch, the texture and spectral feature details of the LR-HSI image and the HR-RGB image of the wheat sample are extracted respectively, and a multi-scale residual module is further introduced to retain rich shallow features; a multi-level feature fusion module is built to use features of different depths to extract features and obtain sufficient diverse feature details, and a dual attention block is introduced to balance the information interaction between global information and local representation, avoiding problems such as local image sharpening or local detail blurring; compared with the prior art, the dual-branch feature extraction module processes the training samples in a targeted manner, and the multi-level feature fusion module supplements the diverse feature details of the training samples, so that the improved Transformer network has sharper edges and finer details in the reference super-resolution of the wheat sample image.

Claims

1. A wheat sample image reference super-resolution method based on an improved Transformer network, characterized by: The method comprises the following steps in sequence: (1) Obtain the LR-HSI image, HR-HSI image, and HR-RGB image of the wheat sample to form an image pair; (2) Preprocessing the acquired image pairs to obtain training samples, and dividing the training samples into a training set and a test set; (3) Using a dual-branch feature extraction module and a multi-level feature fusion module to build an improved Transformer network, and using training samples to train the improved Transformer network; (4) The LR-HSI image and HR-RGB image of the wheat sample to be processed are imported into the trained improved Transformer network, and the trained improved Transformer network outputs the reference super-resolution HR-HSI image; The improved Transformer network includes a dual-branch feature extraction module and a multi-level feature fusion module: The dual-branch feature extraction module includes an LR-HSI image feature extraction branch for extracting texture and spectral feature details of the LR-HSI image, and an HR-RGB image feature extraction branch for extracting texture and spectral feature details of the HR-RGB image. The LR-HSI image feature extraction branch and the HR-RGB image feature extraction branch are parallel structures, and both the LR-HSI image feature extraction branch and the HR-RGB image feature extraction branch are composed of a shallow feature extraction module and a deep feature extraction module. The shallow feature extraction module adopts the Transformer encoder and decoder structure, and the HR-RGB image and the HR-RGB image after downsampling and upsampling operations are converted into feature vectors V through the HR-RGB image feature extraction branch. s , K s , the upsampling and downsampling operations refer to the bicubic interpolation function; the LR-HSI image is converted into a feature vector Q by the LR-HSI image feature extraction branch s ; The deep feature extraction module processes the feature vector Q through stacked convolution layers, normalization layers, activation layers and pooling layers. s , K s 、V s Further processing is done to extract feature vectors Q of different depths before each pooling layer operation. i , K i 、V i , i = 1, 2, 3, a multi-scale residual module is introduced to retain the shallow features of the image pair; the multi-scale residual module uses 1*1, 3*3 and 5*5 convolutions in parallel to obtain shallow feature mapping features. The feature vector space after 1*1 convolution adjustment and concatenation is consistent with the original feature space. The training sample is converted into feature vectors Q, K, V through the information of the dual-branch feature extraction module and the multi-scale residual module; The multi-level feature fusion module combines the feature vectors Q, K, V and Q i , K i 、V i , to perform fusion at different levels, the multi-level feature fusion module first combines the feature vectors Q, K, V with each level of Q i , K i 、V i Concatenate to generate feature vector Q + , K + 、V + ; Then, the fusion weights are calculated by the dual attention module DAB and compared with the feature vectors Q at different levels. + , K + 、V + After multiplication, diverse features are obtained. The sizes of diverse features are adjusted to be consistent with the feature space through upsampling interpolation functions of different scales, 1*1 convolution and linear layers. The upsampling interpolation function is bicubic interpolation.

2. The wheat sample image reference super-resolution method based on the improved Transformer network according to claim 1, characterized in that: In step (1), the wheat samples include healthy grains, moldy grains, grains with scab, underdeveloped grains, wheat straw and grain husks, the dimensions of the LR-HSI image are c×h×B, c and h are the length and width of the LR-HSI image, respectively, and B is the number of channels of the LR-HSI image; the dimensions of the HR-HSI image are kc×kh×B, k is a multiple of the reference super-resolution; the dimensions of the HR-RGB image are l×w×b, l and w are the length and width of the HR-RGB image, and b is the number of channels of the HR-RGB image.

3. The wheat sample image reference super-resolution method based on the improved Transformer network according to claim 1, characterized in that: The step (2) specifically includes the following steps: (2a) performing threshold denoising on the image pair to remove the noise during acquisition; (2b) Register the denoised images using the following steps: (2b1) Convert the LR-HSI image and HR-HSI image with B channels into a pseudo-RGB image with 3 channels using the CIE 1931 color conversion function; (2b2) ​​The pseudo-RGB image and the HR-RGB image are input into the image registration model, the registered image pairs are encapsulated to obtain training samples, and the training samples are divided into a training set and a test set.

4. The wheat sample image reference super-resolution method based on the improved Transformer network according to claim 1, characterized in that: The step (3) specifically refers to: the function expression of the shallow feature extraction module is: (Q s ) shallow =MatMul((vector LR-HSI ),W Q ) (V s ,K s ) shallow =(MatMul((vector HR-RGB ),W v ),MatMul((vector HR-RGB↓↑ ),W k )) Among them, MatMul() represents the matrix multiplication function, vector LR-HSI Represents the original image vector of the LR-HSI image, vector HR-RGB Represents the original image vector of HR-RGB image, vector HR-RGB↓↑ Represents the original image vector of the HR-RGB image after downsampling and upsampling operations, W Q , W k and W v Both represent the linear layer weight matrix, (Q s ) shallow ,(V s ,K s ) shallow Both represent feature vectors after shallow feature extraction, ↓↑ represent image downsampling and upsampling operations; The mathematical expression of the deep feature extraction module is as follows: (Q s )deep=ReLU(FC(MSA((Q s ) shallow ))) (In s ,K s )deep=ReLU(FC(MSA((V s ,K s ) shallow ))) Among them, MSA() is the multi-head attention module function, FC() is the fully connected layer function, ReLU() is the activation layer function, (Q s ) deep Denotes the eigenvector (Q s ) shallow The feature vector after deep feature extraction by LR-HSI image feature extraction branch, (V s ,K s ) deep Represents the eigenvector (V s ,K s ) shallow Feature vector after deep feature extraction by HR-RGB image feature extraction branch; The mathematical expression of the multi-scale residual module is as follows: Among them, Conv 1*1 is a 1*1 convolution, Represents the processing of shallow feature vectors by 1*1, 3*3 and 5*5 convolutions, and M is the output of the multi-scale residual module; The function expression of the dual-branch feature extraction module is: (Q)=((Q s ) deep )+M (K、V)=((K s ) deep ,(V s ) deep )+M Among them, (Q s ) deep ,((K s ) deep ,(V s ) deep ) denotes the feature vector after deep feature extraction, and M is the output of the multi-scale residual module; The calculation formula of the multi-level feature fusion module is as follows: F i+1 =F i ·DAB(Q + 、K + 、V + ) F=conv 1*1 (concat(F1,F2↑,F3↑↑)) Where i = 1, 2, 3, Q + , K + 、V + Represents the feature vectors Q, K, V and Q at each level respectively i , K i 、V i The concatenated feature vector, F i represents the feature vector obtained by multiplying the weight value obtained by the dual attention module DAB with the original feature vector, F represents the feature vector after the fusion of multiple features, ↑ represents 2x bicubic interpolation upsampling, and ↑↑ represents 4x bicubic interpolation upsampling.

5. The wheat sample image reference super-resolution method based on the improved Transformer network according to claim 1, characterized in that: In step (3), the improved Transformer network is trained using the L1 loss function, and the mathematical expression of the L1 loss function is: Among them, ∑ is the summation operation, To perform square root operation, is the true label distribution of the input data, is the predicted value of the improved Transformer network, the superscript k represents the kth sample in the training sample, and N is the total number of samples in each batch.

Images