A crop target detection method and system based on a spectral expansion method

Abstract

The application discloses a kind of crop target detection method and system based on spectral extension method, method includes: obtaining the target RGB image of crop;According to target RGB image, by spectral super-resolution network, based on data-driven mode carries out spectral super-resolution and obtains target hyperspectral image;According to target RGB image, by spectral extension generation confrontation network, based on data-driven mode carries out spectral extension and synthesizes target near-infrared image;Based on preset condition, according to target hyperspectral image synthesis target visible light image;In channel dimension splicing target visible light image and target near-infrared image, input early fusion target detection network obtains crop target detection result.The application reconstructs hyperspectral image and synthesizes near-infrared image by using data-driven based mode, and then applies in crop target detection, can obtain more comprehensive object attribute, can be widely applied in target detection technical field.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to a method and system for detecting crop targets based on spectral extension. Background Technology

[0002] Target detection in crops plays a crucial role in tasks such as yield estimation and pest and disease prediction. However, images acquired by single-band image sensors are affected by environmental factors and are limited by the sensor's own imaging characteristics, thus failing to comprehensively reflect the attributes of objects. Therefore, the use of single-source data limits the accuracy of target detection in downstream computer tasks.

[0003] Hyperspectral remote sensing images offer advantages such as rich information content, high spectral resolution, and a wide spectral range, containing more accurate and identifiable ground feature information, but their acquisition cost is relatively high. Near-infrared spectroscopy, belonging to the overtone and dominant frequency absorption spectra of molecular vibrational spectroscopy, has strong penetrating power and is less affected by the environment, but it lacks the representation of texture details in the target scene, making it difficult for the human eye to interpret. Visible light images are easily constrained by lighting and environmental conditions, but their acquisition is convenient and quick.

[0004] Although multi-source data contains more information about the material properties, it is difficult to capture and there are problems such as inconsistent resolution and registration difficulties in its acquisition. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a crop target detection method and system based on a spectral extension method, which reconstructs hyperspectral images and synthesizes near-infrared images in a data-driven manner and is applied to crop target detection.

[0006] On one hand, embodiments of the present invention provide a crop target detection method based on a spectral extension method, comprising:

[0007] Obtain the target RGB image of the crop;

[0008] Based on the target RGB image, a target hyperspectral image is obtained by performing spectral super-resolution using a data-driven method through a spectral super-resolution network.

[0009] Based on the target RGB image, a near-infrared image of the target is synthesized by a spectral expansion generative adversarial network using a data-driven approach.

[0010] Based on preset conditions, a visible light image of the target is synthesized from the target hyperspectral image;

[0011] The visible light image and the near-infrared image of the target are stitched together in the channel dimension, and then input into the early fusion target detection network to obtain the crop target detection result.

[0012] Optionally, it also includes:

[0013] Based on the inverse mapping between RGB images and hyperspectral images, a multi-layer cascaded spectral super-resolution network is constructed through a feature extraction module, an inter-layer feature integration module, and a feature enhancement module.

[0014] The spectral super-resolution network uses images of different resolutions as inputs for each level.

[0015] The inter-layer feature integration module performs upsampling operations through PixelShuffle.

[0016] The feature enhancement module includes residual blocks and dense blocks.

[0017] Optionally, it also includes:

[0018] The spectral super-resolution network is optimized and trained using pairs of RGB and hyperspectral images as datasets and in conjunction with a loss function.

[0019] Optionally, the spectral super-resolution network is a multi-layer cascaded architecture, and the step of obtaining the target hyperspectral image by performing spectral super-resolution based on the target RGB image through the spectral super-resolution network in a data-driven manner includes:

[0020] The target RGB image is downsampled by multiple factors using PixelUnShuffle to obtain images of various resolutions;

[0021] The original resolution image of the target RGB image and the various resolution images of the multiple resolution images are sequentially input into each layer of the spectral super-resolution network;

[0022] Feature extraction, inter-layer feature integration, and feature enhancement are performed at each level to obtain the target hyperspectral image corresponding to each level;

[0023] The inter-layer feature integration is performed by upsampling through PixelShuffle; the feature enhancement reduces artifacts through residual blocks and dense blocks.

[0024] Optionally, it also includes:

[0025] Based on the mapping relationship between RGB images and near-infrared images, an end-to-end spectral extension generative adversarial network is constructed through a generator and a discriminator.

[0026] The generator includes an encoder, a bottleneck layer, and a decoder. The encoder includes a convolutional layer, a residual channel attention instantiation module, and discrete wavelet transform downsampling. The decoder includes inverse discrete wavelet transform upsampling and a convolutional layer.

[0027] Optionally, it also includes:

[0028] The parameters of the spectral extension generative adversarial network are optimized using least-squares adversarial loss.

[0029] The generator is constrained at the pixel level by feature matching loss and perceptual loss;

[0030] The discriminator and the generator are trained alternately using pairs of RGB and near-infrared images as datasets until Nash equilibrium is reached.

[0031] Optionally, the spectral-expanded generative adversarial network includes a generator, which includes an encoder, a bottleneck layer, and a decoder. The step of synthesizing a target near-infrared image based on the target RGB image using the spectral-expanded generative adversarial network in a data-driven manner includes:

[0032] The first feature map is obtained by performing first convolution processing, residual channel attention instantiation processing, and discrete wavelet transform downsampling processing on the target RGB image through the encoder.

[0033] The second feature map is obtained by performing a second convolution on the first feature map through a bottleneck layer;

[0034] The second feature map is processed by inverse discrete wavelet transform upsampling, element-by-element addition, and third convolution to synthesize a target near-infrared image.

[0035] Optionally, it also includes:

[0036] Based on the Faster R-CNN network architecture, an early fusion target detection network is constructed through a feature extraction module, a region frame prediction module, and a classification prediction module.

[0037] The feature extraction module includes a VGG16 network with a 6-channel input layer.

[0038] Optionally, the early fusion target detection network includes a feature extraction module, a region frame prediction module, and a classification prediction module. The step of stitching the target visible light image and the target near-infrared image together along the channel dimension and inputting them into the early fusion target detection network to obtain the crop target detection result includes:

[0039] The fifth feature map is obtained by receiving and processing the stitched visible light image and near-infrared image of the target through the feature extraction module.

[0040] Candidate frames are generated by the region frame prediction module and projected onto the fifth feature map to obtain the corresponding feature matrix;

[0041] The crop target detection results are obtained by scaling and diffusion processing based on the feature matrix through the classification and prediction module.

[0042] On the other hand, embodiments of the present invention provide a crop target detection system based on a spectral extension method, comprising:

[0043] The first module acquires the target RGB image of the crop;

[0044] The second module is used to obtain a target hyperspectral image by performing spectral super-resolution based on the target RGB image through a spectral super-resolution network in a data-driven manner.

[0045] The third module is used to synthesize a target near-infrared image based on the target RGB image by generating an adversarial network through spectral expansion using a data-driven approach.

[0046] The fourth module is used to synthesize a visible light image of the target based on the target hyperspectral image under preset conditions;

[0047] The fifth module is used to stitch together the visible light image and the near-infrared image of the target in the channel dimension, and input them into the early fusion target detection network to obtain the crop target detection result.

[0048] On the other hand, embodiments of the present invention provide a crop target detection system based on a spectral extension method, including a processor and a memory;

[0049] The memory is used to store programs;

[0050] The processor executes the program to implement the method described above.

[0051] On the other hand, embodiments of the present invention provide a computer-readable storage medium storing a program that is executed by a processor to implement the methods described above.

[0052] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method.

[0053] This invention first acquires a target RGB image of the crop; based on the target RGB image, a hyperspectral image of the target is obtained through a spectral super-resolution network using a data-driven approach; based on the target RGB image, a near-infrared image of the target is synthesized through a spectral extension generative adversarial network using a data-driven approach; based on preset conditions, a visible light image of the target is synthesized from the target hyperspectral image; the visible light image and the near-infrared image of the target are stitched together in the channel dimension and input into an early fusion target detection network to obtain the crop target detection result. This invention uses multi-source data for crop target detection, utilizing hyperspectral and near-infrared images for the crop target detection task. By using a data-driven approach to reconstruct hyperspectral images and synthesize near-infrared images, this invention, when applied to crop target detection, can obtain more comprehensive object attributes. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 A schematic flowchart of a crop target detection method based on spectral extension method provided in an embodiment of the present invention;

[0056] Figure 2 This is a schematic diagram of the structure of the spectral super-resolution network provided in an embodiment of the present invention;

[0057] Figure 3 This is a schematic diagram illustrating the recombination of an RGB image from three bands extracted from a reconstructed hyperspectral image, as provided in an embodiment of the present invention.

[0058] Figure 4 This is a schematic diagram of the structure of a spectral extension generative adversarial network provided in an embodiment of the present invention;

[0059] Figure 5 This is a schematic diagram of the residual channel attention instantiation module provided in an embodiment of the present invention;

[0060] Figure 6 This is a schematic diagram of the structure of an early fusion target detection network provided in an embodiment of the present invention;

[0061] Figure 7 A schematic diagram illustrating the effect of generating near-infrared images from RGB through spectral expansion using an adversarial network under low-light conditions, as provided in an embodiment of the present invention.

[0062] Figure 8 This is a schematic diagram comparing the results of target detection under low-light conditions using RGB images obtained by screening three bands of hyperspectral imaging and synthetic near-infrared imaging with the results of target detection using only real RGB images. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0064] On the one hand, refer to Figure 1 The present invention provides a crop target detection method based on a spectral extension method, comprising:

[0065] S100: Obtain the target RGB image of the crop;

[0066] Specifically, in some embodiments, under low light conditions, multiple frames of RGB images captured by multi-source sensing devices and their corresponding hyperspectral and near-infrared images are acquired and preprocessed to serve as the dataset for network training (conventional processing methods, not the focus of this invention, and will not be elaborated upon).

[0067] S200: Based on the target RGB image, a hyperspectral image of the target is obtained by performing spectral super-resolution using a data-driven method through a spectral super-resolution network.

[0068] It should be noted that the spectral super-resolution network is a multi-layer cascaded architecture. In some embodiments, the target RGB image is downsampled by multiple factors through PixelUnShuffle to obtain images of multiple resolutions. The original resolution image of the target RGB image and the various resolution images are sequentially input into each layer of the spectral super-resolution network. Feature extraction, inter-layer feature integration, and feature enhancement are performed at each layer to obtain the target hyperspectral image corresponding to each layer. Among them, inter-layer feature integration is performed by upsampling through PixelShuffle; feature enhancement reduces artifacts through residual blocks and dense blocks.

[0069] It should also be noted that some embodiments further include: constructing a multi-layer cascaded spectral super-resolution network based on the inverse mapping of RGB images and hyperspectral images through a feature extraction module, an inter-layer feature integration module, and a feature enhancement module; wherein, each layer of the spectral super-resolution network uses images of different resolutions as layer inputs; the inter-layer feature integration module performs upsampling operations through PixelShuffle; and the feature enhancement module includes residual blocks and dense blocks.

[0070] In some embodiments, the method further includes: using pairs of RGB images and hyperspectral images as datasets, and combining them with a loss function to optimize and train the spectral super-resolution network.

[0071] Specifically, in some specific embodiments, let I HS Represents a hyperspectral image, I RGB This represents an RGB image, where for each pixel p, its RGB intensity... and spectral reflectance I HS The relationship between (p, λ) is as follows:

[0072]

[0073] In the formula, B represents the number of bands, usually B = 31, and λ is the wavelength, usually 400–700 nm. At this time, I Hs ∈R N×B I RGB ∈R N×3 Both are matrices, where N is the number of pixels and S∈R. B×3 Let c = (r, g, b) represent the spectral response function, where r, g, and b represent the R, G, and B channels, respectively. Therefore, it can also be written as:

[0074] X = YS

[0075] As can be seen from this equation, the spectral reconstruction process can be regarded as an inverse mapping from X to Y, which shows that under certain conditions, the ill-posed transformation of reconstructing hyperspectral images from RGB is achievable.

[0076] Therefore, a 4-layer cascaded spectral super-resolution network is constructed for spectral super-resolution reconstruction from RGB images, such as... Figure 2 As shown, RGB images were downsampled at their original resolution, and then downsampled by 2x, 4x, and 8x using PixelUnShuffle, resulting in four different resolution images. These images served as inputs for each layer. The module design for each layer was similar, including feature extraction, inter-layer feature integration, and feature enhancement. To reduce information loss, PixelShuffle was used for upsampling during inter-layer feature integration. In the feature enhancement module, residual blocks and dense blocks were used to reduce artifacts. The weights of the entire network model were initialized using Xavier. The network then learned the mapping relationship between visible light and hyperspectral images.

[0077] The training dataset used by the network includes self-collected datasets of paired RGB and hyperspectral images, as well as publicly available datasets such as NTIRE 2020. During the training of the network model, the L1 loss function is used to optimize the network. L1 is defined as follows:

[0078]

[0079] In the formula, x represents the RGB image, y represents the real hyperspectral image, and G(*) represents a 4-layer cascaded spectral super-resolution network. The goal of network training is to minimize the sum of the absolute values ​​of the hyperspectral image G(*) reconstructed from the RGB image and the real hyperspectral image y.

[0080] To achieve this goal, the epoch is set to 1000, with an initial learning rate of lr = 0.0001, and the learning rate variation strategy is as follows: By combining the Adam algorithm to update the weights in the network, the network can ultimately fit the distribution of sub-data in the hyperspectral image very well.

[0081] Then, the acquired target RGB image is input into the pre-trained network to reconstruct the corresponding hyperspectral image.

[0082] S300: Based on the target RGB image, a near-infrared image of the target is synthesized by spectral expansion using a generative adversarial network driven by data.

[0083] It should be noted that the spectral extension generative adversarial network includes a generator, which comprises an encoder, a bottleneck layer, and a decoder. In some embodiments, the encoder performs a first convolution, residual channel attention instantiation, and discrete wavelet transform downsampling on the target RGB image to obtain a first feature map. The bottleneck layer performs a second convolution on the first feature map to obtain a second feature map. The decoder performs inverse discrete wavelet transform upsampling, element-wise addition, and a third convolution on the second feature map to synthesize the target near-infrared image.

[0084] It should also be noted that in some embodiments, the method further includes: constructing an end-to-end spectral extension generative adversarial network based on the mapping relationship between RGB images and near-infrared images through a generator and a discriminator; wherein the generator includes an encoder, a bottleneck layer and a decoder, the encoder includes a convolutional layer, a residual channel attention instantiation module and discrete wavelet transform downsampling, and the decoder includes inverse discrete wavelet transform upsampling and a convolutional layer.

[0085] In some embodiments, the method further includes: optimizing the parameters of the spectral extension generative adversarial network using least-squares adversarial loss; constraining the generator to pixel-level loss using feature matching loss and perceptual loss; and training the discriminator and generator alternately using paired RGB images and near-infrared images as datasets until Nash equilibrium is reached.

[0086] Specifically, in some specific embodiments, let These represent the visible light image, the near-infrared image, and the synthesized near-infrared image (i.e., the target's near-infrared image), respectively. Subsequently, in the RGB near-infrared reconstruction task, the mapping relationship from the visible light image to the corresponding near-infrared image can be expressed as:

[0087]

[0088] Here, F represents the spectral extension generative adversarial network, and θ is the parameter to be optimized in the near-infrared generation task. θ is iteratively optimized via a spectral extrapolation network.

[0089] A novel end-to-end spectral extension generative adversarial network model is established, and its framework is as follows: Figure 4 As shown, this involves an adversarial game between a generator and a discriminator. The generator aims to produce synthetic near-infrared images with a distribution similar to real near-infrared data, deceiving the discriminator, which tries to distinguish the distribution differences between real and synthetic near-infrared data. In traditional methods of stacking generators within convolutional neural networks, the downsampling pooling operation and the upsampling transposed convolution operation lead to severe information loss, resulting in significant artifacts or distortion in the synthetic image. To circumvent this problem, the generator uses discrete wavelet transform to divide the feature map into high-frequency and low-frequency feature maps. This leverages the temporal and frequency domains of discrete wavelet convolution, as well as its invertible features, to retain more feature information. Figure 4 The generator architecture is shown. Furthermore, a residual channel attention instantiation module is used to balance high-frequency and low-frequency information, better preserving globally consistent informative features. Figure 5 The residual channel attention instantiation module is shown.

[0090] In the generator architecture, a 2D discrete wavelet convolutional transform with four kernels is used to downsample the feature map. First, let's introduce the prerequisites for the 2D discrete wavelet convolutional transform. The 2D discrete wavelet transform includes four kernels: one low-pass filter and three high-pass filters. It can decompose an image into one low-frequency image and three high-frequency images. Taking the Haar wavelet as an example, the four filters are defined as follows:

[0091]

[0092]

[0093] Obviously, f LL f LH f HL f HH This forms a 4×4 invertible matrix, with each pair of features being mutually orthogonal. After Haar transform, each feature map is decomposed into four sub-band images, I a ,I b ,Ic ,I d I represents the pixel intensity of the feature map. LL ,I LH ,I HL ,I HH This represents the pixel intensity of four different frequency domain sub-bands.

[0094] I LL =I a +I b +I c +I d

[0095] I LH =-I a -I b +I c +I d

[0096] I HL =-I a +I b -I c +I d

[0097] I HH =I a -I b -I c +I d

[0098] Due to the bioorthogonality of DWT, the pixel intensity of the feature map can be accurately reconstructed by inverse discrete wavelet convolution without losing information. For Haar wavelets, IDWT is defined as follows:

[0099] Ia = (ILL - ILH - IHL + IHH) / 4

[0100] Ib = (ILL - ILH + IHL - IHH) / 4

[0101] Ic = (ILL + ILH - IHL - IHH) / 4

[0102] Id = (ILL + ILH + IHL + IHH) / 4

[0103] From a mathematical perspective, the discrete wavelet convolutional transform uses four fixed convolutional filters with a stride of 2 to perform downsampling, which quadruples the number of channels in the feature map and halves its width and height. To avoid information loss, the inverse discrete wavelet transform is used for upsampling, which reduces the number of channels in the feature map to one-quarter of its original value and doubles its width and height.

[0104] The generator architecture consists of three parts: an encoder, a bottleneck layer, and a decoder. The encoder architecture comprises three repetitive, similar modules. Each module consists of a convolutional layer, two residual channel attention instantiation modules, and a discrete wavelet transform downsampling module. In the first convolutional layer of the first module, the kernel size is 7×7, the stride is 1, and the activation function is ReLU. In the first convolutional layer of the second and third modules, the kernel size is 3×3, the stride is 1, and the activation function is ReLU. This operation makes the feature map representation more compact.

[0105] Each module's convolutional layer is followed by two residual channel attention instantiation modules. In each residual channel attention instantiation module, the input feature map... The residual image is obtained by using two convolutional layers with a kernel size of 3×3 and a stride of 1. To avoid artifacts and improve the realism of the generated data, instance normalization is used. Residual image F RS The channel attention is fed into the channel attention module, using two fully connected layers and a ReLU activation function to better adaptively model the correlation between channels. Weight w c It is then normalized to a value between 0 and 1 by the Sigmoid gating mechanism. The weight w c The attention feature map is obtained by performing an inner product on the residual feature map. To preserve more detail in the feature map, features from different levels are fused together from the input feature map. With attention feature map The intermediate result feature map is obtained by performing element-wise addition.

[0106] Subsequently, the discrete wavelet transform method is used to perform downsampling operations, and each intermediate feature map is decomposed into four sub-band images (i.e., the first feature map). One of them is a low-frequency image, and the other three are high-frequency images, thus increasing the number of channels in the feature map to four times the original number.

[0107] The bottleneck layer consists of four convolutional layers with a kernel size of 3×3 and a stride of 1, using ReLU activation. These layers are used to better aggregate feature representations, thus obtaining the second feature map.

[0108] The decoder section also has three similar modules. In each module, to avoid information loss, inverse discrete wavelet transform is used for upsampling to obtain upsampled feature maps. Will Intermediate result feature map obtained from the corresponding layer of the encoder part Element-wise addition is performed. Then, the feature map is obtained by passing it through three convolutional layers with a kernel size of 3×3, a stride of 1, and an activation function of ReLU.

[0109] The designed generator architecture can simultaneously focus on global semantic information and local detail information, better balance high-frequency image details and low-frequency image content, and generate higher-quality near-infrared images (i.e., target near-infrared images).

[0110] To generate globally semantically consistent and fine-grained images, a multi-scale discriminator architecture was employed. Three discriminators, defined as D1, D2, and D3, shared the same structure. Specifically, the synthesized near-infrared image was downsampled, creating image pyramids at three scales with coefficients of 2 and 4. Each discriminator consisted of several stacked convolutional layers. The input image was processed through these convolutional layers to obtain an n×n matrix, and the average of the matrix was used as the true / false output (i.e., the final output of the discriminator). A point in the matrix could represent a perceptual region in the original image, corresponding to the location of a region in the original image. Although these discriminators shared the same structure, the discriminator operating at the coarsest scale had the largest receptive field, providing a more comprehensive view of the image and guiding the generator to produce globally consistent images. Conversely, the finest-scale discriminator encouraged the generator to produce finer details. The application of the multi-scale discriminator enabled the synthesized near-infrared image to maintain highly consistent content information within the visible light image while preserving texture information similar to that of the real near-infrared image.

[0111] If we define the RGB image as X and the true near-infrared image as Y, then we have [x i} N ∈X,{y i} N ∈Y, where N represents the total number of data sets. Since the discriminator uses a 3-scale structure, its learning is considered a multi-task learning process. The discriminator tries its best to classify real near-infrared signals as real and synthetic near-infrared signals as fake, while the generator tries its best to classify synthetic near-infrared signals as real. This leads to a game theory approach, where least-squares adversarial loss is used to optimize the network parameters, defined as follows:

[0112]

[0113]

[0114] Here, G(x) represents the synthesized near-infrared image, where k = 1, 2, 3 represent the original size image, 2x downsampling, and 4x downsampling. Let b = 1 represent the real data, a = 0 represent the synthesized data, and c = 1 represent the generator trying its best to deceive the discriminator.

[0115] To generate more realistic near-infrared images, the generator needs to be constrained by pixel-level loss. Therefore, feature matching loss and perceptual loss are added. The formula for feature matching loss is as follows:

[0116]

[0117] Since the discriminator has three scales, k is set to 1, 2, and 3 to represent these three scales. Discriminator D k The feature extractor for the i-th layer. T is the total number of layers, N i Let G(x) represent the total number of elements in each layer, x represent the visible light image, y represent the true near-infrared image, and G(x) represent the synthesized near-infrared image. The visible light and true near-infrared images are input into the discriminator to obtain the feature map. Visible light and synthetic near-infrared images are input into the discriminator. In this context, the feature matching loss is obtained by minimizing the feature extractor at each level of the discriminator. and The sum of the absolute values ​​of .

[0118] Perceptual loss is defined as follows:

[0119]

[0120] Here φ i,j The feature map obtained from the j-th convolution before the i-th max-pooling layer in the VGG19 network, where m and n represent the width and height of the feature map, respectively. The perceptual loss is defined as the Manhattan distance between the synthetic near-infrared image G(x) and the real image y.

[0121] The total loss of the generator is:

[0122]

[0123] Where λ1 and λ2 represent penalty parameters, which are set to 10.

[0124] The discriminator and generator are trained alternately using low-light RGB images and real near-infrared images until Nash equilibrium is reached, at which point training terminates. The low-light RGB images are then input into the trained spectral-expanded generative adversarial network to obtain synthetic near-infrared images.

[0125] S400: Based on preset conditions, synthesize a visible light image of the target from the target hyperspectral image;

[0126] Specifically, in some specific embodiments, a new visible light image (i.e., the target visible light image) is formed by selecting three bands with rich spectral and physical geometric information from the target hyperspectral image and combining them.

[0127] S500 stitches together the target's visible light image and target's near-infrared image in the channel dimension, and inputs them into the early fusion target detection network to obtain the crop target detection results.

[0128] It should be noted that in some embodiments, the feature extraction module receives and processes the stitched target visible light image and target near-infrared image to obtain a fifth feature map; the region frame prediction module generates candidate frames and projects them onto the fifth feature map to obtain the corresponding feature matrix; and the classification prediction module performs scaling and diffusion processing based on the feature matrix to obtain the crop target detection result.

[0129] It should also be noted that in some embodiments, the method further includes: constructing an early fusion target detection network based on the Faster R-CNN network architecture through a feature extraction module, a region frame prediction module, and a classification prediction module; wherein the feature extraction module includes a VGG16 network with a 6-channel input layer.

[0130] Specifically, in some specific implementations, Faster R-CNN is chosen as the network architecture, such as... Figure 6 As shown, it consists of three parts: a feature extraction module, a region frame prediction module, and a classification prediction module. Using an image-level fusion model, the input layer of the VGG16 network in the feature extraction module is changed from 3 channels to 6 channels (3 channels for RGB images, 3 channels for near-infrared images, and single-channel near-infrared images are copied and concatenated to form a 3-channel near-infrared image for subsequent experiments). Here, the RGB images are obtained by selecting three bands rich in spectral and physical geometric information from the hyperspectral images in step 4. The VGG16 network is modified to receive image information from both modes, obtaining the corresponding feature maps. Candidate frames are generated using a candidate frame proposal network and projected onto the feature maps to obtain the corresponding feature matrices. Each feature matrix is ​​scaled using a ROI pooling layer, and then the feature maps are diffused through a series of fully connected layers to obtain the prediction results.

[0131] The technical solution of the present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that the following is an explanation of the present invention and should not be regarded as a limitation of the present invention.

[0132] Step 1: Simulate a low light intensity environment and use a MER-232-48GM-PNIR camera to capture near-infrared images.

[0133] Step 2: Train the spectral super-resolution network using the RGB and hyperspectral paired datasets provided in the NTIRE 2020 Spectral Super Resolution task. Input the RGB image obtained under low light intensity into the network (…). Figure 2 The hyperspectral image was reconstructed from the image shown.

[0134] Step 3: The hyperspectral image obtained in step 2 has 31 channels. Through visual observation, three bands with rich texture information are selected and recombined into a new RGB image. Figure 3 (As shown).

[0135] Step 4: Train the spectral extension generative adversarial network using RGB and near-infrared images under low-light conditions. After training, input the low-light RGB image into the test.py file and run the inference module of the generative adversarial network. Figure 4 The synthesized near-infrared image was obtained as shown.

[0136] Step 5: To evaluate the qualitative results of the synthesized near-infrared image, the Normalized Difference Vegetation Index (NDVI) was used to reflect the difference between the synthesized near-infrared image and the real near-infrared image. Figure 7 As shown, the first column is the visible light image (RGB), the second column is the real near-infrared image (NIR), the third column is the synthesized near-infrared image, and the fourth column represents the original NDVI image. The fifth column represents the composite NDVI image. The last column represents the grayscale histograms of the NDVI images in columns four and five (where orange represents real NDVI and green represents synthetic NDVI). The results show that the synthetic near-infrared image can fit the distribution of the real near-infrared image very well.

[0137] Step 6: Over 1000 RGB images were labeled using ImgaeLabel software, generating XML annotation files. Furthermore, the near-infrared images obtained in Step 4 were stitched together with the new RGB images obtained in Step 3 along the channel dimension to obtain a 6-channel MAT file. The resulting 6-channel MAT file was then input into the early fusion object detection network (…). Figure 6 As shown, feature extraction is performed using the VGG16 feature extraction module. Candidate frames are generated using a candidate frame proposal network and projected onto the feature map to obtain the corresponding feature matrix. Each feature matrix is ​​scaled using a ROI pooling layer, and then the feature map is diffused through a series of fully connected layers to obtain the multi-source image object detection prediction result. Another set of control experiments directly uses a 3-channel RGB image as input to the object detection network for feature extraction. Candidate frames are generated using a candidate frame proposal network and projected onto the feature map to obtain the corresponding feature matrix. Each feature matrix is ​​scaled using a ROI pooling layer, and then the feature map is diffused through a series of fully connected layers to obtain the single-source image object detection prediction result. Figure 8 (As shown).

[0138] The final results show that, under low light conditions, the method using multi-source images can detect more sweet pepper crops compared to object detection using single-source RGB images.

[0139] In summary, to address the problems of poor crop target detection performance using only single-source data and high cost of multi-source data under low-light conditions, this invention provides a crop target detection method based on spectral super-resolution reconstruction and spectral extension under low-light conditions. This method uses a data-driven approach to train the network with a large-scale paired dataset, resulting in a spectral super-resolution network and a spectral extension generative adversarial network. These networks can effectively fit the mapping relationship between RGB images and hyperspectral and near-infrared images. Hyperspectral and near-infrared images can be directly reconstructed from RGB images, avoiding the problems of resolution inconsistencies, high sensor costs, and registration difficulties encountered in multi-source data acquisition. In the near-infrared image reconstruction based on the spectral extension generative adversarial network, the good time-frequency and reversible properties of discrete wavelet transform are utilized. Compared with other near-infrared image reconstruction methods, this method can preserve feature information as much as possible, and the reconstructed near-infrared image is free of artifacts. Three information-rich bands are selected from the reconstructed hyperspectral image and recombined to form an RGB image. This RGB image has richer texture features than the original image, providing more complete information features for subsequent target detection. This invention enables better identification of crops under low-light conditions.

[0140] On the other hand, embodiments of the present invention provide a crop target detection system based on a spectral extension method, comprising: a first module for acquiring a target RGB image of a crop; a second module for obtaining a target hyperspectral image by performing spectral super-resolution using a spectral super-resolution network based on a data-driven approach based on the target RGB image; a third module for synthesizing a target near-infrared image by performing spectral extension using a spectral extension generative adversarial network based on a data-driven approach based on the target RGB image; a fourth module for synthesizing a target visible light image based on the target hyperspectral image according to preset conditions; and a fifth module for stitching the target visible light image and the target near-infrared image in the channel dimension and inputting them into an early fusion target detection network to obtain the crop target detection result.

[0141] The content of the method embodiments of the present invention is applicable to the system embodiments. The specific functions implemented in the system embodiments are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above methods.

[0142] Another aspect of this invention provides a crop target detection system based on a spectral extension method, including a processor and a memory;

[0143] The memory is used to store programs;

[0144] The processor executes the program to implement the method described above.

[0145] The content of the method embodiments of the present invention is applicable to the system embodiments. The specific functions implemented in the system embodiments are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above methods.

[0146] Another aspect of this invention provides a computer-readable storage medium storing a program that is executed by a processor to implement the methods described above.

[0147] The content of the method embodiments of the present invention is applicable to the computer-readable storage medium embodiments. The specific functions implemented by the computer-readable storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above methods.

[0148] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method.

[0149] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and sub-operations described as part of a larger operation are executed independently.

[0150] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0151] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0152] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution means, apparatus, or device (such as a computer-based device, a processor-including device, or other means that can fetch and execute instructions from, or in conjunction with, an instruction execution means, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution means, apparatus, or device.

[0153] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0154] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution device. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0155] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0156] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0157] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of the present invention.

Claims

1. A method for detecting crop targets based on spectral extension, characterized in that, include: Obtain the target RGB image of the crop; Based on the target RGB image, a target hyperspectral image is obtained by performing spectral super-resolution using a data-driven method through a spectral super-resolution network. Based on the target RGB image, a near-infrared image of the target is synthesized by a spectral expansion generative adversarial network using a data-driven approach. Based on preset conditions, a visible light image of the target is synthesized from the target hyperspectral image; The visible light image and the near-infrared image of the target are stitched together in the channel dimension, and then input into the early fusion target detection network to obtain the crop target detection result; The spectral-expanded generative adversarial network includes a generator, which comprises an encoder, a bottleneck layer, and a decoder. The step of synthesizing a target near-infrared image based on the target RGB image using the spectral-expanded generative adversarial network in a data-driven manner includes: The first feature map is obtained by performing first convolution processing, residual channel attention instantiation processing, and discrete wavelet transform downsampling processing on the target RGB image through the encoder. The encoder consists of three repetitive and similar modules. Each module comprises a convolutional layer, two residual channel attention instantiation modules, and a discrete wavelet transform downsampling module. In each residual channel attention instantiation module, the input feature map is passed through two convolutional layers with a kernel size of 3×3 and a stride of 1 to obtain a residual image. The residual image is then fed into the channel attention module, where two fully connected layers and a ReLU activation function are used to adaptively model the correlation between channels. The correlation is then combined with normalized weights to perform an inner product to obtain an attention feature map. The input feature map and the attention feature map are then added element-wise to obtain an intermediate result feature map. The second feature map is obtained by performing a second convolution on the first feature map through a bottleneck layer; The second feature map is processed by inverse discrete wavelet transform upsampling, element-by-element addition, and third convolution to synthesize a target near-infrared image. The decoder also has three similar modules. Each module uses inverse discrete wavelet transform for upsampling to obtain an upsampled feature map. The upsampled feature map is then added element-wise to the intermediate result feature map obtained from the corresponding layer of the encoder. After that, it goes through three convolutional layers with a kernel size of 3×3, a stride of 1, and an activation function of ReLU.

2. The crop target detection method based on spectral extension method according to claim 1, characterized in that, Also includes: Based on the inverse mapping between RGB images and hyperspectral images, a multi-layer cascaded spectral super-resolution network is constructed through a feature extraction module, an inter-layer feature integration module, and a feature enhancement module. The spectral super-resolution network uses images of different resolutions as inputs for each level. The inter-layer feature integration module performs upsampling operations through PixelShuffle. The feature enhancement module includes residual blocks and dense blocks.

3. The crop target detection method based on spectral extension method according to claim 2, characterized in that, Also includes: The spectral super-resolution network is optimized and trained using pairs of RGB and hyperspectral images as datasets and in conjunction with a loss function.

4. The crop target detection method based on spectral extension method according to claim 1, characterized in that, The spectral super-resolution network is a multi-layer cascaded architecture. The step of obtaining a target hyperspectral image by performing spectral super-resolution using the spectral super-resolution network based on a data-driven approach, according to the target RGB image, includes: The target RGB image is downsampled by multiple factors using PixelUnShuffle to obtain images of various resolutions; The original resolution image of the target RGB image and the various resolution images of the multiple resolution images are sequentially input into each layer of the spectral super-resolution network; Feature extraction, inter-layer feature integration, and feature enhancement are performed at each level to obtain the target hyperspectral image corresponding to each level; The inter-layer feature integration is performed by upsampling through PixelShuffle; the feature enhancement reduces artifacts through residual blocks and dense blocks.

5. The crop target detection method based on spectral extension method according to claim 1, characterized in that, Also includes: Based on the mapping relationship between RGB images and near-infrared images, an end-to-end spectral extension generative adversarial network is constructed through a generator and a discriminator. The generator includes an encoder, a bottleneck layer, and a decoder. The encoder includes a convolutional layer, a residual channel attention instantiation module, and discrete wavelet transform downsampling. The decoder includes inverse discrete wavelet transform upsampling and a convolutional layer.

6. The crop target detection method based on spectral extension method according to claim 5, characterized in that, Also includes: The parameters of the spectral extension generative adversarial network are optimized using least-squares adversarial loss. The generator is constrained at the pixel level by feature matching loss and perceptual loss; The discriminator and the generator are trained alternately using pairs of RGB and near-infrared images as datasets until Nash equilibrium is reached.

7. The crop target detection method based on spectral extension method according to claim 1, characterized in that, Also includes: Based on the Faster R-CNN network architecture, an early fusion target detection network is constructed through a feature extraction module, a region frame prediction module, and a classification prediction module. The feature extraction module includes a VGG16 network with a 6-channel input layer.

8. The crop target detection method based on spectral extension method according to claim 1, characterized in that, The early fusion target detection network includes a feature extraction module, a region frame prediction module, and a classification prediction module. The step of stitching the target visible light image and the target near-infrared image together along the channel dimension and inputting them into the early fusion target detection network to obtain crop target detection results includes: The fifth feature map is obtained by receiving and processing the stitched visible light image and near-infrared image of the target through the feature extraction module. Candidate frames are generated by the region frame prediction module and projected onto the fifth feature map to obtain the corresponding feature matrix; The crop target detection results are obtained by scaling and diffusion processing based on the feature matrix through the classification and prediction module.

9. A crop target detection system based on a spectral extension method, characterized in that, The crop target detection method based on spectral extension method described in claim 1 includes: The first module acquires the target RGB image of the crop; The second module is used to obtain a target hyperspectral image by performing spectral super-resolution based on the target RGB image through a spectral super-resolution network in a data-driven manner. The third module is used to synthesize a target near-infrared image based on the target RGB image by generating an adversarial network through spectral expansion using a data-driven approach. The fourth module is used to synthesize a visible light image of the target based on the target hyperspectral image under preset conditions; The fifth module is used to stitch together the visible light image and the near-infrared image of the target in the channel dimension, and input them into the early fusion target detection network to obtain the crop target detection result.

Images