Agricultural product variety identification method, system and application based on image and spectrum fusion

Abstract

The present invention discloses a method, system and application for identifying agricultural product varieties based on image and spectrum fusion. The training method comprises: obtaining RGB images and hyperspectral data; obtaining fused images; training a deep learning model; the deep learning model is obtained by adjusting the basic YOLOv5 model, omitting the second upsampling module and the corresponding transfer connection, omitting the corresponding detection head module in the head network, and replacing the first-level C3 module in the backbone network with a convolutional block attention module. The present invention adopts a data fusion strategy to feature fuse hyperspectral information with RGB image information, and based on the improved YOLOv5 deep learning method, establishes a soybean variety identification model based on feature fusion data, and adopts a fused classification model to predict the category of target agricultural product seeds, thereby solving the instability problem of a single feature and achieving accurate and efficient identification of agricultural product varieties.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a method, system and application for identifying agricultural product varieties based on image and spectrum fusion. Background Art

[0002] In modern agriculture, planting grain varieties suitable for the corresponding environment in different regions is the key to ensure high yields. Therefore, it is of great significance to develop an efficient method for detecting grain varieties.

[0003] Hyperspectral imaging technology can capture rich spectral and image information. With the rapid development of deep learning technology, it has been widely used in agricultural production in recent years. However, the seeds of many agricultural products are small, and the appearance and color of seeds of different varieties are not much different. A single spectral or image feature is often difficult to distinguish, or it is easy to misjudge the variety, which has an adverse impact on agricultural planting. Summary of the invention

[0004] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a method, system and application for agricultural product variety identification based on image and spectrum fusion, so as to achieve accurate and efficient identification of agricultural product varieties based on an improved YOLOv5 network structure.

[0005] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention includes:

[0006] In a first aspect, the present invention provides a training method for an agricultural product variety identification model based on image and spectrum fusion, comprising:

[0007] Obtain RGB images and hyperspectral data of agricultural product seeds;

[0008] Using linear interpolation to perform feature fusion on the RGB image and the hyperspectral data to obtain a fused image;

[0009] At least using the fused image to train a deep learning model to obtain an agricultural product variety identification model;

[0010] Wherein, the deep learning model is obtained by adjusting the basic YOLOv5 model;

[0011] The basic YOLOv5 model includes a backbone network, a neck network and a head network; the backbone network includes an initial convolution module, a first convolution module, a first C3 module, a second convolution module, a second C3 module, a third convolution module, a third C3 module, a fourth convolution module, a fourth C3 module, and a spatial pyramid pooling module connected in series in sequence; the spatial pyramid pooling module generates a pooling feature based on the fourth C3 feature output by the fourth C3 module; in the neck network, the pooling feature is input into the fifth convolution module to generate a fifth convolution feature, the fifth convolution feature is input into the first upsampling module to generate a first upsampling feature, the first upsampling feature is input into the first fusion module and fused with the third C3 feature output by the third C3 module to form a first fusion feature, the first fusion feature is input into the fifth C3 module to generate a fifth C3 feature, the fifth C3 feature is input into the sixth convolution module to generate a sixth convolution feature, the sixth convolution feature is input into the second upsampling module to generate a second upsampling feature, The second up-sampled feature is input into the second fusion module, and is fused with the second C3 feature output by the second C3 module to form a second fused feature. The second fused feature is input into the sixth C3 module to generate a sixth C3 feature. The sixth C3 feature is input into the first detection head module of the head network for feature detection and input into the seventh convolution module to generate a seventh convolution feature. The seventh convolution feature is input into the third fusion module and fused with the sixth convolution feature to generate a third fused feature. The third fused feature is input into the seventh C3 module to generate a seventh C3 feature. The seventh C3 feature is input into the second detection head module of the head network for feature detection and input into the eighth convolution module to generate an eighth convolution feature. The eighth convolution feature is input into the fourth fusion module and fused with the fifth convolution feature to generate a fourth fused feature. The fourth fused feature is input into the eighth C3 module to generate an eighth C3 feature. After passing through the ninth convolution module, the eighth C3 feature is input into the third detection head of the head network for feature detection.

[0012] Compared with the basic YOLOv5 model, the deep learning model omits the second upsampling module, the second fusion module, the sixth C3 module, the first detection head module, the seventh convolution module, the third fusion module, the seventh C3 module and the corresponding transfer connections, the input of the second detection head module is changed to the sixth convolution feature, and the input of the eighth convolution module is changed to the fifth C3 feature; and the first C3 module in the backbone network is replaced by a convolution block attention module, and the convolution block attention module includes a channel attention submodule and a spatial attention submodule.

[0013] In a second aspect, the present invention further provides a method for identifying agricultural product varieties based on image and spectrum fusion, which comprises:

[0014] Providing an agricultural product variety identification model obtained by the above training method;

[0015] Obtain target RGB images and target hyperspectral data corresponding to target agricultural product seeds;

[0016] Using linear interpolation to perform feature fusion on the target RGB image and target hyperspectral data to obtain a target fused image;

[0017] The target fusion image is input into the agricultural product variety identification model to obtain a classification result.

[0018] In a third aspect, the present invention further provides an agricultural product variety identification system based on image and spectrum fusion, which comprises:

[0019] An image acquisition module is used to acquire target RGB images and target hyperspectral data corresponding to target agricultural product seeds;

[0020] A feature fusion module is used to perform feature fusion on the target RGB image and the target hyperspectral data by using linear interpolation to obtain a target fused image;

[0021] A variety identification module, used for inputting the target fusion image into an agricultural product variety identification model to obtain a classification result;

[0022] Among them, the agricultural product variety identification model is trained by the above-mentioned training method.

[0023] Based on the above technical solution, compared with the prior art, the beneficial effects of the present invention at least include:

[0024] The technical solution provided by the present invention obtains hyperspectral data and RGB images of different agricultural product varieties through hyperspectral equipment and high-resolution cameras, and adopts data fusion strategy to fuse the hyperspectral information with the RGB image information. Based on the improved YOLOv5 deep learning method, a soybean variety identification model based on feature fusion data is established, and the fused classification model is used to predict the category of target agricultural product seeds, which solves the instability problem of single features and realizes accurate and efficient identification of agricultural product varieties.

[0025] The above description is only an overview of the technical solution of the present invention. In order to enable those skilled in the art to more clearly understand the technical means of the present application and implement them according to the contents of the specification, the following is a description of the preferred embodiments of the present invention in conjunction with detailed drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a flow chart of a method for identifying agricultural product varieties provided by a typical implementation case of the present invention;

[0027] Figure 2It is a schematic diagram of the strategy framework of a method for identifying agricultural product varieties provided by a typical implementation case of the present invention;

[0028] Figure 3 It is a schematic diagram of the RGB image preprocessing process provided by a typical implementation case of the present invention;

[0029] Figure 4 This is an example of a spectrum diagram after hyperspectral data preprocessing provided by a typical implementation case of the present invention;

[0030] Figure 5 It is a schematic diagram of the network structure of a deep learning model improved based on the basic YOLOv5 model provided in a typical implementation case of the present invention;

[0031] Figure 6a This is a classification effect test diagram of a method for identifying agricultural product varieties provided by a typical implementation case of the present invention;

[0032] Figure 6b It is a comparison chart of classification effects of a method for identifying agricultural product varieties provided by a typical implementation case of the present invention and a control case. DETAILED DESCRIPTION

[0033] In view of the deficiencies in the prior art, the inventor of this case has proposed the technical solution of the present invention after long-term research and extensive practice. The technical solution, its implementation process and principle will be further explained as follows.

[0034] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0035] Moreover, relational terms such as “first” and “second” are merely used to distinguish one component or method step from another with the same name, but do not necessarily require or imply any such actual relationship or order between these components or method steps.

[0036] The purpose of the present invention is to provide an innovative technical means to accurately identify and analyze agricultural product varieties by organically integrating hyperspectral technology with RGB image processing technology, aiming to make up for the defects and shortcomings of existing technologies in agricultural product variety detection.

[0037] See also Figure 1 , Figure 2 as well as Figure 5 As shown, the embodiment of the present invention provides a training method for an agricultural product variety identification model based on image and spectrum fusion, which includes the following steps:

[0038] Obtain RGB images and hyperspectral data of agricultural product seeds;

[0039] Using linear interpolation to perform feature fusion on the RGB image and the hyperspectral data to obtain a fused image;

[0040] At least the fused image is used to train the deep learning model to obtain an agricultural product variety identification model.

[0041] Wherein, the deep learning model is obtained by adjusting the basic YOLOv5 model;

[0042] The basic YOLOv5 model includes a backbone network, a neck network and a head network; the backbone network includes an initial convolution module, a first convolution module, a first C3 module, a second convolution module, a second C3 module, a third convolution module, a third C3 module, a fourth convolution module, a fourth C3 module, and a spatial pyramid pooling module connected in series in sequence; the spatial pyramid pooling module generates a pooling feature based on the fourth C3 feature output by the fourth C3 module; in the neck network, the pooling feature is input into the fifth convolution module to generate a fifth convolution feature, the fifth convolution feature is input into the first upsampling module to generate a first upsampling feature, the first upsampling feature is input into the first fusion module and fused with the third C3 feature output by the third C3 module to form a first fusion feature, the first fusion feature is input into the fifth C3 module to generate a fifth C3 feature, the fifth C3 feature is input into the sixth convolution module to generate a sixth convolution feature, the sixth convolution feature is input into the second upsampling module to generate a second upsampling feature, The second up-sampled feature is input into the second fusion module, and is fused with the second C3 feature output by the second C3 module to form a second fused feature. The second fused feature is input into the sixth C3 module to generate a sixth C3 feature. The sixth C3 feature is input into the first detection head module of the head network for feature detection and input into the seventh convolution module to generate a seventh convolution feature. The seventh convolution feature is input into the third fusion module and fused with the sixth convolution feature to generate a third fused feature. The third fused feature is input into the seventh C3 module to generate a seventh C3 feature. The seventh C3 feature is input into the second detection head module of the head network for feature detection and input into the eighth convolution module to generate an eighth convolution feature. The eighth convolution feature is input into the fourth fusion module and fused with the fifth convolution feature to generate a fourth fused feature. The fourth fused feature is input into the eighth C3 module to generate an eighth C3 feature. After passing through the ninth convolution module, the eighth C3 feature is input into the third detection head of the head network for feature detection.

[0043] Compared with the basic YOLOv5 model, the deep learning model omits the second upsampling module, the second fusion module, the sixth C3 module, the first detection head module, the seventh convolution module, the third fusion module, the seventh C3 module and the corresponding transfer connections, the input of the second detection head module is changed to the sixth convolution feature, and the input of the eighth convolution module is changed to the fifth C3 feature; and the first C3 module in the backbone network is replaced by a convolution block attention module, and the convolution block attention module includes a channel attention submodule and a spatial attention submodule.

[0044] In the present invention, the main technical idea is to realize the variety identification of agricultural product seeds by fusing RGB images and hyperspectral data. Based on the above overall idea, the initial attempt is to realize identification through the mature YOLOv5 model. However, in practice, it is found that the traditional YOLOv5 model (that is, the basic YOLOv5 model mentioned above) has a significant improvement in speed and accuracy (compared with YOLOv4), but for the specific case of agricultural product seeds, the seed image in this case has the characteristics of a single task, high similarity of image features of seeds of different varieties, and higher data dimension of the fused image. The performance of the traditional YOLOv5 model is not excellent, which is specifically reflected in the excessive training overhead, low model operation efficiency and unsatisfactory detection accuracy.

[0045] Therefore, in the process of forming the present invention, the inventors of the present invention improved the basic YOLOv5 model, combined with the characteristics of the fused image, and used the CBAM module (i.e. the above-mentioned convolutional block attention module) to specifically replace the most basic C3 module (i.e. the first C3 module) in the basic YOLOv5 model, and deliberately simplified the double-layer upsampling neck architecture in the basic YOLOv5 model to a single-layer upsampling neck architecture, so as to simultaneously improve the detection accuracy and model operation efficiency, while also saving training overhead.

[0046] It is worth noting that the original purpose of simplifying the neck structure in the present invention is to simplify the model to improve efficiency, but in actual testing, it is found that this improvement can not only improve efficiency, but also significantly improve the detection accuracy of the fused image; this should be due to the characteristics of the above-mentioned fused image, which leads to the multi-scale feature fusion performed by the double-layer upsampling neck frame in the basic YOLOv5 model. Too many variables are introduced, which is not conducive to the detection accuracy in this single scenario.

[0047] In addition, regarding the replacement of the C3 module with the CBAM module, although the original C3 module can improve the performance of the lightweight semantic segmentation network while maintaining the largest possible receptive field, its perception ability for the fused image is insufficient; and the CBAM module has the characteristics of a hybrid model, containing both a channel attention mechanism and a spatial attention mechanism, and can comprehensively perceive the feature information of the fused image from multiple dimensions, which is relatively important for improving detection accuracy. In the present invention, only the original C3 module is replaced to maintain the model efficiency and lightweight while significantly improving the detection accuracy, without changing other C3 modules.

[0048] The above technical solution can take soybean as an example to carry out model training for soybean variety detection, which specifically includes the following main steps:

[0049] Obtain spectral information and RGB image information of different varieties of soybeans;

[0050] The spectral data of soybean were preprocessed by different methods;

[0051] The data fusion strategy is adopted to fuse the preprocessed soybean spectral data with its corresponding RGB image information through linear interpolation to form a fused image.

[0052] Based on the deep learning method, a soybean classification model based on the feature fusion data is established.

[0053] Of course, the specific agricultural products are not limited to the listed soybeans. Other agricultural products similar to soybeans, such as wheat, rice, etc., which have smaller seeds and are difficult to accurately distinguish the specific varieties with the naked eye (or based only on RGB images) are within the scope of the target objects that can be identified by the present invention.

[0054] The improvements of some better implementation schemes of the present invention are not only in the improvement of the model structure, but also include improvements to the CBAM module itself, that is, in some implementation schemes, the scaling ratio of the fully connected layer in the channel attention submodule is 1 / 64-1 / 24, and 1 / 32 can be used in typical cases, and the convolution kernel size of the convolution layer in the spatial attention submodule is 8-10, and a convolution kernel size of 9 can be used in typical implementation cases.

[0055] The above preferred implementation scheme significantly reduces the number of parameters and expands the receptive field by adjusting the fully connected layer in the original channel attention module to a smaller scaling ratio, for example, adjusting the original 1 / 16 to 1 / 32, and increasing the convolution kernel size of the convolution layer in the original spatial attention module, for example, increasing the convolution kernel size of the convolution layer in the original spatial attention module from 7 to 9, thereby further optimizing the model efficiency and detection accuracy based on the above technical solutions.

[0056] As for the process of acquiring the fused image, in some implementation schemes, the hyperspectral data is preprocessed and then linearly interpolated with the RGB image; the preprocessing includes any one of moving average, filter, multiple scattering correction, and standard normal variable, or a combination of two or more thereof.

[0057] In some embodiments, the preprocessing is selected from a combination of multiple scatter correction and standard normal variance.

[0058] In some embodiments, the linear interpolation process may specifically include:

[0059] Separating the hyperspectral data into equal wavelength intervals to obtain interpolation data;

[0060] Inserting the interpolation data into each row of pixels in the RGB image to obtain the fused image;

[0061] The number of partitions of the interpolation data is equal to or a multiple of the pixel width value in the RGB image.

[0062] In some embodiments, the generation process of the interpolation data can be expressed as:

[0063] ;

[0064] in, represents the interpolation step size; represents the maximum wavelength in the hyperspectral data; represents the minimum wavelength in the hyperspectral data; Indicates wavelength value; represents the number of divisions of the interpolation data; represents the interpolation data; represents the original spectrum value; Represents a natural number; m and n represent the wavelength index of the original band; Representation and The closest wavelength index, where i is greater than n, Representation and Another adjacent wavelength, and Greater than .

[0065] In some embodiments, the generation process of the fused image can be expressed as:

[0066] ;

[0067] in, represents the row index, and h represents the column index; represents the fused image; represents the RGB image; represents the spectral data after the linear interpolation process, that is, the interpolation data. It should be noted that With the above Although they all represent the interpolation data, they are slightly different in different formulas. In the formula for generating the fused image, the subscripts are added HS Indicates the image data label to be fused.

[0068] In some embodiments, the RGB image is obtained by processing the original image, and the image processing includes at least image segmentation so that the RGB image only shows a single seed of the agricultural product.

[0069] Still taking the soybean variety detection described above as an example, the first step of obtaining the RGB image information of soybeans specifically includes: taking pictures of soybeans through an integrating sphere and a high-resolution camera to obtain the RGB image information of soybeans. Preprocessing the spectral data of soybeans includes: preprocessing the spectral data with a filter + standard normal variable. Then, the preprocessed data of the soybean hyperspectral data is interpolated into the length of the image using a linear interpolation method. The preprocessed soybean spectral data is feature fused with its corresponding RGB image information through linear interpolation to form fused data. The specific fusion strategy adopted is to add the spectral information to each pixel of each row in the image to form fused data.

[0070] The improved CBAM attention mechanism is added to the network structure of the deep learning algorithm. The specific operation can be: split the sample data in the feature fusion data into a training set, a validation set, and a test set, and label the sample soybean varieties, and then divide the sample data into a training set and a validation set based on the optimized YOLOv5 model described above, and establish a soybean variety identification model.

[0071] Finally, the prediction accuracy of the established soybean variety identification model can be tested through the test set.

[0072] As a further application of the above technical solution, a second aspect of an embodiment of the present invention further provides a method for identifying agricultural product varieties based on image and spectrum fusion, which comprises the following steps:

[0073] Providing an agricultural product variety identification model obtained by the training method provided in any of the above embodiments;

[0074] Obtain target RGB images and target hyperspectral data corresponding to target agricultural product seeds;

[0075] Using linear interpolation to perform feature fusion on the target RGB image and target hyperspectral data to obtain a target fused image;

[0076] The target fusion image is input into the agricultural product variety identification model to obtain a classification result.

[0077] Correspondingly, the third aspect of the present invention further provides an agricultural product variety identification system based on image and spectrum fusion, which includes:

[0078] An image acquisition module is used to acquire target RGB images and target hyperspectral data corresponding to target agricultural product seeds;

[0079] A feature fusion module is used to perform feature fusion on the target RGB image and the target hyperspectral data by using linear interpolation to obtain a target fused image;

[0080] A variety identification module, used for inputting the target fusion image into an agricultural product variety identification model to obtain a classification result;

[0081] Among them, the agricultural product variety identification model is trained by the training method provided by any of the above-mentioned implementation methods.

[0082] In addition, an embodiment of the present invention further provides a readable storage medium, in which a computer program is stored. When the computer program is executed, the steps of the above-mentioned training method or the above-mentioned agricultural product variety identification method are executed.

[0083] The technical solution of the present invention is further described in detail below through several embodiments and in conjunction with the accompanying drawings. However, the selected embodiments are only used to illustrate the present invention, and do not limit the scope of the present invention.

[0084] Example 1

[0085] This embodiment provides a method for detecting grain varieties based on spectral and RGB image fusion, including:

[0086] S1: Obtain spectral data and RGB images of each type of grain;

[0087] Specifically, hyperspectral information of different varieties of agricultural products is obtained through a hyperspectral camera, and ultra-clear soybean RGB images are taken in an integrating sphere.

[0088] S2: preprocessing the RGB image;

[0089] Specifically, RGB image preprocessing includes color correction, image enhancement and segmentation. Color correction can correct the color deviation in the image, so that the RGB image can more realistically reflect the appearance characteristics of the grain. Image enhancement improves the quality and clarity of the image by adjusting parameters such as image contrast, brightness and sharpness. Segmentation separates each type of agricultural product in the image into a single grain image, which facilitates subsequent feature extraction and recognition. Figure 3 shown.

[0090] S3: preprocess the spectral data, select the best preprocessing method and perform linear interpolation to make the length of the spectral data consistent with the length of the RGB image;

[0091] Specifically, hyperspectral image preprocessing is an important step to improve data quality and analysis accuracy, aiming to remove noise, correct spectral distortion, remove background, perform data compression and dimension reduction, smooth spectral curves, perform radiation and geometric correction, and perform spectral matching and normalization, so as to ensure the reliability and comparability of hyperspectral data and facilitate subsequent analysis and application. In this embodiment, moving average (MA), filter (MSG), multiple scattering correction (SG), and standard normal variable (SNV) are used to process hyperspectral data. MA is a method for smoothing spectral data. Moving average averages the data by applying a sliding window to the spectral data, thereby reducing noise and fluctuations and enhancing the interpretability of the signal; MSC is mainly used to correct spectral shifts and deformations caused by scattering effects. The main goal is to eliminate the influence of sample physical properties (such as particle size, surface roughness, etc.) on the spectrum, thereby improving the accuracy and reliability of the analysis; SG can effectively eliminate high-frequency random noise; SNV preprocessing aims to eliminate systematic deviations in spectral data caused by differences in optical path length or concentration changes, so as to better highlight the chemical differences between samples. In order to better process the spectral data, this embodiment also explores the combination of spectral preprocessing methods, such as MA+MSC, SG+SNV, etc. Since there may be some optical path errors such as high-frequency noise in the spectral data, by comprehensively comparing different combinations, the most suitable SG+SNV processed data is finally selected for linear interpolation. The processed hyperspectral data spectrum is shown in FIG. Figure 4 shown.

[0092] In this embodiment, first, the hyperspectral data Arrange in order of wavelength and calculate the interpolation step size. First, the hyperspectral data needs to be Arrange them in order of wavelength. Next, calculate the interpolation step size. , in and are the maximum and minimum wavelengths respectively. Generate 640 equally spaced interpolation points:

[0093] ;

[0094] Where i=0,1,…,639.

[0095] Then, the new spectral data is calculated by linear interpolation The formula is as follows:

[0096] ;

[0097] Among them, j is The nearest wavelength index. The final result is a spectral data with a dimension of 640.

[0098] S4: Fusion of spectral data with RGB image;

[0099] Specifically, Figure 2 As shown in the process diagram, based on the RGB image, the interpolated spectral information is added to the pixels corresponding to each row of the image to form a fused image. The specific steps are as follows:

[0100] 1. For each row of each RGB image, add the expanded spectral value to the corresponding pixel value.

[0101] 2. Form a fused image of size H × W × 3, where H and W are the height and width of the image, respectively.

[0102] The specific operations of image fusion are as follows:

[0103] 1. Read RGB image and hyperspectral data .

[0104] 2. Yes Interpolate and get .

[0105] 3. and The pixel values ​​corresponding to each row are added to form a fused image .

[0106] The relevant formula of the above process can be expressed as:

[0107] ;

[0108] In this way, the fused image not only retains the spatial information of the RGB image, but also introduces the spectral information of the hyperspectral image, allowing the model to obtain more feature details during training.

[0109] S5: The fused image is passed into the improved YOLOv5 structure and CBAM attention mechanism for training, and the output model is used for prediction;

[0110] Specifically, in order to enhance the model's ability to focus on important features, this embodiment introduces and improves the CBAM attention mechanism, and uses the improved CBAM module (convolutional block attention module) to replace the first-level C3 module in the backbone network. The specific improvements of the CBAM module are as follows:

[0111] The fully connected layer in the original channel attention module is scaled to a smaller scale, which greatly reduces the number of parameters and can be expressed as:

[0112] ;

[0113] The convolution kernel of the convolution layer in the original spatial attention module is enlarged to expand the receptive field and improve the detection performance of the model. The formula can be expressed as:

[0114] ;

[0115] In addition, in this embodiment, if Figure 2 As shown, in order to further optimize the performance of YOLOv5 to adapt to specific detection tasks, its network structure has been carefully improved and the network structure has been simplified in a targeted manner. The traditional YOLOv5 network structure includes two upsampling operations, which is very necessary when processing targets of various sizes or requiring fine feature extraction. However, in this embodiment, due to the large size of the target object and the relatively single detection task, an overly complex network structure may introduce unnecessary computational overhead, and may not significantly improve the detection accuracy, and may even introduce too many variables to ultimately interfere with the detection results. Therefore, in this embodiment, it is decided to specifically reduce the last level of upsampling operations and their corresponding connection layers, convolutional layers, and detection head modules in the head network, etc., to simplify the network structure and improve efficiency. Specifically, one of the upsampling layers is removed, and the parameters and connection methods of the subsequent convolutional layers are adjusted accordingly to ensure that the network can still learn enough feature information for accurate target detection.

[0116] In this embodiment, the fused feature map is sent to the optimized YOLOv5 model for training, and the training results have achieved very good results. When the soybean variety detection test is performed, the average mAP of different varieties reaches 99.1%. Figure 6a and Figure 6b The test results are shown.

[0117] Comparative Example

[0118] This comparative example is substantially the same as Example 1, and mainly performs comparative experiments on the omission of multiple factors. The specific main differences from Example 1 are as follows:

[0119] No feature fusion was performed, and only RGB images were used for training and prediction, denoted as YOLOV5s;

[0120] Perform feature fusion, but do not improve YOLOV5, nor introduce the CBAM module, denoted as YOLOV5s_R;

[0121] Perform feature fusion, but do not improve YOLOV5. Introduce the CBAM module without adjusting parameters, denoted as YOLOV5s_R_NC;

[0122] Perform feature fusion and improve YOLOV5, but do not introduce the CBAM module, denoted as YOLOV5s_R_Y;

[0123] Feature fusion is performed to improve YOLOV5, but the introduced CBAM module is not parameter adjusted, recorded as YOLOV5s_R_Y_NC;

[0124] Example 1, denoted as YOLOV5s_R_Y_C.

[0125] Table 1. Performance comparison results of each experiment before and after improvement

[0126]

[0127] After in-depth analysis of the experimental data, it is not difficult to find that the method based on the fusion of hyperspectral and RGB images has shown significant advantages in object detection tasks. Compared with the traditional YOLOV5s model, the fused model has significantly improved in accuracy, recall and mAP indicators. This result not only verifies the effectiveness of image fusion technology in object detection, but also provides new ideas for further improving detection performance.

[0128] From the detailed data, we can see that the fused YOLOV5s_R model has improved in P / %, R / %, mAP@0.5 / %, mAP@0.5:0.95 / %, and Accuracy / % of the test set. This shows that integrating the information of hyperspectral images into RGB images can effectively enhance the perception ability of the model, enabling it to more accurately identify target objects.

[0129] In addition, further improvements were made based on the YOLOV5s_R model. The performance of the model was further improved by introducing CBAM (Convolutional Block Attention Module) and optimizing the YOLOV5 network structure. Experimental results show that these improvements not only improve accuracy, but also maintain a high recall rate.

[0130] It is worth noting that although the accuracy of the YOLOV5s_R_Y model has slightly decreased, the mAP and test set accuracy indicators have still improved. This shows that the improved network structure may have a certain impact on the accuracy in some cases, but it can still improve the performance of the model as a whole.

[0131] Finally, although the optimized CBAM reduces the complexity of the network structure, the experimental results show that the accuracy of YOLOV5s_R_Y_C has not decreased. On the contrary, the accuracy index on the test set has improved.

[0132] In summary, the method based on hyperspectral and RGB image fusion and subsequent improvement measures have achieved remarkable results in object detection tasks. These results not only provide a new direction for the development of object detection technology, but also provide strong support for applications in related fields.

[0133] Based on the above embodiments and comparative examples, it can be clearly seen that the technical solution provided in the embodiments of the present invention obtains hyperspectral data and RGB images of different agricultural product varieties through hyperspectral equipment and high-resolution cameras, and adopts a data fusion strategy to feature fuse the hyperspectral information with the RGB image information, and based on the improved YOLOv5 deep learning method, establishes a soybean variety identification model based on feature fusion data, and adopts a fused classification model to predict the category of target agricultural product seeds, which solves the instability problem of a single feature and realizes accurate and efficient identification of agricultural product varieties.

[0134] It should also be noted that based on the implementation methods provided in the embodiments of the present invention, the inventors of the present invention also conducted experiments on seeds or fruits of other agricultural products, especially agricultural products with particle sizes approximately equal to or smaller than soybeans. The principles used are the same and excellent results were achieved, which will not be elaborated here.

[0135] It should be understood that the above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with the technology to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made according to the spirit of the present invention should be included in the protection scope of the present invention.

Claims

1. A training method for agricultural product variety identification model based on image and spectrum fusion, characterized in that: include: Obtain RGB images and hyperspectral data of agricultural product seeds; Using linear interpolation to perform feature fusion on the RGB image and the hyperspectral data to obtain a fused image; At least using the fused image to train a deep learning model to obtain an agricultural product variety identification model; Wherein, the deep learning model is obtained by adjusting the basic YOLOv5 model; The basic YOLOv5 model includes a backbone network, a neck network and a head network; the backbone network includes an initial convolution module, a first convolution module, a first C3 module, a second convolution module, a second C3 module, a third convolution module, a third C3 module, a fourth convolution module, a fourth C3 module, and a spatial pyramid pooling module connected in series in sequence; the spatial pyramid pooling module generates a pooling feature based on the fourth C3 feature output by the fourth C3 module; in the neck network, the pooling feature is input into the fifth convolution module to generate a fifth convolution feature, the fifth convolution feature is input into the first upsampling module to generate a first upsampling feature, the first upsampling feature is input into the first fusion module and fused with the third C3 feature output by the third C3 module to form a first fusion feature, the first fusion feature is input into the fifth C3 module to generate a fifth C3 feature, the fifth C3 feature is input into the sixth convolution module to generate a sixth convolution feature, the sixth convolution feature is input into the second upsampling module to generate a second upsampling feature, The second up-sampled feature is input into the second fusion module, and is fused with the second C3 feature output by the second C3 module to form a second fused feature. The second fused feature is input into the sixth C3 module to generate a sixth C3 feature. The sixth C3 feature is input into the first detection head module of the head network for feature detection and input into the seventh convolution module to generate a seventh convolution feature. The seventh convolution feature is input into the third fusion module and fused with the sixth convolution feature to generate a third fused feature. The third fused feature is input into the seventh C3 module to generate a seventh C3 feature. The seventh C3 feature is input into the second detection head module of the head network for feature detection and input into the eighth convolution module to generate an eighth convolution feature. The eighth convolution feature is input into the fourth fusion module and fused with the fifth convolution feature to generate a fourth fused feature. The fourth fused feature is input into the eighth C3 module to generate an eighth C3 feature. After passing through the ninth convolution module, the eighth C3 feature is input into the third detection head of the head network for feature detection. Compared with the basic YOLOv5 model, the deep learning model omits the second upsampling module, the second fusion module, the sixth C3 module, the first detection head module, the seventh convolution module, the third fusion module, the seventh C3 module and the corresponding transfer connections, the input of the second detection head module is changed to the sixth convolution feature, and the input of the eighth convolution module is changed to the fifth C3 feature; and the first C3 module in the backbone network is replaced by a convolution block attention module, and the convolution block attention module includes a channel attention submodule and a spatial attention submodule.

2. The training method according to claim 1, characterized in that: The scaling ratio range of the fully connected layer in the channel attention submodule is set to 1 / 64-1 / 24, and the convolution kernel size range of the convolution layer in the spatial attention submodule is set to 8-10.

3. The training method according to claim 1, characterized in that: Preprocessing the hyperspectral data and then performing linear interpolation with the RGB image; The preprocessing includes any one of moving average, filter, multiple scatter correction, and standard normal variable, or a combination of two or more thereof.

4. The training method according to claim 3, characterized in that: The pre-processing is selected from a combination of multiple scatter correction and standard normal variance.

5. The training method according to claim 1, characterized in that: The linear interpolation process specifically includes: Separating the hyperspectral data into equal wavelength intervals to obtain interpolation data; Inserting the interpolation data into each row of pixels in the RGB image to obtain the fused image; The number of partitions of the interpolation data is equal to or a multiple of the pixel width value in the RGB image.

6. The training method according to claim 5, characterized in that: The generation process of the interpolation data is expressed as: ; in, represents the interpolation step size; represents the maximum wavelength in the hyperspectral data; represents the minimum wavelength in the hyperspectral data; Indicates wavelength value; represents the number of divisions of the interpolation data; represents the interpolation data; represents the original spectrum value; Represents a natural number; m and n represent the wavelength index of the original band; Representation and The closest wavelength index, where i is greater than n, Representation and Another adjacent wavelength, and Greater than .

7. The training method according to claim 5, characterized in that: The generation process of the fused image is expressed as: ; Among them, g represents the row index and h represents the column index; represents the fused image; represents the RGB image; represents the spectral data after the linear interpolation process, that is, the interpolation data.

8. The training method according to claim 1, characterized in that: The RGB image is obtained by processing the original image, and the image processing at least includes image segmentation so that the RGB image only displays a single agricultural product seed.

9. A method for identifying agricultural product varieties based on image and spectrum fusion, characterized in that: include: Providing an agricultural product variety identification model obtained by the training method according to any one of claims 1 to 8; Obtain target RGB images and target hyperspectral data corresponding to target agricultural product seeds; Using linear interpolation to perform feature fusion on the target RGB image and target hyperspectral data to obtain a target fused image; The target fusion image is input into the agricultural product variety identification model to obtain a classification result.

10. An agricultural product variety identification system based on image and spectrum fusion, characterized in that: include: An image acquisition module is used to acquire target RGB images and target hyperspectral data corresponding to target agricultural product seeds; A feature fusion module is used to perform feature fusion on the target RGB image and the target hyperspectral data by using linear interpolation to obtain a target fused image; A variety identification module, used for inputting the target fusion image into an agricultural product variety identification model to obtain a classification result; Wherein, the agricultural product variety identification model is trained by the training method described in any one of claims 1-8.

Images