A multi-modal perception-based seedling posture quality detection method

Abstract

The application relates to a rice seedling posture quality detection method based on multi-modal perception, and relates to a rice seedling posture quality detection method based on multi-modal perception.The application aims to solve the problems of poor adaptability of existing farmland seedling posture quality detection technology in a complex environment and low accuracy of unmanned aerial vehicle farmland seedling posture detection.The process is as follows: farmland RGB images and multispectral images of a seedling growth period are collected;preprocessed farmland RGB images and multispectral images of the seedling growth period are obtained;the preprocessed farmland RGB images and multispectral images of the seedling growth period are input into a deep learning model for training, and a trained deep learning model is obtained;farmland RGB images and multispectral images of a seedling growth period to be detected are classified and predicted based on the trained deep learning model, and it is predicted that the seedling class in the farmland RGB images and multispectral images of the seedling growth period to be detected is qualified seedlings, damaged seedlings or floating seedlings.The application is used in the field of unmanned aerial vehicle farmland seedling posture detection.

Description

Technical Field

[0001] This invention relates to a method for detecting the quality of rice transplanting posture based on multimodal perception. Background Technology

[0002] With the deepening development of precision agriculture and the increasing maturity of drone technology, intelligent monitoring of crop growth using drone remote sensing technology has become an important part of modern agriculture. As a major food crop, the plant phenotype during the rice growth process directly affects the final yield and quality. In rice production, the transplanting stage is crucial, and the quality of the seedling posture (such as floating, damaged, and properly positioned seedlings) directly affects the survival rate, tillering ability, and subsequent growth and development of rice. Therefore, rapid and accurate detection of the posture quality of rice seedlings in farmland is of great significance for guiding agricultural production and improving management efficiency.

[0003] Currently, manual inspection is the main method for detecting the posture and quality of rice seedlings in farmland. However, this method suffers from drawbacks such as low efficiency, high labor intensity, and strong subjectivity, making it difficult to meet the needs of modern agricultural precision management. While traditional ground-based observation methods have improved efficiency to some extent, they are limited by the field of view and operational efficiency, making it difficult to conduct rapid surveys of large areas of farmland. Unmanned aerial vehicle (UAV) remote sensing technology, with its advantages of high mobility, wide coverage, and low cost, provides a new solution for the automated and intelligent detection of rice seedling posture and quality in large areas of farmland. UAVs equipped with high-resolution visible light cameras can acquire RGB images of farmland, providing basic data for seedling target identification. However, relying solely on RGB images for seedling posture and quality detection faces many challenges. First, farmland environments are highly complex, with numerous background interference factors, including soil, water bodies, and weeds. These factors share certain similarities with seedlings in terms of color, texture, and other characteristics, which can easily lead to false positives and false negatives in the target detection process. Second, the appearance differences of seedlings in different postures may not be significant in RGB images. For example, slightly damaged seedlings may be difficult to distinguish from healthy seedlings. In addition, a single RGB image lacks vegetation physiological and biochemical information, making it difficult to accurately determine the health status of seedlings.

[0004] In recent years, multispectral imaging technology has been widely used in agriculture. Multispectral cameras can acquire reflectance information of ground objects in multiple specific spectral bands, providing a richer data source for monitoring crop growth. Seedlings with different postures and health conditions exhibit varying reflectance characteristics in different spectral bands. For example, healthy seedlings have higher reflectance in the near-infrared band, while damaged seedlings show reduced reflectance. Floating seedlings, due to root damage and decreased water absorption capacity, also exhibit altered spectral reflectance characteristics. Fusion of multispectral and RGB information can more comprehensively describe seedling characteristics, improving the accuracy and robustness of posture quality detection. However, how to effectively fuse heterogeneous RGB and multispectral data from multiple sources and fully exploit the complementary information between different modalities remains a key issue that has not yet been fully investigated in the field of rice transplanting quality detection.

[0005] Deep learning technology has made significant progress in image recognition and object detection, providing a new technical means for detecting the posture quality of rice seedlings in farmland. Deep learning-based object detection models, such as the YOLO (You Only Look Once) series, can achieve fast and accurate object recognition in images. However, directly applying general object detection models to the posture quality detection of rice seedlings in farmland still has some shortcomings. First, rice seedlings in farmland scenes are usually small in scale, densely distributed, and easily occluded by other features, placing higher demands on the model's detection accuracy and robustness. Second, the differences in visual features between seedlings with different postures may be subtle, requiring the model to have stronger feature extraction and discrimination capabilities. Therefore, it is necessary to design an efficient and accurate deep learning model and algorithm that combines the characteristics of rice seedling posture quality detection to meet the actual needs of the task. Summary of the Invention

[0006] The purpose of this invention is to solve the problems of poor adaptability of existing rice seedling posture quality detection technology in complex environments and low accuracy of UAV rice seedling posture detection, and to propose a rice transplanting posture quality detection method based on multimodal perception.

[0007] The specific process of a rice transplanting posture quality detection method based on multimodal perception is as follows:

[0008] Step S1: Collect RGB and multispectral images of farmland during the seedling growth period;

[0009] Step S2: Preprocess the RGB and multispectral images of farmland during the seedling growth period to obtain preprocessed RGB and multispectral images of farmland during the seedling growth period.

[0010] Step S3: Construct a deep learning model by inputting the preprocessed RGB and multispectral images of farmland during the seedling growth period into the deep learning model for training, and obtain a trained deep learning model.

[0011] Step S4: Based on the trained deep learning model, classify and predict the RGB and multispectral images of farmland during the seedling growth period to be tested, and predict the seedling category in the RGB and multispectral images of farmland during the seedling growth period to be qualified seedlings, damaged seedlings, or drifting seedlings.

[0012] The beneficial effects of this invention are as follows:

[0013] This invention proposes a method for detecting the attitude quality of rice seedlings in farmland using unmanned aerial vehicles (UAVs) based on multimodal perception. This method integrates RGB and multispectral images acquired by the UAV, fully utilizing the advantages of different modal data. It designs a multi-domain interactive cueing module and a cross-band attention fusion module to enhance seedling feature representation, suppress background interference, and improve detection accuracy. Simultaneously, this invention designs an online hard example mining strategy to guide model training, improving the model's robustness and generalization ability. This invention aims to provide an efficient, accurate, and reliable UAV-based solution for detecting the attitude quality of rice seedlings in farmland, providing technical support for precision agriculture.

[0014] To address the shortcomings of existing technologies for detecting the posture and quality of rice seedlings in complex environments, multimodal information fusion, and intelligent analysis, this invention aims to overcome these technological bottlenecks and construct a precise UAV-based method for detecting the posture and quality of rice seedlings in farmland, based on multimodal perception. This method aims to accurately identify different postures, such as floating seedlings, damaged seedlings, and seedlings in suitable conditions, providing technical support for improving agricultural management and ensuring food security. The core objective of this invention is to solve the following key technical problems:

[0015] This invention addresses the problem of accurate identification of rice seedlings against complex farmland backgrounds. Farmland environments are complex and variable, with uneven lighting conditions and various interfering factors such as soil, water, and weeds. These factors share similarities with rice seedlings in color, texture, and other features, posing a challenge to accurate seedling identification. Especially from a drone's perspective, small-scale seedlings are easily obscured by background noise, making them difficult to distinguish effectively using traditional image processing methods. Therefore, designing a detection method that effectively suppresses background interference and highlights the characteristics of the seedlings is the primary technical problem this invention aims to solve.

[0016] This invention addresses the challenge of effectively fusing multispectral imagery from unmanned aerial vehicles (UAVs) to improve the accuracy of seedling attitude quality detection. A single RGB image provides only limited visible light information, making it difficult to accurately distinguish seedlings in different attitudes. For example, floating and damaged seedlings may appear similar to healthy seedlings in RGB images, hindering effective differentiation. Multispectral images offer rich physiological and biochemical information about vegetation; seedlings in different attitudes exhibit variations in reflectance across different spectral bands, enabling more refined attitude quality detection. However, RGB and multispectral images differ in data format and spatial resolution. Effectively fusing this data from both modalities and fully leveraging their complementary information is one of the key technical problems this invention aims to solve.

[0017] This invention also aims to address the problem of improving the model's ability to recognize seedlings in different postures with fine detail. The differences in appearance features between seedlings in different postures can be subtle; for example, slightly damaged seedlings and healthy seedlings may only differ slightly in color and shape. Traditional image classification or object detection methods may struggle to effectively capture these subtle differences, resulting in low recognition accuracy. Therefore, designing a method that can effectively extract and distinguish the subtle features of seedlings in different postures is the key technical problem this invention needs to solve.

[0018] In summary, this invention aims to provide a method for accurate detection of seedling posture quality in farmland using unmanned aerial vehicles (UAVs) based on multimodal perception. By fusing RGB and multispectral image information, it solves the problem of accurate seedling identification under complex background interference, improves the model's fine-grained discrimination ability for seedlings in different postures, and enhances the model's learning ability for difficult samples. This enables high-precision detection of different postures such as floating seedlings, damaged seedlings, and reasonable seedlings, providing important technical support for refined agricultural management.

[0019] This invention proposes a method for detecting the attitude quality of rice seedlings in farmland using unmanned aerial vehicles (UAVs) based on multimodal perception. By effectively fusing RGB and multispectral image information collected by the UAV platform and combining it with deep learning technology, it achieves accurate and efficient assessment of the attitude quality of rice seedlings. Compared with traditional manual inspection methods and detection methods that rely solely on single-modal data, this invention achieves significant technical advantages in detection accuracy, robustness, and information fusion capabilities, providing strong technical support for precision agricultural production.

[0020] Specifically, the Multi-Domain Interactive Prompt Module (MDIPM) designed in this invention effectively suppresses interference from complex farmland backgrounds and enhances the model's focus on potential seedling areas. This module extracts saliency information from images from multiple perspectives, including frequency domain, color space, and learnable attention mechanisms, guiding the network to focus on seedling targets. This reduces the false detection rate caused by background factors such as soil, water, and weeds, significantly improving the detection rate and localization accuracy of seedling targets. Furthermore, the Cross-Band Attention Fusion Module (CBAF) constructed in this invention achieves effective fusion of RGB and multispectral images, fully utilizing the rich vegetation physiological and biochemical characteristics contained in multispectral information. The CBAF module enables RGB features to focus on specific spectral information related to different seedling postures, enhancing the model's ability to identify abnormal postures such as drifting and damaged seedlings. The online difficult example mining strategy for seedlings designed in this invention dynamically selects samples with high model prediction uncertainty for focused learning during training, effectively avoiding overfitting of the model on a large number of simple samples, thereby improving the model's ability to identify complex and difficult samples.

[0021] In summary, the UAV-based method for detecting the attitude quality of rice seedlings in farmland, based on multimodal perception, proposed in this invention effectively integrates the texture information of RGB images and the spectral information of multispectral images, significantly improving the detection accuracy of seedling targets in complex backgrounds and the ability to identify seedlings in different attitudes. The designed MDIPM and CBAF modules can effectively suppress background interference, highlight seedling features, and improve the effectiveness of feature representation; the designed online hard example mining strategy for seedlings enhances the robustness and generalization ability of the model. This method provides a reliable technical means for the automated and accurate evaluation of the attitude quality of farmland seedlings, has broad application prospects, and can provide agricultural producers with timely and accurate field information, guide agricultural production management, and improve production efficiency and economic benefits. Attached Figure Description

[0022] Figure 1 The diagram shows the structure of the deep learning model of this invention. m1 is a spectral image in the 450nm band, m2 is a spectral image in the 660nm band, m3 is a spectral image in the 750nm band, and m4 is a spectral image in the 840nm band.

[0023] Figure 2 This is a flowchart of the method of the present invention;

[0024] Figure 3 This is a structural diagram of the multi-domain interactive prompt module of the present invention. Detailed Implementation

[0025] Specific implementation method one: Combining Figure 1 , Figure 2This embodiment describes a method for detecting rice transplanting posture quality based on multimodal perception, which specifically includes the following steps:

[0026] This invention proposes a method for detecting the attitude quality of rice seedlings in farmland using unmanned aerial vehicles (UAVs) based on multimodal perception. It aims to overcome the problems of insufficient target recognition accuracy in complex farmland backgrounds, low utilization rate of single-modal information, and limited ability to discern subtle differences in seedling attitudes among different plants. The core of this invention lies in the effective fusion of multimodal information and the enhancement of seedling attitude features, thereby achieving high-precision and robust seedling attitude quality detection. The following are the key points and areas to be protected in this invention:

[0027] 1. Multi-domain Interactive Hint Module. To address the problem of interference from complex farmland backgrounds, this invention innovatively designs a multi-domain interactive hint module. This module extracts image information from multiple perspectives, including frequency domain saliency, color channel features, and learnable weed suppression, generating diverse attention masks or weight maps. Through the interactive fusion of multi-domain information, it effectively guides the network to focus on potential seedling areas, suppressing background noise interference, thereby improving the model's accuracy and recall in seedling target detection under complex backgrounds.

[0028] 2. Cross-band Attention Fusion Module. To fully leverage the complementary information between RGB and multispectral images and enhance the ability to distinguish seedlings in different postures, this invention proposes a cross-band attention fusion module. This module utilizes specific bands (such as near-infrared and red light) of multispectral images to extract features closely related to different seedling states (such as drifting and damaged seedlings), and effectively fuses these features with RGB features through a cross-attention mechanism. This module enables the network to pay attention to subtle differences in the spectral dimensions of seedlings in different postures, significantly enhancing the model's accuracy in identifying abnormal seedlings such as drifting and damaged seedlings.

[0029] 3. A Seedling Posture Quality Detection Framework Integrating a Multi-Domain Interactive Cueing Module and a Cross-Band Attention Fusion Module. This invention effectively integrates the proposed multi-domain interactive cueing module and cross-band attention fusion module into a target detection framework, constructing an end-to-end multimodal seedling posture quality detection model. The multi-domain interactive cueing module suppresses background interference and focuses on potential seedling regions in the early stages of feature extraction; the cross-band attention fusion module fuses multispectral information in the intermediate feature layer, enhancing the ability to distinguish seedlings with different postures. This staged, progressive multimodal information fusion strategy fully leverages the advantages of different modal data, significantly improving overall detection performance.

[0030] 4. An Adaptive Model Training Method Based on Online Hard Example Mining Strategy. To improve the robustness and generalization ability of the model, especially its performance when facing complex scenarios and difficult examples, this invention proposes an adaptive model training method based on an online hard example mining strategy. This method dynamically evaluates the difficulty of samples during training and assigns higher weights to difficult examples, guiding the model to focus more on samples that are difficult to identify, thereby improving the model's learning efficiency and final detection performance.

[0031] In summary, the key points of this invention lie in its innovative proposal of a multi-domain interactive prompting module and a cross-band attention fusion module for detecting the posture quality of rice seedlings in farmland, effectively integrating them into a deep learning object detection framework. Simultaneously, an online hard example mining strategy is designed to guide model training, further improving model performance. The combination of these key technologies enables this invention to effectively solve the challenge of detecting the posture quality of rice seedlings in complex farmland environments.

[0032] Step S1: Collect RGB and multispectral images of farmland during the seedling growth period;

[0033] Step S2: Preprocess the RGB and multispectral images of farmland during the seedling growth period to obtain preprocessed RGB and multispectral images of farmland during the seedling growth period.

[0034] Step S3: Construct a deep learning model by inputting the preprocessed RGB and multispectral images of farmland during the seedling growth period into the deep learning model for training, and obtain a trained deep learning model.

[0035] Step S4: Based on the trained deep learning model, classify and predict the RGB and multispectral images of farmland during the seedling growth period to be tested, and predict the seedling category in the RGB and multispectral images of farmland during the seedling growth period to be qualified seedlings, damaged seedlings, or drifting seedlings.

[0036] Specific Implementation Method Two: This implementation method differs from Specific Implementation Method One in that: in step S1, RGB and multispectral images of farmland during the seedling growth period are collected; the specific process is as follows:

[0037] UAVs with high-precision positioning and attitude control are selected to ensure stability during flight and accuracy of data collection.

[0038] Integrate visible light (RGB) cameras and multispectral cameras into the drone platform;

[0039] Multispectral cameras need to be able to cover key wavelengths for seedling growth (covering the 450nm, 660nm, 750nm, and 840nm wavelengths);

[0040] The key wavelengths for seedling growth are 450nm, 660nm, 750nm, and 840nm.

[0041] Set the drone's flight altitude and camera parameters;

[0042] Simultaneous acquisition of RGB and multispectral images of rice seedlings during their growth period

[0043] Based on the scope and terrain conditions of the farmland to be monitored, a reasonable flight route is planned to ensure that the image coverage is not less than 90% and the lateral overlap rate is not less than 75%.

[0044] Set the drone's flight altitude and camera parameters to obtain image data that meets the resolution requirements (at least 2cm / pixel) to ensure that individual seedlings can be effectively identified.

[0045] Simultaneously acquire RGB and multispectral images of key stages of seedling growth (tillering stage or young panicle differentiation stage), at which time the posture characteristics of seedlings are more obvious;

[0046] Ensuring the temporal alignment of data from different modalities provides a foundation for subsequent multimodal information fusion.

[0047] The other steps and parameters are the same as in Specific Implementation Method 1.

[0048] Specific Implementation Method Three: This implementation method differs from Specific Implementation Method One or Two in that: in step S2, the RGB and multispectral images of farmland during the seedling stage are preprocessed to obtain preprocessed RGB and multispectral images of farmland during the seedling stage; the specific process is as follows:

[0049] S21. Image geometric correction and atmospheric correction; the specific process is as follows:

[0050] (1) Perform geometric correction on the acquired RGB and multispectral images to eliminate geometric deformation caused by sensor pose changes and lens distortion, and restore the true geometric shape of the images;

[0051] Geometric correction is achieved using spatial resection methods based on camera calibration parameters and UAV attitude data.

[0052] (2) Atmospheric correction is performed on the geometrically corrected RGB and multispectral images to eliminate the influence of atmospheric absorption, scattering, and other factors on the spectral information of ground features, and to obtain true surface reflectance data; expressed as:

[0053]

[0054] in,

[0055] L λ The spectral radiance of the RGB image is expressed in W / (m²).2 ·sr·nm), W is watt (unit of power), m 2 sr is square meters (unit of area), sr is steradian (unit of solid angle), nm is nanometer (unit of wavelength), and · is a multiplication sign;

[0056] d represents the Earth-Sun distance, measured in astronomical units (AU). AU is the standard unit for measuring the average Earth-Sun distance in astronomy.

[0057] E sunλ The average extraterrestrial irradiance of the sun in a given wavelength band, expressed in W / (m²). 2 ·nm);

[0058] θ z The solar zenith angle;

[0059] ρ λ Surface reflectance;

[0060] S22. Perform band registration on the atmospherically corrected multispectral image to obtain the band-registered multispectral image; the specific process is as follows:

[0061] Register different bands of the atmospherically corrected multispectral image to ensure that the same ground feature corresponds to the same spatial location in images of different bands, thus providing a basis for subsequent band information fusion.

[0062] The registration method uses manual selection of control points;

[0063] S23. Label the atmospherically corrected RGB image and the band-registered multispectral image; the specific process is as follows:

[0064] The atmospherically corrected RGB images and the band-registered multispectral images were labeled as qualified seedlings, damaged seedlings, or drifting seedlings.

[0065] Use the interactive labeling tool X-AnyLabeling to label the quality status information (damaged seedlings, floating seedlings, qualified seedlings) of each rice seedling in the field.

[0066] S24. Image enhancement and denoising; the specific process is as follows:

[0067] (1) To address the potential issues of low contrast and blur in farmland images, the CLAHE (ContrastLimitedAdaptive Histogram Equalization) image enhancement algorithm is used to enhance the labeled RGB and multispectral images, resulting in enhanced images that improve the visibility of seedling targets.

[0068] (2) Denoise the enhanced image to obtain the denoised image; the specific process is as follows:

[0069] The enhanced image is convolved with a Gaussian kernel to obtain the denoised image.

[0070] A Gaussian filtering denoising algorithm is used to remove noise from the image and improve the robustness of feature extraction. The denoising algorithm used is expressed by the following formula:

[0071]

[0072] Where σ represents the standard deviation of the Gaussian kernel, (x,y) represents the coordinate position of a pixel in the image, and G(x,y) represents the Gaussian kernel;

[0073] The denoised image I'(x,y) is the convolution of the original image I(x,y) and the Gaussian kernel G(x,y):

[0074]

[0075] Where k represents the radius of the Gaussian kernel, (i,j) represents the relative coordinate index inside the Gaussian kernel, and G(i,j) represents the Gaussian kernel function.

[0076] Other steps and parameters are the same as in specific implementation method one or two.

[0077] Specific Implementation Method Four: This implementation method differs from Specific Implementation Methods One to Three in that: a deep learning model is constructed in step S3; the specific process is as follows:

[0078] Deep learning models consist of a backbone, a neck, and a prediction head.

[0079] The backbone consists of, in sequence, a first convolutional layer, a second convolutional layer, a first C3k2 module, a third convolutional layer, a second C3k2 module, a fourth convolutional layer, a third C3k2 module, a fifth convolutional layer, a fourth C3k2 module, an SPPF module, and a C2PSA module;

[0080] The Neck section includes the following modules: UpSample, Concat (first fully connected layer), CBAF (first cross-band attention fusion module), MDIPM (first multi-domain interactive prompting module), C3k2 module, UpSample, Concat (second fully connected layer), C3k2 module, UpSample, Concat (third fully connected layer), Concat (third fully connected layer), C3k2 module, Concat (seventh fully connected layer), Concat (sixth fully connected layer), Concat (fourth fully connected layer), Concat (eighth fully connected layer), Concat (seventh fully connected layer), Concat (fifth fully connected layer), Concat (ninth fully connected layer), Concat (eighth fully connected layer), and C3k2 module.

[0081] The prediction head includes a first prediction head, a second prediction head, a third prediction head, and a fourth prediction head.

[0082] The other steps and parameters are the same as those in one of the specific implementation methods one to three.

[0083] Specific Implementation Method Five: This implementation method differs from Specific Implementation Methods One through Four in that the specific working process of the Backbone part is as follows:

[0084] After preprocessing in step S2, the RGB image of the farmland during the seedling stage is sequentially input into the first convolutional layer, the second convolutional layer, and the first C3k2 module. The first C3k2 module outputs feature map A.

[0085] Feature map A is sequentially input into the third convolutional layer and the second C3k2 module, and the second C3k2 module outputs feature map B.

[0086] Feature map B is sequentially input into the fourth convolutional layer and the third C3k2 module, and the third C3k2 module outputs feature map C.

[0087] Feature map C is sequentially input into the fifth convolutional layer and the fourth C3k2 module, and the fourth C3k2 module outputs feature map D.

[0088] Feature map D is input into the SPPF module and the C2PSA module in sequence, and the C2PSA module outputs feature map E.

[0089] The other steps and parameters are the same as those in one of the specific implementation methods one to four.

[0090] Specific Implementation Method Six: This implementation method differs from Specific Implementation Methods One through Five in that the specific working process of the Neck section is as follows:

[0091] Feature map E is processed by the first upsampling upsampler, which outputs feature map F.

[0092] Feature map C and feature map F are passed through the first fully connected layer Concat, and the first fully connected layer Concat outputs feature map G;

[0093] Feature map B is processed by the first cross-band attention fusion module CBAF, which outputs feature map H.

[0094] Feature map A and the RGB image of farmland during the seedling stage after preprocessing in step S2 are processed by the first multi-domain interactive prompting module MDIPM, which outputs feature map I.

[0095] Feature map G passes through the fifth C3k2 module, and the fifth C3k2 module outputs feature map J;

[0096] Feature map J is processed by a second upsampling upsampler, and the second upsampling upsampler outputs feature map J′.

[0097] Feature map J′, feature map B, and feature map H are passed through the second fully connected layer Concat, and the second fully connected layer Concat outputs feature map K.

[0098] Feature map K passes through the sixth C3k2 module, and the sixth C3k2 module outputs feature map L;

[0099] Feature map L is upsampled by a third upsampler, UpSample, which outputs feature map M.

[0100] After preprocessing in step S2, the RGB image of the farmland during the seedling stage, feature map M, feature map A, and feature map I are processed by the third fully connected layer Concat, which outputs feature map N.

[0101] Feature map N is input to the seventh C3k2 module, and the seventh C3k2 module outputs feature map O.

[0102] Feature map O is input to the sixth convolutional layer, and the sixth convolutional layer outputs feature map O′;

[0103] Feature map O′ and feature map L are sequentially input into the fourth fully connected layer Concat and the eighth C3k2 module. The eighth C3k2 module outputs feature map P.

[0104] The feature map P is input into the seventh convolutional layer, and the seventh convolutional layer outputs the feature map Q;

[0105] The feature map Q output by the seventh convolutional layer and the feature map J output by the fifth C3k2 module are sequentially input into the fifth fully connected layer Concat and the ninth C3k2 module. The ninth C3k2 module outputs feature map R.

[0106] Feature map R is input to the eighth convolutional layer, and the eighth convolutional layer outputs feature map S;

[0107] Feature map S and feature map E are input into the sixth fully connected layer Concat. The output feature map of the sixth fully connected layer Concat is input into the tenth C3k2 module. The output feature map T of the tenth C3k2 module is the feature map.

[0108] The other steps and parameters are the same as those in one of the specific implementation methods one to five.

[0109] Specific Implementation Method Seven: This implementation method differs from Specific Implementation Methods One through Six in that the specific working process of the prediction part Head is as follows:

[0110] The feature map T output by the tenth C3k2 module is input into the fourth prediction head, and the fourth prediction head outputs the classification result.

[0111] The feature map R output by the ninth C3k2 module is input into the third prediction head, and the third prediction head outputs the classification result.

[0112] The feature map P output by the eighth C3k2 module is input into the second prediction head, and the second prediction head outputs the classification result.

[0113] The feature map O output by the seventh C3k2 module is input into the first prediction head, and the first prediction head outputs the classification result;

[0114] Each of the first, second, third, and fourth prediction heads outputs a prediction result containing the bounding box location, confidence level, and class probability.

[0115] Multi-scale feature maps can be used to obtain the detection results of seedling posture quality (normal, floating, damaged seedlings) through the prediction head.

[0116] Each prediction head is responsible for detecting seedling targets at a specific scale. Larger-scale feature maps are used to detect larger seedling targets, while smaller-scale feature maps are used to detect smaller seedling targets.

[0117] like Figure 1 As shown, the prediction part contains prediction heads (Detect) at multiple scales.

[0118] The preprocessed RGB image is input into the Backbone part of the YOLOv11 object detection framework for feature extraction;

[0119] The backbone typically consists of a series of convolutional layers, pooling layers, and activation functions. For example, the C3K2 module is used to extract shallow texture features and deep semantic features of an image.

[0120] Multi-scale feature maps are extracted at different stages of the backbone for subsequent multi-scale feature fusion and object detection.

[0121] The other steps and parameters are the same as those in one of the specific implementation methods one to six.

[0122] Specific Implementation Method Eight: This implementation method differs from Specific Implementation Methods One to Seven in that the specific working process of the first multi-domain interactive prompting module MDIPM is as follows:

[0123] To address the interference of complex backgrounds on seedling target detection, this invention designs a novel multi-domain interactive cue module. The core idea of ​​this module is to extract saliency information and attention weights from different angles of the image, guiding the network to focus more on potential seedling regions and suppressing background noise interference. The structure of this module is as follows: Figure 3 As shown.

[0124] Specifically, the multi-domain interactive prompt module includes the following three sub-modules:

[0125] 1) Obtain the saliency mask attention map; the specific process is as follows: this submodule aims to extract the saliency information of the image from the frequency domain perspective.

[0126] 11) Transfer the image As input;

[0127] image The first C3k2 module outputs feature map A and the RGB image of farmland during the seedling stage after preprocessing in step S2;

[0128] Where R represents a real number, H represents the height of the image, and W represents the width of the image;

[0129] 12) Transfer the image Convert to grayscale image I gray ∈R H×W ;

[0130] image The first C3k2 module outputs feature map A and the RGB image of farmland during the seedling stage after preprocessing in step S2;

[0131] 13) For grayscale image I gray Performing a two-dimensional discrete Fourier transform yields the grayscale image F(u,v) after the two-dimensional discrete Fourier transform:

[0132] F(u,v)=F(I gray (u,v)

[0133] Wherein, F(I) gray ) represents the grayscale image I gray Perform a two-dimensional discrete Fourier transform operation; (u,v) represents the grayscale image I. gray The coordinates in the frequency domain; F(u,v) represents the coordinates of the grayscale image I. gray Grayscale image after performing a two-dimensional discrete Fourier transform;

[0134] 14) The spectrum F is obtained by centering and shifting the grayscale image F(u,v) after the two-dimensional discrete Fourier transform. shift (u,v);

[0135] 15) Calculate the amplitude spectrum M(u,v) = 20 × log(|F shift (u,v)|+1), used for visualization;

[0136] To highlight high-frequency information, an ideal high-pass filter mask Mask(u,v) with a zero center region is constructed:

[0137]

[0138] Where r represents the radius of the circular region at the center of the mask;

[0139] 16) Apply the mask (u,v) to the spectrum F shift (u,v) yields the spectrum F after applying the mask. masked (u,v)=F shift (u,v)×Mask(u,v);

[0140] 17) Recover the saliency map of the spatial domain using the inverse Fourier transform.

[0141] in, I represents the inverse Fourier transform. saliency (x,y) represents a significance plot;

[0142] 18) For the significance graph I saliency Normalizing (x,y) yields I saliency_norm (x,y);

[0143] 19) Based on the threshold τ saliency Generate a binarized saliency mask attention map M saliency :

[0144]

[0145] Among them, M saliency (x,y) represents the binarized saliency mask attention map, M saliency ∈R H×W×1 ;

[0146] The saliency mask M for binarization saliency (x,y) can effectively highlight high-frequency information in an image, corresponding to areas of dramatic change in the image, such as the edges of seedlings.

[0147] 2) Obtain the color attention weight map; the specific process is as follows:

[0148] Given that rice seedlings typically exhibit a specific shade of green, this submodule aims to use color information to guide the network to focus on potential seedling areas.

[0149] 21) Convert the RGB image of the rice seedling stage farmland after preprocessing in step S2 to the HSV color space to obtain image I in the HSV color space. HSV ∈R H×W×3 ;

[0150] 22) Define the learnable hue range θ H =[Hmin H max ], saturation range θ S =[S min ,S max ], and brightness range θ V =[V min V max ];

[0151] Among them, H min Indicates the minimum hue value; H max Indicates the maximum hue value; S min S represents the minimum saturation value. max V represents the maximum saturation value; min Indicates the minimum brightness value; V max Indicates the maximum brightness;

[0152] These thresholds are dynamically learned and optimized by the model during training;

[0153] 23) Calculate the image I in the HSV color space. HSV The probability mask P for each pixel belonging to the seedling region color :

[0154] P color (x,y)=σ(w H ×(I HSV (x,y,0)-H center ) 2 +w S ×(I HSV (x,y,1)-S center ) 2 +w V ×(I HSV (x,y,2)-V center ) 2 )

[0155] in,

[0156] H center For the hue range θ H The center value of S; center For the saturation range θ S The center value of V; center The brightness range θ V The central value;

[0157] I HSV (x,y,0) represents the image I. HSV The hue component of the HSV color space for the pixel (x,y);

[0158] I HSV (x,y,1) represents the image I.HSV The saturation components of the HSV color space for the pixel (x,y);

[0159] I HSV (x,y,2) represents the image I. HSV The lightness component of the HSV color space for the pixel (x,y);

[0160] σ(·) is the Sigmoid function;

[0161] w H ,w S ,w V These are learnable weights;

[0162] 24) For the probability mask P color The color attention weight map A is obtained by normalization. color :

[0163]

[0164] Wherein, min(P) color P is the probability mask. color The minimum value; max(P) color P is the probability mask. color The maximum value;

[0165] 25) To increase smoothness, the color attention weight map A is adjusted. color Gaussian filtering is performed to obtain the Gaussian-filtered color attention weight map A. color ;

[0166] 3) Obtain attention-guided feature maps; the specific process is as follows:

[0167] To further suppress potential interference factors such as weeds, this submodule introduces a learnable attention branch.

[0168] 31) The RGB image of the rice seedling stage farmland after preprocessing in step S2 is passed through a convolutional block Conv weed Generate attention weight map A weed ; indicates as:

[0169] A weed =σ(Conv weed (F shallow )),

[0170] Where σ is the Sigmoid activation function;

[0171] Conv weed For convolutional blocks, convolutional blocks Conv weed It consists of, in sequence, a convolutional layer, a max-pooling layer, and another convolutional layer; Aweed ∈R H ×W×1 ;

[0172] F shallow The image shows the RGB image of farmland during the seedling stage after preprocessing in step S2.

[0173] Shallow features contain richer texture and edge information, which helps to distinguish between weeds and seedlings. 32)

[0175] The saliency mask attention map M saliency The feature map F is obtained by passing it through two convolutional layers. saliency ;

[0176] Color attention weight map A after Gaussian filtering color The feature map F is obtained by passing it through two convolutional layers. color ;

[0177] Attention weight map A weed The feature map F is obtained by passing it through two convolutional layers. weed ;

[0178] 33) Transfer feature map F saliency Feature map F color Feature map F weed The concatenation is performed along the channel dimension, and the concatenated feature map is obtained by passing it through a linear layer.

[0179] The dimensionality-reduced feature map and the output feature map A of the first C3k2 module are input into the cross-attention module for fusion to obtain the final attention-guided feature map.

[0180] The other steps and parameters are the same as those in any of the specific implementation methods one to seven.

[0181] Specific Implementation Method Nine: This implementation method differs from Specific Implementation Methods One to Eight in that the specific working process of the first cross-band attention fusion module CBAF is as follows:

[0182] The designed Cross-Band Attention Fusion Module (CBAF) aims to fuse RGB image features with specific band information provided by multispectral images to enhance the ability to identify seedlings in different postures.

[0183] The module's input includes: the feature map output by the second C3k2 module. And preprocessed multi-band spectral images of 450nm, 660nm, 750nm, and 840nm; C3 represents the number of channels in feature map B, W3 represents the width of feature map B, and H3 represents the height of feature map B.

[0184] The specific steps are as follows: 1)

[0186] Multispectral images in the 450nm and 660nm bands were stitched together along the channel dimension. The stitched images were then input into the seedling state feature extractor 1 (SCFE1) for feature extraction, resulting in the damaged seedling feature D1.

[0187] The main function of SCFE1 is to fuse dual-band information, extract features, and reduce dimensionality so that its feature dimension is consistent with the feature map B output by the second C3k2 module.

[0188] Seedling condition feature extractor 1 is SCFE1 (Seedling Condition Feature Extractor 1);

[0189] SCFE1 contains the first convolutional block, the second convolutional block, and the third convolutional block in sequence;

[0190] Each of the first, second, and third convolutional blocks contains, in sequence, a convolutional layer, a ReLU activation function layer, a convolutional layer, a ReLU activation function layer, a convolutional layer, and a max pooling layer;

[0191] Blue light (450nm) and red light (660nm) bands are more sensitive to chlorophyll absorption and can effectively reflect chlorophyll loss, thus they are more suitable for extracting characteristics of damaged seedlings. 2)

[0193] Multispectral images in the 750nm and 840nm bands are stitched together along the channel dimension. The stitched images are then input into seedling state feature extractor 2 (with the same structure as seedling state feature extractor 1) for feature extraction, resulting in the floating seedling feature D2.

[0194]

[0195] Seedling condition feature extractor 2 is SCFE2 (Seedling Condition Feature Extractor 2);

[0196] SCFE2 contains a first convolutional block, a second convolutional block, and a third convolutional block in sequence;

[0197] Each of the first, second, and third convolutional blocks contains, in sequence, a convolutional layer, a ReLU activation function layer, a convolutional layer, a ReLU activation function layer, a convolutional layer, and a max pooling layer;

[0198] SCFE2 (Seedling Condition Feature Extractor 2) is designed for dual-band fusion, feature extraction, and dimensionality reduction. The near-infrared bands (750nm and 840nm) are sensitive to changes in vegetation cell structure and water content, making them more suitable for extracting features of seedlings that have drifted.

[0199] 3) Cross-Attention Fusion:

[0200] Using the damaged seedling feature D1 as the key K, and the feature map B output by the second C3k2 module as the query Q and value V respectively, a cross-attention fusion operation is performed to obtain the feature map.

[0201] This operation allows RGB features to focus on spectral information related to damaged seedlings, thereby enhancing the network's ability to identify damaged seedlings.

[0202] Using the floating seedling feature D2 as the key K, and the feature map B output by the second C3k2 module as the query Q and value V respectively, a cross-attention fusion operation is performed to obtain the feature map.

[0203] This operation enables RGB features to focus on spectral information related to floating seedlings, thereby improving the network's ability to recognize floating seedlings.

[0204] The calculation formula for cross-attention fusion operation is as follows:

[0205]

[0206] Where, d k It represents the dimension of key K; the superscript T indicates transpose; softmax represents the activation function; Attention(Q,K,V) represents cross-attention fusion operation;

[0207] 4) The feature map obtained after cross-attention fusion operation and The fusion process yields the cross-band attention map F. CBAF The specific process is as follows:

[0208] Fusion is achieved by convolutional layers reducing the dimensionality of channels after concatenation.

[0209]

[0210] Among them, Conv fuse For convolutional layers used to process the fused feature maps, Concat() is used for concatenation, F CBAF Cross-band attention map;

[0211] 5) Cross-band attention map F CBAF The feature map of interest is obtained by performing a Hadamard product with the feature map B output by the second C3k2 module.

[0212]

[0213] Among them, ⊙ represents the Hadamarda accumulation.

[0214] like Figure 1 As shown, in the fusion layer, a feature pyramid network (FPN) structure is used for multi-scale feature fusion.

[0215] Feature maps from different stages of the backbone are fused through upsampling and concat operations to obtain feature maps containing information at different scales;

[0216] The specific fusion methods include two pathways: bottom-up and top-down, and features of the same scale are fused through lateral connections.

[0217] The other steps and parameters are the same as those in specific implementation methods one through eight.

[0218] Specific Implementation Method Ten: This implementation method differs from Specific Implementation Methods One through Nine in that: the training of the deep learning model to obtain a trained deep learning model is specifically as follows:

[0219] Using a designed online hard example mining strategy to guide adaptive model training includes the following aspects:

[0220] (1) Input the RGB and multispectral images of the seedling stage farmland after the preprocessing in step S2 into the deep learning model. The first prediction head of the deep learning model outputs the category prediction probability distribution p.

[0221] Based on the true label y and the predicted probability distribution p output by the first prediction head, calculate the cross-entropy loss for each sample. The calculation formula is as follows:

[0222]

[0223] Where C is the number of categories, and its value is 3;

[0224] yi It is the true label of each sample belonging to the i-th category;

[0225] p i It is the probability p that each sample belongs to the i-th class predicted by the deep learning model. i ;

[0226] i = 1, 2 or 3, where i represents qualified seedlings, damaged seedlings or floating seedlings; (2)

[0228] The preprocessed RGB and multispectral images of farmland during the seedling stage, obtained in step S2, are input into the deep learning model. The first prediction head of the deep learning model outputs the category prediction probability distribution p, and the prediction entropy H(p) for each sample is calculated to measure the uncertainty of the model's prediction. The higher the prediction entropy, the more uncertain the model's prediction for that sample. The calculation formula is as follows:

[0229]

[0230] Furthermore, to evaluate the stability of the model's predictions, the Dropout technique was used for T forward propagations, resulting in T predicted probability distributions p. (1) ,p (2) ,...,p (T) Calculate the prediction variance Var(p) for each sample using the following formula:

[0231]

[0232] Where Var(·) represents variance calculation;

[0233] This indicates that the RGB and multispectral images of the rice seedling stage farmland after preprocessing in step S2 are input into the deep learning model for the first time, and the probability of each sample belonging to the i-th category is predicted by the first prediction head of the deep learning model.

[0234] This indicates that the RGB and multispectral images of farmland during the seedling stage, after preprocessing in step S2, are input into the deep learning model for the second time, and the probability of each sample belonging to the i-th category is predicted by the first prediction head of the deep learning model.

[0235] This indicates that the Tth input of the RGB and multispectral images of the rice seedling stage farmland after preprocessing in step S2 into the deep learning model, and the probability of each sample belonging to the i-th category predicted by the first prediction head of the deep learning model;

[0236] (3) Construct the scoring function for each sample:

[0237] Taking into account both cross-entropy loss and uncertainty metric, a scoring function S is designed to evaluate the difficulty level of each sample; the scoring function is as follows:

[0238]

[0239] Where α, β, and γ are hyperparameters used to balance the weights of the loss function and different uncertainty measures, and they are set to 1.0, 0.1, and 0.1, respectively.

[0240] (4) Calculate the score S of all samples, and select the K% of samples with the highest scores as difficult samples based on the scores.

[0241] K is a preset fixed value, which is empirically set to 30% in this method.

[0242] (5) For samples selected as difficult examples, higher weights are given when calculating the loss function, so that the model pays more attention to learning these difficult examples.

[0243] The weight function w(S) associated with the scores of difficult examples is defined as follows:

[0244] w(S)=1+λ×tanh(S)

[0245] Where λ is a hyperparameter that controls the magnitude of weight adjustment;

[0246] For each hard example, the gradient is calculated using a weighted loss function:

[0247]

[0248] in,

[0249] S(p i The score is calculated using a scoring function for the difficult example sample;

[0250] w(S(p i )) represents the weights of the difficult sample;

[0251] For each non-difficult example, the gradient is calculated using a weighted loss function with a weight of 1:

[0252]

[0253] Update deep learning model parameters using the backpropagation algorithm;

[0254] (6) Train the deep learning model until the weighted loss function is calculated. Convergence yields a well-trained deep learning model.

[0255] The other steps and parameters are the same as those in specific implementation methods one through nine.

[0256] While the technical solution of this invention can effectively achieve the intended objectives, there may be alternative solutions when performance requirements are adjusted. However, these alternative solutions often fall short of the technical effects of this invention in handling complex farmland environments and fully utilizing multimodal information.

[0257] The core technical challenge of this invention lies in how to effectively fuse RGB and multispectral information and accurately identify seedlings in different poses against complex backgrounds. This invention effectively solves these problems by constructing a multi-domain interactive prompting module and a cross-band attention fusion module, combined with a designed online hard example mining strategy. The multi-domain interactive prompting module aims to suppress background interference and highlight the seedling target; the cross-band attention fusion module aims to fuse multispectral information and enhance the ability to distinguish different poses; and the online hard example mining strategy improves the robustness and generalization of the model.

[0258] If considering alternatives to S4, the most direct approach is to rely solely on single-modal data for seedling posture quality detection. For example, using only RGB images captured by a drone for detection. In this approach, traditional image processing methods, such as color segmentation and texture analysis, can be employed, combined with machine learning classifiers for seedling target identification and posture classification. However, relying solely on RGB images is easily affected by factors such as lighting changes, shadows, and weeds, making it difficult to accurately distinguish seedlings with different postures, especially those with minor visible damage or slight drift. In contrast, this invention integrates multispectral information, utilizing the differences in spectral reflectance of seedlings with different postures, thus more effectively improving recognition accuracy.

[0259] In the area of ​​multimodal information fusion, while other fusion strategies exist, such as simple feature concatenation or weighted fusion, these methods may not fully tap into the deep correlations and complementary information between different modalities. Feature concatenation simply connects feature vectors from different modalities, lacking intermodal interaction; weighted fusion requires manual weight setting, making it difficult to adaptively adjust the importance of different modalities. In contrast, the cross-band attention fusion module (CBAF) designed in this invention dynamically learns the contribution of different spectral bands to seedling attitude discrimination through an attention mechanism and effectively fuses it with RGB features, enabling more refined utilization of multimodal information.

[0260] Furthermore, while other hard example mining strategies exist in the seedling posture quality detection step S5, such as the fixed threshold hard example selection method based on cross-entropy loss, these methods lack consideration of model prediction uncertainty in their assessment of sample difficulty. In contrast, the online hard example mining strategy employed in this invention dynamically selects hard examples based on the model's uncertainty in predicting samples and assigns them higher training weights. This allows the model to focus more on samples that are difficult to learn from, thereby more effectively improving the model's robustness and generalization ability.

[0261] In summary, while alternative solutions exist for certain aspects of the technical solution, these alternatives may have limitations in addressing complex farmland background interference, effectively integrating multimodal information, and improving the model's fine-grained recognition ability for seedlings in different postures, thus failing to achieve the technical effects of this invention. The multimodal perception method, along with its specific module design and training strategy, proposed in this invention is specifically designed to solve the problem of detecting the posture quality of farmland seedlings, and is therefore necessary and innovative.

[0262] What are the advantages of this invention compared to the closest existing technology?

[0263] Currently, technologies for detecting the attitude quality of rice seedlings in farmland mainly focus on manual inspection and analysis methods based on single-modal remote sensing images. Compared with these existing technologies, the UAV-based method for detecting the attitude quality of rice seedlings in farmland based on multimodal perception proposed in this invention shows significant advantages in terms of information fusion depth, detection accuracy and robustness, and intelligence level, overcoming many limitations of existing technologies. Specifically, these advantages are reflected in the following aspects:

[0264] 1. Significantly Improved Accuracy of Seedling Target Detection in Complex Farmland Backgrounds: Existing seedling detection methods, especially those relying solely on RGB images, often exhibit low detection accuracy in complex and variable farmland environments. Factors such as soil, water bodies, weeds, and changes in lighting can easily cause severe interference, leading to false positives and false negatives of seedling targets. Traditional image processing techniques struggle to effectively distinguish these interfering factors from seedling targets, while general target detection models directly applied to RGB images cannot fully utilize prior knowledge of farmland scenes. In contrast, this invention innovatively proposes a multi-domain interactive prompting module. This module extracts saliency information from multiple dimensions, including the frequency domain and color space, and incorporates a learnable weed suppression mechanism, effectively guiding the network to focus on potential seedling areas and significantly suppressing background noise interference.

[0265] 2. Refined Seedling Posture Quality Recognition Capability: Traditional RGB image-based seedling posture quality detection methods face challenges in distinguishing seedlings with different postures, especially for subtle differences in appearance, such as slightly damaged seedlings versus healthy seedlings, where RGB information alone is insufficient for accurate differentiation. This invention, by fusing multispectral information, leverages the reflectance differences of seedlings with different postures across different spectral bands to achieve more refined posture quality recognition. For example, healthy and damaged seedlings exhibit significant differences in reflectance in the near-infrared band, and drifting seedlings, due to changes in moisture state, also display different spectral reflectance characteristics compared to healthy seedlings. The cross-band attention fusion module proposed in this invention effectively captures these subtle spectral differences, combining them with texture and shape information provided by RGB images to more accurately distinguish drifting, damaged, and healthy seedlings, improving the accuracy and reliability of posture quality assessment—a feat difficult to achieve with existing technologies relying solely on RGB images.

[0266] 3. Improved Model Robustness and Generalization Ability: Traditional deep learning model training typically employs uniform or random sampling methods, which can lead to excessive effort being spent on a large number of simple samples, while neglecting the learning of difficult examples with low discrimination and easy confusion. This limits the model's generalization ability in complex scenarios. To address this, this invention designs an online difficult example mining strategy for seedlings. During training, this strategy dynamically identifies "difficult example" samples that the model struggles to accurately recognize (such as some floating and damaged seedlings that are difficult to distinguish), assigning these samples higher training weights and forcing the model to focus more on learning these difficult examples. This adaptive training method effectively improves the model's robustness and generalization ability, enhancing its practical application value.

[0267] This invention may have other embodiments. Without departing from the spirit and essence of this invention, those skilled in the art can make various corresponding changes and modifications according to this invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.

Claims

1. A method for detecting the quality of rice transplanting posture based on multimodal perception, characterized in that: The specific process of the method is as follows: Step S1: Collect RGB and multispectral images of farmland during the seedling growth period; Step S2: Preprocess the RGB and multispectral images of farmland during the seedling growth period to obtain preprocessed RGB and multispectral images of farmland during the seedling growth period. Step S3: Construct a deep learning model by inputting the preprocessed RGB and multispectral images of farmland during the seedling growth period into the deep learning model for training, and obtain a trained deep learning model. Step S4: Based on the trained deep learning model, classify and predict the RGB and multispectral images of farmland during the seedling growth period to be tested, and predict the seedling category in the RGB and multispectral images of farmland during the seedling growth period to be qualified seedlings, damaged seedlings, or drifting seedlings. The deep learning model is constructed in S3; the specific process is as follows: Deep learning models consist of a backbone, a neck, and a prediction head. The backbone consists of, in sequence, a first convolutional layer, a second convolutional layer, a first C3k2 module, a third convolutional layer, a second C3k2 module, a fourth convolutional layer, a third C3k2 module, a fifth convolutional layer, a fourth C3k2 module, an SPPF module, and a C2PSA module; The Neck section includes the following modules: UpSample, Concat (first fully connected layer), CBAF (first cross-band attention fusion module), MDIPM (first multi-domain interactive prompting module), C3k2 module, UpSample, Concat (second fully connected layer), C3k2 module, UpSample, Concat (third fully connected layer), Concat (third fully connected layer), C3k2 module, Concat (seventh fully connected layer), Concat (sixth fully connected layer), Concat (fourth fully connected layer), Concat (eighth fully connected layer), Concat (seventh fully connected layer), Concat (fifth fully connected layer), Concat (ninth fully connected layer), Concat (eighth fully connected layer), and C3k2 module. The prediction header includes a first prediction header, a second prediction header, a third prediction header, and a fourth prediction header; The specific working process of the first multi-domain interactive prompt module MDIPM is as follows: 1) Obtain the saliency mask attention map; the specific process is as follows: 11) Transfer the image As input; ; image The first C3k2 module outputs feature map A and the RGB image of farmland during the seedling stage after preprocessing in step S2; in, Represent real numbers, Indicates the height of the image. Indicates the width of the image; 12) Transfer the image Convert to grayscale ; 13) For grayscale images Perform a two-dimensional discrete Fourier transform to obtain the grayscale image after the two-dimensional discrete Fourier transform. : in, Indicates grayscale image The operation of performing a two-dimensional discrete Fourier transform; Represents a grayscale image The coordinates in the frequency domain; Indicates grayscale image Grayscale image after performing a two-dimensional discrete Fourier transform; 14) Grayscale image after two-dimensional discrete Fourier transform The spectrum is obtained by centering and shifting. ; 15) Construct an ideal high-pass filter mask with zero values ​​in the central region. : in, Indicates the radius of the circular region at the center of the mask; 16) Place the mask Applied to the spectrum Obtain the spectrum after applying the mask ; 17) Recover the saliency map of the spatial domain using the inverse Fourier transform. ； in, Indicates the inverse Fourier transform. Represents a saliency plot; 18) For saliency plots Normalization yields ; 19) Based on the threshold Generate a binarized saliency mask attention map : in, This represents the saliency mask attention map for binarization. ; 2) Obtain the color attention weight map; the specific process is as follows: 21) Convert the RGB image of the rice seedling stage farmland after preprocessing in step S2 to the HSV color space to obtain the image in the HSV color space. ; 22) Define the learnable tonal range saturation range and brightness range ; in, Indicates the minimum value of the hue; Indicates the maximum value of the hue; This represents the minimum saturation value; This indicates the maximum saturation value; This indicates the minimum brightness value; Indicates the maximum brightness; 23) Calculate the image in the HSV color space. Probability mask for each pixel belonging to the seedling region : in, For hue range The central value; Saturation range The central value; Brightness range The central value; Representing an image pixels The hue component of the HSV color space; Representing an image pixels The saturation component of the HSV color space; Representing an image pixels The lightness component of the HSV color space; For the Sigmoid function; These are learnable weights; 24) Probability Mask The color attention weight map is obtained by normalization. : in, For probability mask The minimum value; For probability mask The maximum value; 25) Color attention weighting map Gaussian filtering is performed to obtain the color attention weight map after Gaussian filtering. ; 3) Obtain attention-guided feature maps; the specific process is as follows: 31) The RGB image of the rice seedling stage farmland after preprocessing in step S2 is passed through a convolution block. Generate attention weight map ; indicates as: in, Use the Sigmoid activation function; For convolutional blocks, convolutional blocks It consists of convolutional layers, max pooling layers, and another convolutional layer in sequence; ; The image shows the RGB image of farmland during the seedling stage after preprocessing in step S2. 32)、 Saliency mask attention map The feature map is obtained by passing it through two convolutional layers in sequence. ; Color attention weight map after Gaussian filtering The feature map is obtained by passing it through two convolutional layers in sequence. ; Attention weight map The feature map is obtained by passing it through two convolutional layers in sequence. ; 33) Feature map Feature map Feature map The concatenation is performed along the channel dimension, and the concatenated feature map is obtained by passing it through a linear layer; The dimensionality-reduced feature map and the output feature map A of the first C3k2 module are input into the cross-attention module for fusion to obtain the final attention-guided feature map. .

2. The rice transplanting posture quality detection method based on multimodal perception according to claim 1, characterized in that: In step S1, RGB and multispectral images of farmland are acquired during the seedling growth period; the specific process is as follows: Integrate visible light cameras and multispectral cameras into the drone platform; Simultaneously acquire RGB and multispectral images of seedlings during their growth period.

3. The rice transplanting posture quality detection method based on multimodal perception according to claim 2, characterized in that: In step S2, the RGB and multispectral images of farmland during the seedling stage are preprocessed to obtain preprocessed RGB and multispectral images of farmland during the seedling stage; the specific process is as follows: S21. Image geometric correction and atmospheric correction; the specific process is as follows: (1) Perform geometric correction on the acquired RGB and multispectral images; (2) Perform atmospheric correction on the geometrically corrected RGB and multispectral images to obtain accurate surface reflectance data; represented as: in, This represents the spectral radiance of the RGB image. The Earth-Sun distance; The average extraterrestrial irradiance of the sun in a given wavelength band; The solar zenith angle; Surface reflectance; S22. Perform band registration on the atmospherically corrected multispectral image to obtain the band-registered multispectral image; the specific process is as follows: Register different bands of the atmospherically corrected multispectral image to ensure that the same ground feature corresponds to the same spatial location in images of different bands; S23. Label the atmospherically corrected RGB image and the band-registered multispectral image; the specific process is as follows: The atmospherically corrected RGB images and the band-registered multispectral images were labeled as qualified seedlings, damaged seedlings, or drifting seedlings. S24. Image enhancement and denoising; the specific process is as follows: (1) Enhance the labeled RGB image and multispectral image to obtain the enhanced image; (2) Denoise the enhanced image to obtain the denoised image; the specific process is as follows: The enhanced image is convolved with a Gaussian kernel to obtain the denoised image.

4. The rice transplanting posture quality detection method based on multimodal perception according to claim 3, characterized in that: The specific working process of the Backbone component is as follows: After preprocessing in step S2, the RGB image of the farmland during the seedling stage is sequentially input into the first convolutional layer, the second convolutional layer, and the first C3k2 module. The first C3k2 module outputs feature map A. Feature map A is sequentially input into the third convolutional layer and the second C3k2 module, and the second C3k2 module outputs feature map B. Feature map B is sequentially input into the fourth convolutional layer and the third C3k2 module, and the third C3k2 module outputs feature map C. Feature map C is sequentially input into the fifth convolutional layer and the fourth C3k2 module, and the fourth C3k2 module outputs feature map D. Feature map D is input into the SPPF module and the C2PSA module in sequence, and the C2PSA module outputs feature map E.

5. The rice transplanting posture quality detection method based on multimodal perception according to claim 4, characterized in that: The specific working process of the Neck section is as follows: Feature map E is processed by the first upsampling upsampler, which outputs feature map F. Feature map C and feature map F are passed through the first fully connected layer Concat, and the first fully connected layer Concat outputs feature map G; Feature map B is processed by the first cross-band attention fusion module CBAF, which outputs feature map H. Feature map A and the RGB image of farmland during the seedling stage after preprocessing in step S2 are processed by the first multi-domain interactive prompting module MDIPM, which outputs feature map I. Feature map G passes through the fifth C3k2 module, and the fifth C3k2 module outputs feature map J; Feature map J is processed by a second upsampling upsampler, which outputs a feature map. ; Feature map Feature map B and feature map H are passed through the second fully connected layer Concat, and the second fully connected layer Concat outputs feature map K; Feature map K passes through the sixth C3k2 module, and the sixth C3k2 module outputs feature map L; Feature map L is upsampled by a third upsampler, UpSample, which outputs feature map M. After preprocessing in step S2, the RGB image of the farmland during the seedling stage, feature map M, feature map A, and feature map I are processed by the third fully connected layer Concat, which outputs feature map N. Feature map N is input to the seventh C3k2 module, and the seventh C3k2 module outputs feature map O. Feature map O is input to the sixth convolutional layer, and the sixth convolutional layer outputs the feature map. ; Feature map The feature map L is sequentially input into the fourth fully connected layer Concat and the eighth C3k2 module. The eighth C3k2 module outputs the feature map P. The feature map P is input into the seventh convolutional layer, and the seventh convolutional layer outputs the feature map Q; The feature map Q output by the seventh convolutional layer and the feature map J output by the fifth C3k2 module are sequentially input into the fifth fully connected layer Concat and the ninth C3k2 module. The ninth C3k2 module outputs feature map R. Feature map R is input to the eighth convolutional layer, and the eighth convolutional layer outputs feature map S; Feature map S and feature map E are input into the sixth fully connected layer Concat. The output feature map of the sixth fully connected layer Concat is input into the tenth C3k2 module. The output feature map T of the tenth C3k2 module is the feature map.

6. The rice transplanting posture quality detection method based on multimodal perception according to claim 5, characterized in that: The specific working process of the prediction section Head is as follows: The feature map T output by the tenth C3k2 module is input into the fourth prediction head, and the fourth prediction head outputs the classification result. The feature map R output by the ninth C3k2 module is input into the third prediction head, and the third prediction head outputs the classification result. The feature map P output by the eighth C3k2 module is input into the second prediction head, and the second prediction head outputs the classification result. The feature map O output by the seventh C3k2 module is input into the first prediction head, and the first prediction head outputs the classification result; Each of the first, second, third, and fourth prediction heads outputs a prediction result containing the bounding box location, confidence level, and class probability.

7. The rice transplanting posture quality detection method based on multimodal perception according to claim 6, characterized in that: The specific working process of the first cross-band attention fusion module CBAF is as follows: 1)、 Multispectral images in the 450nm and 660nm bands are stitched together along the channel dimension. The stitched images are then input into the seedling state feature extractor 1 for feature extraction to obtain damaged seedling features. ; Seedling status feature extractor 1 is SCFE1; SCFE1 contains the first convolutional block, the second convolutional block, and the third convolutional block in sequence; Each of the first, second, and third convolutional blocks contains, in sequence, a convolutional layer, a ReLU activation function layer, a convolutional layer, a ReLU activation function layer, a convolutional layer, and a max pooling layer; 2)、 Multispectral images in the 750nm and 840nm bands were stitched together along the channel dimension. The stitched images were then input into the seedling state feature extractor 2 for feature extraction to obtain the floating seedling features. ; Seedling status feature extractor 2 is SCFE2; SCFE2 contains a first convolutional block, a second convolutional block, and a third convolutional block in sequence; Each of the first, second, and third convolutional blocks contains, in sequence, a convolutional layer, a ReLU activation function layer, a convolutional layer, a ReLU activation function layer, a convolutional layer, and a max pooling layer; 3) Cross-attention fusion: Characteristics of damaged seedlings As the key K, the feature map B output by the second C3k2 module is used as the query Q and the value V, respectively, and a cross-attention fusion operation is performed to obtain the feature map. ; floating seedling characteristics As the key K, the feature map B output by the second C3k2 module is used as the query Q and the value V, respectively, and then subjected to cross-attention fusion operation to obtain the feature map. ; The calculation formula for cross-attention fusion operation is as follows: in, It represents the dimension of key K; the superscript T indicates transpose. Indicates the activation function; Indicates cross-attention fusion operation; 4) The feature map obtained after cross-attention fusion operation and By fusing the data, a cross-band attention map is obtained. The specific process is as follows: Fusion is achieved by convolutional layers reducing the dimensionality of channels after concatenation. in, For use in processing the fused feature maps, convolutional layers For splicing, Cross-band attention map; 5) Cross-band attention map Feature map output by the second C3k2 module Features that have gained attention after the Hadamard product : in, For Hadama accumulation.

8. The rice transplanting posture quality detection method based on multimodal perception according to claim 7, characterized in that: The process of training a deep learning model to obtain a trained deep learning model is as follows: (1)、 The preprocessed RGB and multispectral images of farmland during the seedling stage, obtained in step S2, are input into the deep learning model. The first prediction head of the deep learning model outputs the category prediction probability distribution. ; Based on real labels and the predicted probability distribution of the first prediction head output Calculate the cross-entropy loss for each sample. The calculation formula is as follows: in, This represents the number of categories, with a value of 3. Is each sample belonging to the first... Real labels for each category; The deep learning model predicts that each sample belongs to the first... The probability of each category ; =1, 2, or 3 This indicates qualified seedlings, damaged seedlings, or drifting seedlings; (2)、 The preprocessed RGB and multispectral images of farmland during the seedling stage, obtained in step S2, are input into the deep learning model. The first prediction head of the deep learning model outputs the category prediction probability distribution. Calculate the prediction entropy for each sample. The calculation formula is as follows: Using Dropout technology The second forward propagation yields... Each predicted probability distribution Calculate the prediction variance for each sample. The calculation formula is as follows: in, Indicates variance calculation; This indicates that the RGB and multispectral images of the rice seedling stage farmland after preprocessing in step S2 are input into the deep learning model for the first time, and the probability of each sample belonging to the i-th category is predicted by the first prediction head of the deep learning model. This indicates that the RGB and multispectral images of farmland during the seedling stage, after preprocessing in step S2, are input into the deep learning model for the second time, and the probability of each sample belonging to the i-th category is predicted by the first prediction head of the deep learning model. This indicates that the Tth input of the RGB and multispectral images of the rice seedling stage farmland after preprocessing in step S2 into the deep learning model, and the probability of each sample belonging to the i-th category predicted by the first prediction head of the deep learning model; (3) Construct the scoring function for each sample: The scoring function is as follows: in, It's a hyperparameter; (4) Calculate the scores for all samples. And select the highest-rated candidate based on the score. The samples are used as difficult examples; (5)、 Define a weight function related to the scores of difficult examples. as follows: in, It is a hyperparameter that controls the magnitude of weight adjustment; For each hard example, the gradient is calculated using a weighted loss function: in, The scores obtained by using a scoring function for the difficult examples; Weights for difficult examples; For each non-difficult example, the gradient is calculated using a weighted loss function with a weight of 1: Update deep learning model parameters using the backpropagation algorithm; (6) Train the deep learning model until the weighted loss function is calculated. Convergence yields a well-trained deep learning model.

Images