A fruit recognition method for a picking robot based on millimeter-wave radar

Abstract

The present invention relates to the field of fruit recognition of a picking robot in agriculture, and in particular to a fruit recognition method of a picking robot based on a millimeter wave radar, comprising: S1, building a MIMO SAR imaging system, installing a millimeter wave radar and a binocular vision sensor; S2, building a Stable Diffusion model; S3, SAR image fusion optimization; S4, generating and outputting a stereo image; S5, a picking robot picking the blocked fruit. The present invention recognizes and locates the fruit blocked by leaves by fusing the SAR image collected by the millimeter wave radar with the binocular vision image collected by the binocular vision sensor, and generates a stereo image and positioning coordinates, so that the control system can control the picking robot to pick the blocked fruit. The fruit recognition method can not only recognize the blocked fruit, but also can use the picking robot to automatically pick the blocked fruit, replacing manual picking, which not only improves the picking efficiency, but also reduces the labor intensity of the pickers.

Description

Technical Field

[0001] The invention relates to the field of fruit recognition of a picking robot in agriculture, and in particular to a fruit recognition method of a picking robot based on a millimeter wave radar. Background Art

[0002] Picking robots, also known as collection robots, are mainly used in agricultural production. In the fruit harvest season, picking robots can greatly reduce the labor intensity of workers. To a certain extent, they have replaced the pure manual method of picking fruits. With the development and progress of science and technology, picking robots have gradually entered the public's field of vision. Using them to pick fruits not only has fast picking speed and high efficiency, but also greatly reduces the labor intensity of farmers.

[0003] Current agricultural picking robots mainly use visual recognition of fruits or laser radar recognition of fruits or a combination of the two to identify fruits, and then implement it through fruit-related image features and image processing. For example, the invention patent with the authorization announcement as "CN102682286B" and the invention name as "Fruit recognition method for picking robots based on laser vision system" mainly uses a laser vision system to identify fruits, and then the picking robot picks the fruits through recognition and positioning by the laser vision system.

[0004] The harvesting robot usually identifies the fruit through a visual system, and the visual system of the harvesting robot has a variety of perception capabilities, including visual perception, cooperative visual mechanical control, visual recognition, three-dimensional reconstruction, coordinated visual mechanical positioning and fault tolerance. These capabilities can be applied to various types of fruits and crops, and the effectiveness of vision-based picking may depend on factors such as the size, shape, color, texture of the fruit or crop, and the complexity of the surrounding environment. This system includes crop recognition and positioning, stereoscopic vision and cooperative behavior control of the robot manipulator, and error handling. The system identifies crops by collecting a set of images and extracting information about the target. To identify the fruit, the system processes the generated three-dimensional reconstruction of the fruit, fruit branches, and obstacles on the fruit branches through spatial coordinate calculations. After completing these calculations, the calculated spatial coordinates are transmitted to the robot's drive system. The system controls the robot's work and implements algorithms and intelligent decisions through visual software. When the environment of the crop changes under the lighting and occlusion conditions of the field environment, the accuracy of recognition and positioning will be affected.

[0005] Millimeter wave radar is a radar that works in the millimeter wave band (millimeter wave). Usually millimeter waves refer to the 30-300GHz frequency domain (wavelength is 1-10mm). Compared with other sensors, millimeter wave radar has significant advantages, with extremely high operating frequency, short wavelength, high resolution, and all-weather measurement capabilities. Compared with weather factors, it is less affected and can still be used normally in rain and snow conditions. It is particularly suitable for navigation and positioning.

[0006] The radar echo antennas on the left and right sides of the millimeter-wave radar are used to form the left and right echo difference. The echo difference is calculated to obtain the phase difference. The phase difference can be used to obtain the left and right distance difference, that is, the angular resolution. At the same time, the Doppler effect formed by the movement of the detected object can be used to accurately calculate the moving speed of the object. The Doppler image is formed, and the running speed and distribution image of the object are finally obtained. The millimeter-wave radar imaging is combined with binocular vision imaging. Through feature recognition and fusion technology, the target image is constructed, and the recognition and positioning of the obscured object are realized.

[0007] Whether it is the above-mentioned patented technology that has been disclosed or the current more cutting-edge fruit recognition technology, it is mainly used to identify unobstructed fruits with a high recognition rate. However, for fruits that are partially or completely obscured by leaves, not only is the recognition rate poor, but sometimes there are problems of being unable to recognize or recognizing errors. The main reason is that under normal circumstances, since the fruits are obscured by branches and leaves, it is difficult to distinguish and judge accurately through images and lidar. As a result, the obscured fruits cannot be effectively picked by picking robots and can only be picked manually. Manual picking is not only labor-intensive but also difficult, which affects the efficiency of fruit picking. Summary of the invention

[0008] In order to identify fruits that are partially or completely blocked by leaves, and enable a picking robot to pick fruits automatically to replace manual picking and reduce labor intensity, the present invention provides a fruit recognition method for a picking robot based on millimeter wave radar, which identifies and locates fruits blocked by leaves by fusing SAR images collected by the millimeter wave radar with binocular vision images collected by a binocular vision sensor, and generates a stereo image and positioning coordinates, so that the control system can control the picking robot to pick the blocked fruits. The fruit recognition method can not only identify the blocked fruits, but also use the picking robot to automatically pick the blocked fruits, replacing manual picking, which not only improves the picking efficiency, but also reduces the labor intensity of the pickers.

[0009] The present invention provides a fruit recognition method for a picking robot based on millimeter wave radar, which adopts the following technical solution:

[0010] A fruit recognition method for a picking robot based on millimeter wave radar comprises the following steps:

[0011] S1. Build a MIMO SAR imaging system and install a millimeter-wave radar and binocular vision sensor at the end of the robotic arm;

[0012] Millimeter wave radar and binocular vision sensors image the target object and preliminarily analyze the imaging characteristics;

[0013] A crop microwave scattering model is established based on the first-order solution of the radiation transfer equation, and the residual scattering of the fruit is derived using the crop microwave scattering model. Then, the backscattering coefficient of the obstruction is corrected using the AIEM-Oh model to obtain the corrected backscattering coefficient.

[0014] S2. Build a Stable Diffusion model;

[0015] S21. Model selection and adaptability;

[0016] S22. Application of latent diffusion models (LDMs);

[0017] S23, setting positive and negative prompt words to obtain the required target SAR image;

[0018] S24, RGB detection auxiliary annotation;

[0019] S25. Verification and evaluation of annotated datasets;

[0020] S26, iterative optimization of potential diffusion model;

[0021] S3, SAR image fusion optimization;

[0022] S31. Improved multi-scale fusion technology:

[0023] Based on the traditional feature pyramid network, a new multi-scale Transformer structure is introduced. The dynamic weight allocation mechanism in deep learning is used to dynamically adjust the image fusion weight according to the feature information of different scales, so that features with higher information content under occlusion conditions will be able to obtain more weight;

[0024] S32, Dynamic Attention Module:

[0025] Design a dynamic attention module that adjusts weights by learning the correlations between different features;

[0026] S33, mask pre-training process:

[0027] Using the mask pre-training process, the latent diffusion model is trained to identify the areas in the image that are masked by random masks and try to reconstruct the information of these masked areas;

[0028] S4, generate a stereo image and output it;

[0029] Using the F-PointNet open source algorithm, after distinguishing between fruits and leaves, the SAR image and the binocular vision image are fused through edge feature recognition and pixel matrix fusion technology to generate a stereo image combining 3D point cloud and RGB, and automatically fit the shape of the fruit of the occluded part, so that the picking robot can identify and locate the occluded fruit, generate positioning coordinates, and then upload the positioning coordinates of the identified and located occluded fruit to the control system;

[0030] S5. The control system controls the picking head of the mechanical part of the picker to pick the blocked fruits to be picked according to the uploaded positioning coordinates of the blocked fruits to be picked.

[0031] Preferably, it is characterized in that the crop microwave scattering model proposed in step S1 based on the first-order solution of the radiation transfer equation is a basic model for solving the backscattering of crops, and the scattering coefficient equation of the fruit is derived as follows:

[0032] τ 2 =exp[-2×B×m veg ×sec(θ)];

[0033]

[0034]

[0035] Where: is the total backscattering coefficient; p and q are polarization modes; is the backscattering coefficient of the stubble layer; is the backscattering coefficient of the obstruction; τ is the attenuation factor; m veg represents the water content of fruit per unit volume; θ is the incident angle; B is the empirical coefficient, and B is obtained by nonlinear least squares fitting. is the scattering term of the interaction between occlusion and corn stubble; f residue is the occlusion coverage; NDRI min NDRI max They are the NRDI values ​​for completely unobstructed and completely obstructed situations, respectively.

[0036] Preferably, the AIEM-oh model in step S1 estimates the microwave backscattering coefficient, and the specific estimation formula is as follows:

[0037]

[0038] Among them, k is the wave number of the radar wave, k is related to the frequency of the radar wave, and the parameter k remains unchanged in the crown scattering analysis;

[0039] β is the incident angle. For close-range horizontal imaging, the radar wave is almost parallel to the plane of the tree canopy;

[0040] s is the crown structural feature representing the size, shape and arrangement of leaves in the crown scattering scenario;

[0041] W is the surface autocorrelation roughness spectrum;

[0042] is the backscattering coefficient under polarization states p and q, that is, the scattered signal caused by occluding components such as leaves and other tree canopies.

[0043] Preferably, step S21 includes:

[0044] S211, millimeter wave radar data acquisition, pre-generation of multiple SAR images;

[0045] First, the millimeter-wave radar collects data and generates images. The potential diffusion model is selected and fine-tuned to make the potential diffusion model generate higher-quality images. Based on the characteristics of SAR images, multiple SAR images are pre-generated. SAR images are millimeter-wave radar images.

[0046] S212, expand SAR image;

[0047] Then the SAR image is expanded and the parameters of the potential diffusion model are adjusted to make the potential diffusion model better adapt to the characteristics of the SAR image, thereby generating another part of high-quality image, and then the two parts are imaged to generate a data set;

[0048] S213, binocular vision sensor image acquisition, pre-generation of binocular vision image;

[0049] Specifically, the binocular vision sensor collects images and uploads the collected images to the control system, and the control system generates a binocular vision image.

[0050] Preferably, step S22 includes:

[0051] In SAR image augmentation, the SAR image can be fine-tuned by latent diffusion models (LDMs) to adapt to the special properties of SAR images.

[0052] Step S23 includes:

[0053] By setting positive and negative prompt words, the required target SAR image is obtained, guiding the latent diffusion model to generate SAR images that better meet the target detection requirements;

[0054] Step S24 includes:

[0055] Use traditional RGB images to assist in the annotation of SAR images;

[0056] All RGB images are annotated auxiliaryly, and the annotated RGB images generate an annotation dataset after the annotation is completed.

[0057] Preferably, step S25 includes:

[0058] Verify the labeled data set, test the data set using the target detection algorithm, generate test results, evaluate the test results, and manually check the imaging quality and annotation accuracy of the test SAR images during the test process;

[0059] Step S26 includes:

[0060] According to the test results and feedback of target detection in S25, the potential diffusion model is iteratively optimized. The iterative optimization includes adjusting the model parameters of the potential diffusion model, optimizing the training data set, or adjusting the settings of positive and negative prompt words to improve the quality of the generated SAR image and the accuracy of target detection.

[0061] Preferably, the Transformer structure in step S31 includes:

[0062] 1) Basic feature extraction layer:

[0063] Use convolutional layers to extract preliminary feature representations from the input image;

[0064] For example, a convolutional layer consists of multiple convolutional blocks, each of which includes a convolutional layer, a normalization layer, and an activation function;

[0065] Design multiple resolution streams, each stream represents features at different scales;

[0066] 2) Multi-scale feature processing:

[0067] For each scale stream, convolution kernels of different sizes are used to capture scale-specific features;

[0068] Use skip connections to combine low-level features with high-level features;

[0069] 3) Multi-scale Transformer layer:

[0070] For each feature scale, a Transformer encoder is set up, including a self-attention layer and a feed-forward network to capture long-range dependencies within the features;

[0071] The output of each scale is processed by a self-attention layer to fuse the global information from within the same scale;

[0072] 4) Fusion module:

[0073] Design a fusion module that can accept the output of Transformer encoders of different scales;

[0074] A dynamic weight mechanism is used to calculate the fusion weights of features at different scales;

[0075] The dynamic weight mechanism is based on attention;

[0076] 5) Output layer:

[0077] The fusion result of the fusion module is output, and the comprehensive features obtained from the fusion module are used for the final task target detection.

[0078] Preferably, the dynamic attention module in step S32 includes:

[0079] 1) Feature extraction:

[0080] The dynamic attention module receives feature maps from different scales;

[0081] In the multi-scale structure, these feature maps come from different levels of convolutional networks or previous Transformer layers;

[0082] 2) Content perception and weight generation:

[0083] Learn a weight generator for each feature map of different scales;

[0084] The weight generator can be a small neural network, and its purpose is to dynamically generate the weight corresponding to the feature map according to the content of the feature map;

[0085] The weight generator evaluates the importance of each pixel or region in the feature map and outputs a weight map whose spatial dimensions match the input feature map;

[0086] 3) Adaptive feature weighting:

[0087] Use the generated weight map to perform element-by-element multiplication with the feature map of the corresponding scale to strengthen or weaken the features of a specific area;

[0088] Each position in the feature map will receive different weights according to the importance of its content, so that the area with more information will be highlighted;

[0089] 4) Inter-scale fusion:

[0090] Fuse weighted feature maps of different scales;

[0091] Fusion can be done by addition or more complex operations. More complex operations can use fully connected layers.

[0092] During the fusion process, additional dynamic weight learning can be introduced to balance the contribution of feature maps of different scales;

[0093] 5) Attention modulation:

[0094] Applying a standard self-attention mechanism on the fused features allows the dynamic attention module to capture long-range dependencies;

[0095] This step is implemented using a multi-head attention mechanism, where each attention mechanism processes different representations of different subspaces;

[0096] 6) Output layer:

[0097] The dynamically modulated feature map can be directly used for the next task-related processing.

[0098] Preferably, the mask pre-training process in step S33 includes:

[0099] 1) Data preparation;

[0100] Select training images: Select images from the dataset for the object detection task;

[0101] Target region recognition: Use existing target detection annotations to identify target regions in images. The target detection standard uses bounding box annotations.

[0102] 2) Generation of weighted masks;

[0103] Calculate mask probabilities: assign a mask probability to each pixel or small region in the image, based on whether the pixel or region contains a significant part of the object;

[0104] Methods for calculating mask probabilities;

[0105] Feature Importance Assessment:

[0106] For each pixel or predefined small area, evaluate its importance in object detection based on the bounding box annotation of the object;

[0107] Calculate the distance of each pixel relative to the center of the nearest bounding box. The smaller the distance, the higher the importance of the pixel.

[0108] Mask probability calculation:

[0109] Define a function f(d), where d is the distance from the pixel to the center of the bounding box. The output of function f is the mask probability of the pixel, and it should satisfy: the smaller the distance, the higher the mask probability;

[0110] Use the following exponential decay function:

[0111] f(d)=e -λd ;

[0112] Among them, λ is the attenuation coefficient, which is used to adjust the rate at which the mask probability changes with distance;

[0113] Apply mask threshold:

[0114] Generate a random number matrix of the same size as the image, with each element value between 0 and 1;

[0115] Compare this random number matrix with the calculated mask probability matrix. If the random number is less than or equal to the mask probability of the corresponding position, the position is masked.

[0116] Mask Application:

[0117] Based on the calculated probability, a mask is randomly generated and applied to the image. This is achieved by generating a matrix of random numbers and comparing them with the mask probability of each region. If the random number is lower than the mask probability of a region, the region is masked.

[0118] 3) Pre-training model construction;

[0119] Model structure:

[0120] Use a multi-scale Transformer network with a dynamic attention module as the base architecture;

[0121] Modify input processing:

[0122] Adjust the pre-trained model to accept masked images as input;

[0123] 4) Training objectives and loss functions;

[0124] a) Autoencoder structure:

[0125] Use an autoencoder structure for mask pre-training, which consists of an encoder and a decoder;

[0126] Encoder: responsible for extracting the features of the mask image;

[0127] Decoder: Responsible for reconstructing the original part of the image based on these features, that is, the unmasked part;

[0128] b) Mask application:

[0129] Randomly apply masks to images in the training dataset by randomly selecting image regions and setting their pixel values ​​to zero or the average value;

[0130] c) Reconstruction goals:

[0131] Train the autoencoder to reconstruct the masked image region, with the goal of making the reconstructed image close to the original unmasked image;

[0132] d) Loss function;

[0133] Use the mean square error calculation formula, the calculation formula is as follows;

[0134]

[0135] Among them, X original is the original image, X reconstructed is the reconstructed image and n is the number of pixels.

[0136] In summary, the present invention includes the following beneficial technical effects:

[0137] 1. The present invention fuses the SAR image collected by the millimeter-wave radar with the binocular vision image collected by the binocular vision sensor, thereby identifying and locating the fruits blocked by leaves, and generating a stereo image and positioning coordinates, so that the control system can control the picking robot to pick the blocked fruits. The fruit recognition method can not only identify the blocked fruits, but also use the picking robot to automatically pick the blocked fruits, replacing manual picking, which not only improves the picking efficiency but also reduces the labor intensity of the pickers.

[0138] 2. The present invention improves the accuracy of millimeter-wave radar in identifying obscured fruits by establishing a scattering characteristic model of fruit trees and using the AIEM-Oh model to correct the backscattering coefficient of the obstruction, and also facilitates the millimeter-wave radar to generate higher quality SAR images.

[0139] 3. The present invention sets positive and negative prompt words, labels, verifies and evaluates SAR images, and then iteratively optimizes the latent diffusion model, thereby improving the generation quality of SAR images by the latent diffusion model, and also making the SAR images generated by the latent diffusion model higher in quality and more accurate in positioning.

[0140] 4. The present invention performs multi-scale fusion of SAR images, adjusts weights of the dynamic attention module and performs mask pre-training. The dynamic attention module can balance the contribution of feature maps of different scales, thereby improving the quality of the fused SAR image and improving the recognition ability of the latent diffusion model for targets under occlusion conditions, thereby more accurately identifying the occluded fruit and more accurately positioning it. BRIEF DESCRIPTION OF THE DRAWINGS

[0141] Figure 1 It is a SAR original signal diagram in step S22 in the embodiment of the present invention.

[0142] Figure 2 It is a SAR generation signal diagram in step S22 in an embodiment of the present invention.

[0143] Figure 3 It is a SAR test signal diagram in step S22 in an embodiment of the present invention. DETAILED DESCRIPTION

[0144] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0145] The embodiment of the invention discloses a fruit recognition method for a picking robot based on millimeter wave radar.

[0146] The fruit recognition method of the picking robot based on millimeter wave radar specifically includes the following steps:

[0147] S1. Build a MIMO SAR imaging system and install a millimeter-wave radar and binocular vision sensor at the end of the robotic arm;

[0148] Millimeter-wave radar and binocular vision sensors image the target object and preliminarily analyze the imaging characteristics.

[0149] A fruit tree scattering characteristic model is established, and the crop microwave scattering model proposed based on the first-order solution of the radiation transfer equation is used to derive the residual scattering of the fruit. Then, the backscattering coefficient of the obstruction is corrected using the AIEM-Oh model:

[0150] Specifically, the crop microwave scattering model proposed based on the first-order solution of the radiation transfer equation is the basic model for solving the backscattering of crops, and the scattering coefficient equation of the fruit is derived as follows:

[0151] τ 2 =exp[-2×B×m veg ×sec(θ)];

[0152]

[0153]

[0154] Where: is the total backscattering coefficient; p and q are polarization modes; is the backscattering coefficient of the stubble layer; is the backscattering coefficient of the obstruction; τ is the attenuation factor; m veg represents the water content of fruit per unit volume; θ is the incident angle; B is the empirical coefficient, and B is obtained by nonlinear least squares fitting. is the scattering term of the interaction between occlusion and corn stubble; f residue is the occlusion coverage; NDRI min NDRI max They are the NRDI values ​​for completely unobstructed and completely obstructed conditions, respectively;

[0155] The AIEM-oh model is used to estimate the microwave backscattering coefficient. The specific estimation formula is as follows:

[0156]

[0157] Among them, k is the wave number of the radar wave, k is related to the frequency of the radar wave, and the parameter k remains unchanged in the crown scattering analysis.

[0158] β is the incident angle. For close-range horizontal imaging, the radar wave is almost parallel to the plane of the tree canopy. In this case, the incident angle may have a smaller effect on the contribution of the scattered signal.

[0159] In the context of canopy scattering, s may represent the canopy structural characteristics such as the size, shape, and arrangement of leaves.

[0160] W is the surface autocorrelation roughness spectrum.

[0161] is the backscattering coefficient under polarization states p and q, that is, the scattered signal caused by occluding components such as leaves and other tree canopies.

[0162] S2. Build a Stable Diffusion model, which specifically includes the following steps:

[0163] S21. Model selection and adaptability:

[0164] S211, millimeter wave radar data acquisition, pre-generation of multiple SAR images;

[0165] First, the millimeter-wave radar collects data and generates images. The potential diffusion model is selected and fine-tuned to generate higher-quality images. Based on the characteristics of SAR images, such as texture and contrast, multiple SAR images are pre-generated. SAR images are millimeter-wave radar images.

[0166] S212, expand SAR image;

[0167] Then the SAR image is expanded. Specifically, the various parameters of the potential diffusion model are adjusted to make the potential diffusion model better adapt to the characteristics of the SAR image, so as to generate another part of high-quality image, and then the two parts are imaged to generate a data set;

[0168] S213, binocular vision sensor image acquisition, pre-generation of binocular vision image;

[0169] Specifically, the binocular vision sensor collects images and uploads the collected images to the control system, and the control system generates a binocular vision image;

[0170] S22. Application of Latent Diffusion Models (LDMs):

[0171] In SAR image augmentation, latent diffusion models (LDMs) can be used to fine-tune SAR images to adapt to the special properties of SAR images, such as SAR-specific texture patterns and noise patterns. Figures 1 to 3 ,in, Figure 1 is the original SAR signal, Figure 2 Generate a signal for SAR, Figure 3 It is the SAR test signal;

[0172] S23, set positive and negative prompt words to obtain the required target SAR image:

[0173] By setting positive and negative cue words, the desired target SAR image can be obtained. Specifically, the latent diffusion model can be guided to generate SAR images that better meet the needs of target detection. For example, positive cue words may include "clear" and "high contrast", while negative cue words may be "blurry" and "noisy". These cue words help generate images that are more suitable for target detection.

[0174] S24, RGB detection auxiliary annotation:

[0175] Use traditional RGB images to assist in the annotation of SAR images.

[0176] Perform auxiliary annotation on all RGB images, and generate annotated datasets for the annotated RGB images;

[0177] Since SAR images are usually difficult to interpret intuitively, auxiliary annotation with the help of RGB images can improve the accuracy and efficiency of annotation. This auxiliary annotation process can be automated through machine learning algorithms to improve the consistency and scalability of annotation.

[0178] S25. Verification and evaluation of labeled datasets:

[0179] The labeled data set is verified, the data set is tested using the target detection algorithm, and the test results are generated and evaluated. In addition, during the testing process, manual inspection tests should also be considered to ensure the imaging quality of the SAR images and the accuracy of the annotations.

[0180] The verification and evaluation of labeled datasets are mainly to verify and evaluate their applicability in practical applications.

[0181] S26. Iterative optimization of potential diffusion model:

[0182] According to the test results and feedback of the target detection in S25, the potential diffusion model is iteratively optimized, including adjusting the model parameters of the potential diffusion model, optimizing the training data set, or adjusting the setting of positive and negative prompt words to improve the quality of the generated SAR image and the accuracy of target detection.

[0183] S3, SAR image fusion optimization;

[0184] Specifically, the multi-scale fusion guided by modeling of the masked image includes the following steps:

[0185] S31. Improved multi-scale fusion technology:

[0186] Based on the traditional feature pyramid network (FPN), a new multi-scale Transformer structure is introduced, which is a structure that can effectively capture the long-distance dependencies between different regions of the image. This strategy uses the dynamic weight allocation mechanism in deep learning to dynamically adjust the image fusion weight according to the feature information of different scales. This means that features with higher information content will be able to obtain more weight under occlusion conditions.

[0187] A Transformer-based multi-scale network structure consists of the following components:

[0188] 1) Basic feature extraction layer:

[0189] Use convolutional layers to extract preliminary feature representations from the input image.

[0190] For example, a convolutional layer consists of multiple convolutional blocks, each of which includes a convolutional layer, a normalization layer, and an activation function;

[0191] Design multiple resolution streams, each stream represents features at different scales, for example:

[0192] Stream1,Stream1,Stream2,...,StreamN.

[0193] 2) Multi-scale feature processing:

[0194] For each scale stream, convolution kernels of different sizes are used to capture scale-specific features.

[0195] Skip connections are used to combine low-level features with high-level features.

[0196] 3) Multi-scale Transformer layer:

[0197] For each feature scale, a Transformer encoder is set up, including self-attention layers and feed-forward networks, to capture long-range dependencies within features.

[0198] The output of each scale is processed by a self-attention layer to fuse the global information from within the same scale.

[0199] 4) Fusion module:

[0200] Design a fusion module that can accept the output of Transformer encoders of different scales.

[0201] A dynamic weight mechanism is used to calculate the fusion weights of features at different scales.

[0202] For example, the dynamic weight mechanism can be based on attention;

[0203] 5) Output layer:

[0204] The fusion result of the fusion module is output, and the comprehensive features obtained from the fusion module are used for the final task target detection.

[0205] S32, Dynamic Attention Module:

[0206] Design a dynamic attention module that adjusts the weights by learning the correlations between different features.

[0207] The dynamic attention module is more suitable for dealing with the problem of fruit being blocked by leaves because it can identify which features are most critical for target recognition and adjust their importance accordingly. Specifically, the features of the dynamic attention module are as follows:

[0208] 1) Feature extraction:

[0209] The dynamic attention module receives feature maps from different scales;

[0210] In the multi-scale structure, these feature maps come from different levels of convolutional networks or previous Transformer layers.

[0211] 2) Content perception and weight generation:

[0212] A weight generator is learned separately for each feature map of different scales.

[0213] The weight generator can be a small neural network, and its purpose is to dynamically generate the weight corresponding to the feature map according to the content of the feature map.

[0214] The weight generator evaluates the importance of each pixel or region in the feature map and outputs a weight map whose spatial dimensions match the input feature map.

[0215] 3) Adaptive feature weighting:

[0216] The generated weight map is used to perform element-by-element multiplication with the feature map of the corresponding scale to strengthen or weaken the features of a specific area.

[0217] Each position in the feature map will receive different weights according to the importance of its content, so that areas with more information will be highlighted.

[0218] 4) Inter-scale fusion:

[0219] The weighted feature maps of different scales are fused.

[0220] Fusion can be done by addition or more complex operations. More complex operations can use fully connected layers.

[0221] During the fusion process, additional dynamic weight learning can be introduced to balance the contribution of feature maps of different scales.

[0222] 5) Attention modulation:

[0223] Applying a standard self-attention mechanism on the fused features allows the dynamic attention module to capture long-range dependencies.

[0224] This step can be achieved using a multi-head attention mechanism, where each head processes a different representation of a different subspace.

[0225] 6) Output layer:

[0226] The dynamically modulated feature map can be directly used for the next task-related processing, such as the decision layer for detection tasks.

[0227] S33, mask pre-training process:

[0228] To improve the latent diffusion model’s ability to detect masked regions, we employ a mask pre-training process, in which the latent diffusion model is trained to recognize regions in an image that are masked by random masks and attempts to reconstruct the information of these masked regions.

[0229] This training helps the latent diffusion model better recognize and handle occluded objects when faced with real-world occlusion situations.

[0230] In particular, in mask pre-training, we use a weighted mask sampling strategy. This means that we assign higher mask probabilities to image patches that contain more target information. This strategy allows the model diffusion model to pay more attention to those information-rich areas during training, thereby improving the ability to recognize targets under occlusion conditions.

[0231] Specifically, the mask pre-training process is as follows:

[0232] 1) Data preparation;

[0233] Select training images: Select images from the dataset for the object detection task.

[0234] Target region recognition: Use existing target detection annotations to identify target regions in the image. The target detection criterion can be a bounding box annotation.

[0235] 2) Generation of weighted masks;

[0236] Calculate mask probability: Assign a mask probability to each pixel or small region in the image. The mask probability is based on whether the pixel or region contains a significant part of the object.

[0237] Methods for calculating mask probabilities;

[0238] Feature Importance Assessment:

[0239] For each pixel or a predefined small region, evaluate its importance in object detection. This can be done based on the bounding box annotation of the object.

[0240] Specifically, the distance of each pixel relative to the center of the nearest bounding box can be calculated, and the smaller the distance, the higher the importance of the pixel.

[0241] Mask probability calculation:

[0242] Define a function f(d), where d is the distance from the pixel to the center of the bounding box. The output of function f is the mask probability of the pixel, and it should satisfy: the smaller the distance, the higher the mask probability.

[0243] For example, an exponential decay function like this could be used:

[0244] f(d)=e -λd ;

[0245] Among them, λ is the attenuation coefficient, which is used to adjust the rate at which the mask probability changes with distance.

[0246] Apply mask threshold:

[0247] Generate a random number matrix of the same size as the image, with each element having a value between 0 and 1.

[0248] This random number matrix is ​​compared with the calculated mask probability matrix. If the random number is less than or equal to the mask probability of the corresponding position, then the position is masked.

[0249] Mask Application:

[0250] Based on the calculated probabilities, a mask is randomly generated and applied to the image. This can be done by generating a matrix of random numbers and comparing it with the mask probability for each region. If the random number is lower than the mask probability for a region, the region is masked.

[0251] 3) Pre-training model construction;

[0252] Model structure:

[0253] A multi-scale Transformer network with a dynamic attention module is used as the base architecture.

[0254] Modify input processing:

[0255] Adjust the pre-trained model to accept masked images as input.

[0256] 4) Training objectives and loss functions;

[0257] a) Autoencoder structure:

[0258] Use the autoencoder structure for mask pre-training. The autoencoder consists of an encoder and a decoder.

[0259] Encoder: responsible for extracting the features of the mask image.

[0260] Decoder: Responsible for reconstructing the original part of the image based on these features, that is, the unmasked part;

[0261] b) Mask application:

[0262] Randomly apply a mask to the images in the training dataset. This can be done by randomly selecting regions of the image and setting their pixel values ​​to zero or the average.

[0263] c) Reconstruction goals:

[0264] The autoencoder is trained to reconstruct the masked image region, with the goal of making the reconstructed image as close as possible to the original unmasked image.

[0265] d) Loss function;

[0266] The mean square error (MSE) calculation formula is used. Specifically, the calculation formula is as follows:

[0267]

[0268] Among them, X original is the original image, X reconstructed is the reconstructed image and n is the number of pixels.

[0269] S4, generate a stereo image and output it;

[0270] Using the mature F-PointNet (Frustum PointNet, 2017) open source algorithm, after distinguishing between fruits and leaves, the SAR image and the binocular vision image are fused through edge feature recognition and pixel matrix fusion technology to generate a stereo image combining 3D point cloud and RGB, and automatically fit the shape of the fruit of the occluded part, so that the picking robot can identify and locate the occluded fruit, generate positioning coordinates, and then upload the positioning coordinates of the identified and located occluded fruit to be picked to the control system;

[0271] S5. The control system controls the picking head of the mechanical part of the picker to pick the blocked fruits to be picked according to the uploaded positioning coordinates of the blocked fruits to be picked.

[0272] The above are all preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Therefore, any equivalent changes made based on the structure, shape, and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A fruit recognition method for a picking robot based on millimeter wave radar, characterized in that: The following steps are involved: S1. Build a MIMO SAR imaging system and install a millimeter-wave radar and binocular vision sensor at the end of the robotic arm; Millimeter wave radar and binocular vision sensors image the target object and preliminarily analyze the imaging characteristics; A crop microwave scattering model is established based on the first-order solution of the radiation transfer equation, and the residual scattering of the fruit is derived using the crop microwave scattering model. Then, the backscattering coefficient of the obstruction is corrected using the AIEM-Oh model to obtain the corrected backscattering coefficient. S2. Build a Stable Diffusion model; S21. Model selection and adaptability; S211, millimeter wave radar data acquisition, pre-generation of multiple SAR images; First, the millimeter-wave radar collects data and generates images. The latent diffusion model is selected and fine-tuned to make the latent diffusion model generate high-quality images. Based on the characteristics of SAR images, multiple SAR images are pre-generated. SAR images are millimeter-wave radar images. S212, expand SAR image; Then the SAR image is expanded, and the parameters of the potential diffusion model are adjusted to adapt the potential diffusion model to the characteristics of the SAR image, thereby generating another part of the high-quality image, and then the two parts are imaged to generate a data set; S213, binocular vision sensor image acquisition, pre-generation of binocular vision image; Specifically, the binocular vision sensor collects images and uploads the collected images to the control system, and the control system generates a binocular vision image; S22. Application of latent diffusion models (LDMs); S23, setting positive and negative prompt words to obtain the required target SAR image; S24, RGB detection auxiliary annotation; S25. Verification and evaluation of annotated datasets; S26, iterative optimization of potential diffusion model; S3, SAR image fusion optimization; S31. Improved multi-scale fusion technology: Based on the traditional feature pyramid network, a new multi-scale Transformer structure is introduced. The dynamic weight allocation mechanism in deep learning is used to dynamically adjust the image fusion weight according to the feature information of different scales, so that features with higher information content under occlusion conditions will be able to obtain more weight; S32, Dynamic Attention Module: Design a dynamic attention module that adjusts weights by learning the correlations between different features; S33, mask pre-training process: Using the mask pre-training process, the latent diffusion model is trained to identify the areas in the image that are masked by random masks and try to reconstruct the information of these masked areas; S4, generate and output a stereoscopic image; Using the F-PointNet open source algorithm, after distinguishing between fruits and leaves, the SAR image and the binocular vision image are fused through edge feature recognition and pixel matrix fusion technology to generate a stereo image combining 3D point cloud and RGB, and automatically fit the shape of the fruit of the occluded part, so that the picking robot can identify and locate the occluded fruit, generate positioning coordinates, and then upload the positioning coordinates of the identified and located occluded fruit to the control system; S5. The control system controls the picking head of the mechanical part of the picker to pick the blocked fruits to be picked according to the uploaded positioning coordinates of the blocked fruits to be picked.

2. The fruit recognition method of a picking robot based on millimeter wave radar according to claim 1, characterized in that: The crop microwave scattering model proposed in step S1 based on the first-order solution of the radiation transfer equation is the basic model for solving the backscattering of crops, and the scattering coefficient equation of the fruit is derived as follows: t 2 =exp[-2×B×m veg ×sec(θ)]; Where: is the total backscattering coefficient; p and q are polarization modes; is the backscattering coefficient of the stubble layer; is the backscattering coefficient of the obstruction; τ is the attenuation factor; m veg Indicates the water content of fruit per unit volume; θ is the incident angle; B is the empirical coefficient, and B is obtained by nonlinear least squares fitting. is the scattering term of the interaction between occlusion and corn stubble; f residue is the occlusion coverage; NDRI min NDRI max They are the NRDI values ​​for completely unobstructed and completely obstructed situations, respectively.

3. The fruit recognition method of a picking robot based on millimeter wave radar according to claim 1, characterized in that: The AIEM-oh model in step S1 estimates the microwave backscattering coefficient. The specific estimation formula is as follows: Among them, k is the wave number of the radar wave, k is related to the frequency of the radar wave, and the parameter k remains unchanged in the crown scattering analysis; β is the angle of incidence; s is the crown structural feature representing the size, shape and arrangement of leaves in the crown scattering scenario; W is the surface autocorrelation roughness spectrum; is the backscattering coefficient under polarization states p and q, that is, the scattered signal caused by the shading components of leaves and canopy.

4. The fruit recognition method of a picking robot based on millimeter wave radar according to claim 1, characterized in that: Step S22 includes: In SAR image augmentation, latent diffusion models (LDMs) are used to fine-tune SAR images to adapt to the properties of SAR images. Step S23 includes: By setting positive and negative prompt words, the required target SAR image is obtained, and the latent diffusion model is guided to generate SAR images that meet the target detection requirements; Step S24 includes: Use traditional RGB images to assist in the annotation of SAR images; All RGB images are annotated auxiliaryly, and the annotated RGB images generate an annotation dataset after the annotation is completed.

5. The fruit recognition method of a picking robot based on millimeter wave radar according to claim 4, characterized in that: Step S25 includes: Verify the labeled data set, test the data set using the target detection algorithm, generate test results, evaluate the test results, and manually check the imaging quality and annotation accuracy of the test SAR images during the test process; Step S26 includes: According to the test results and feedback of target detection in S25, the potential diffusion model is iteratively optimized. The iterative optimization includes adjusting the model parameters of the potential diffusion model, optimizing the training data set, or adjusting the settings of positive and negative prompt words to improve the quality of the generated SAR image and the accuracy of target detection.

6. The fruit recognition method of a picking robot based on millimeter wave radar according to claim 1, characterized in that: The Transformer structure in step S31 includes: 1) Basic feature extraction layer: Use convolutional layers to extract preliminary feature representations from the input image; The convolution layer consists of multiple convolution blocks, each of which includes a convolution layer, a normalization layer, and an activation function; Design multiple resolution streams, each stream represents features at different scales; 2) Multi-scale feature processing: For each scale stream, convolution kernels of different sizes are used to capture scale-specific features; Use skip connections to combine low-level features with high-level features; 3) Multi-scale Transformer layer: For each feature scale, a Transformer encoder is set up, including a self-attention layer and a feed-forward network to capture long-range dependencies within the features; The output of each scale is processed by a self-attention layer to fuse the global information from within the same scale; 4) Fusion module: Design a fusion module that can accept the output of Transformer encoders of different scales; A dynamic weight mechanism is used to calculate the fusion weights of features at different scales; The dynamic weight mechanism is based on attention; 5) Output layer: The fusion result of the fusion module is output, and the comprehensive features obtained from the fusion module are used for the final task target detection.

7. The fruit recognition method of a picking robot based on millimeter wave radar according to claim 1, characterized in that: The dynamic attention module in step S32 includes: 1) Feature extraction: The dynamic attention module receives feature maps from different scales; In the multi-scale structure, these feature maps come from different levels of convolutional networks or previous Transformer layers; 2) Content perception and weight generation: Learn a weight generator for each feature map of different scales; The weight generator is a small neural network whose purpose is to dynamically generate the weights corresponding to the feature map based on the content of the feature map; The weight generator evaluates the importance of each pixel or region in the feature map and outputs a weight map whose spatial dimensions match the input feature map; 3) Adaptive feature weighting: Use the generated weight map to perform element-by-element multiplication with the feature map of the corresponding scale to strengthen or weaken the features of a specific area; Each position in the feature map will receive different weights according to the importance of its content, so that areas with high information content will be highlighted; 4) Inter-scale fusion: Fuse weighted feature maps of different scales; Fusion is done through addition or complex operations, and complex operations use fully connected layers; In the fusion process, additional dynamic weight learning is introduced to balance the contribution of feature maps of different scales; 5) Attention modulation: Applying a standard self-attention mechanism on the fused features allows the dynamic attention module to capture long-range dependencies; This step is implemented using a multi-head attention mechanism, where each attention mechanism processes different representations of different subspaces; 6) Output layer: The dynamically modulated feature map is directly used for the next task-related processing.

8. The method for fruit recognition by a picking robot based on millimeter wave radar according to claim 1, characterized in that: The mask pre-training process in step S33 includes: 1) Data preparation; Select training images: Select images from the dataset for the object detection task; Target region recognition: Use existing target detection annotations to identify target regions in images. The target detection standard uses bounding box annotations. 2) Generation of weighted masks; Calculate mask probabilities: assign a mask probability to each pixel or small region in the image, based on whether the pixel or region contains a significant part of the object; Methods for calculating mask probabilities; Feature Importance Assessment: For each pixel or predefined small area, evaluate its importance in object detection based on the bounding box annotation of the object; Calculate the distance of each pixel relative to the center of the nearest bounding box. The smaller the distance, the higher the importance of the pixel. Mask probability calculation: Define a function f(d), where d is the distance from the pixel to the center of the bounding box. The output of function f is the mask probability of the pixel, and it should satisfy: the smaller the distance, the higher the mask probability; Use the following exponential decay function: f(d)=e -λd ; Among them, λ is the attenuation coefficient, which is used to adjust the rate at which the mask probability changes with distance; Apply mask threshold: Generate a random number matrix of the same size as the image, with each element value between 0 and 1; Compare this random number matrix with the calculated mask probability matrix. If the random number is less than or equal to the mask probability of the corresponding position, the position is masked. Mask Application: Based on the calculated probability, a mask is randomly generated and applied to the image. This is achieved by generating a matrix of random numbers and comparing them with the mask probability of each region. If the random number is lower than the mask probability of a region, the region is masked. 3) Pre-training model construction; Model structure: Use a multi-scale Transformer network with a dynamic attention module as the base architecture; Modify input processing: Adjust the pre-trained model to accept masked images as input; 4) Training objectives and loss functions; a) Autoencoder structure: Use an autoencoder structure for mask pre-training, which consists of an encoder and a decoder; Encoder: responsible for extracting the features of the mask image; Decoder: Responsible for reconstructing the original part of the image based on these features, that is, the unmasked part; b) Mask application: Randomly apply masks to images in the training dataset by randomly selecting image regions and setting their pixel values ​​to zero or the average value; c) Reconstruction goals: Train the autoencoder to reconstruct the masked image region, with the goal of making the reconstructed image close to the original unmasked image; d) Loss function; Use the mean square error calculation formula, the calculation formula is as follows; Among them, X original is the original image, X reconstructed is the reconstructed image and n is the number of pixels.

Images