Near-color system fruit identification and positioning method based on deep learning and binocular vision

Abstract

The invention provides a near-color system fruit identification and positioning method based on deep learning and binocular vision. The method comprises the following steps: collecting near-color system fruit patterns as an original data set; labeling the near-color fruits in the data set by using a labeling tool; building an improved near-color system fruit identification model; training the improved near-color system fruit identification model; using the trained model to identify a near-color system green fruit, and obtaining a two-dimensional plane coordinate of a fruit center point under a pixel coordinate; in combination with depth information provided by a binocular vision system, two-dimensional pixel coordinates of a fruit center point are converted into three-dimensional space coordinates, and the fruit width is calculated according to parallax information. And finally, the calculated three-dimensional coordinates and width information of the fruits are transmitted to the picking robot to guide the picking robot to perform picking operation, so that rapid identification and positioning of the near-color fruits are realized, size information is provided for subsequent fruit grading treatment, and development of an agricultural automatic picking technology is greatly promoted.

Description

Technical Field

[0001] The present invention belongs to the technical field of detection and visual positioning of fruits of similar colors, and specifically relates to a method for detecting and positioning fruits of similar colors based on deep learning and binocular vision. Background Art

[0002] In actual orchard picking, green apples are more challenging to accurately detect and locate than red apples due to their similar color to the surrounding environment, as well as interference from factors such as lighting variations, fruit growth status, and overlapping fruit. To address this issue, combining computer vision and machine learning technologies can effectively improve the accuracy and robustness of green apple detection. The development of this technology will not only help reduce labor costs and promote agricultural automation, but also enhance the market competitiveness of green apples and promote the sustainable development of the agricultural industry.

[0003] In recent years, many solutions have emerged for detecting and locating similar-colored fruit based on vision and object detection technologies. These solutions employ visual sensors, such as monocular, binocular, or depth cameras, combined with object detection technology to identify and locate fruit. However, current approaches to identifying and locating similar-colored fruit based on vision and object detection technologies have the following drawbacks:

[0004] Traditional target detection methods mainly use sliding windows and manual feature extraction, which suffer from problems such as a large amount of redundant calculations, slow operation speed, and poor robustness in complex environments, making it difficult to achieve satisfactory detection results.

[0005] The current mainstream target detection algorithms have problems such as high model complexity, large amount of computation and parameters, high requirements for hardware performance, and inability to simultaneously take into account both detection accuracy and real-time performance, which will result in the picking robot being unable to capture fruit information in a timely manner. Summary of the Invention

[0006] To address these issues, the present invention provides a method for detecting and locating similar-colored fruit based on deep learning and binocular vision. This method utilizes an improved similar-colored fruit detection model to identify similar-colored fruit in an orchard, and combines binocular vision positioning technology to calculate the fruit's three-dimensional coordinates and width. This method achieves rapid identification and location of similar-colored fruit in an orchard while maintaining high-precision detection.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: a method for detecting and locating fruits of similar color based on deep learning and binocular vision, comprising the following steps:

[0008] S1: Collect samples of green apples with similar colors in the orchard as the original image dataset;

[0009] S2: Use annotation tools to annotate the original image dataset and use the annotated images as the dataset;

[0010] S3: Build an improved green apple detection model with similar colors;

[0011] S4: Use the dataset to train the improved close-color green apple detection model;

[0012] S5: Use the trained close-color green apple detection model to identify close-color green apples and obtain the two-dimensional plane coordinates (u, v) of the fruit center point in the pixel coordinate system.

[0013] S6: Combine the depth information provided by the binocular camera to obtain the three-dimensional coordinates of the center point of the fruit and obtain the width information of the fruit.

[0014] Furthermore, the above S2 includes: using a labeling tool to label the green apple samples of similar colors in the original image dataset, using a minimum circumscribed rectangular frame to frame the position of the fruit in the image, setting the label category to greenapple, using the labeled images as a dataset, and dividing the dataset into a model training set and a validation set in proportion.

[0015] Furthermore, the above S3 includes:

[0016] S31: The backbone network of the original DETR network model is replaced with the ResNet18 feature extraction network, and some residual layers are replaced with deformable convolution (DCNv2), so that the model can better adapt to the changes of fruits at different scales and angles;

[0017] S32: Before the ResNet18 backbone network enters the Transformer, the extended spatial pyramid pooling model DSPP and the multi-scale residual aggregation module FRAM are integrated. A DSPP module is designed for each of the four downsampling layers of ResNet18. The feature information output by the four DSPP modules is aggregated and processed by the FRAM module. The processed feature information of the FRAM module is input into the Transformer queue. This helps reduce feature noise and minimize the loss of underlying features during the feature extraction process. The fusion of the two modules enhances the model's detection ability for objects of different scales and improves the accuracy of detecting fruits of similar colors.

[0018] S33: Introducing a multi-scale deformable multi-head attention mechanism in the Transformer part to improve the convergence speed of the overall network model. The formula of the multi-scale deformable multi-head attention mechanism is as follows:

[0019]

[0020] Where m represents the mth attention head in the multi-head attention mechanism, z q Input features for Query, Is the transfer matrix from input feature to Value, W m It is to perform a linear transformation on the result after attention is applied to the value to obtain the output results of different heads. is the input feature map set, x l represents the feature map of layer l, The normalized coordinates of each reference point q represent the relative position on the feature map, m represents the index of the attention head, l represents the input feature level, k represents the sampling point, and Δp mlqk and A mlqk They represent the sampling offset of the kth sampling point in the first feature layer and the attention weight of the mth attention head, respectively. Used to normalize coordinates Rescales the input feature map to layer l so that sampling and attention computation are performed at the correct feature scale.

[0021] Furthermore, the above S4 includes: using the model training set to train the improved close-color fruit detection model to obtain the close-color green apple detection model weights for detecting close-color green apples. The weight file is in .pth format, and the weights include close-color feature texture parameters, color sensitivity layer, surface feature parameters, shape encoder and attention mechanism feature information.

[0022] Furthermore, the above S6 includes:

[0023] S61: According to the binocular camera triangulation positioning principle, the depth information corresponding to the two-dimensional plane coordinates (u, v) of the fruit center point is obtained, that is, the distance Z between the fruit and the binocular camera C , the formula is as follows:

[0024]

[0025] Where f is the focal length of the camera; b is the baseline distance, that is, the distance between the projection centers of the two camera lenses; X L -X R is the disparity, obtained by pixel matching;

[0026] S62: Calculate the three-dimensional coordinates (X, V) of the fruit center point in the pixel coordinate system in the camera coordinate system through the conversion relationship between the pixel coordinate system and the camera coordinate system. C ,Y C ,Z C ), the conversion method between the pixel coordinate system and the camera coordinate system is as follows:

[0027]

[0028] Among them, f x 、fy , u0, v0 are the internal parameters of the binocular camera;

[0029] S63: The three-dimensional coordinates (X C ,Y C ,Z C ) is converted to the three-dimensional coordinates of the orchard coordinate system (X W ,Y W ,Z W ), the conversion method from the camera coordinate system to the orchard coordinate system is:

[0030]

[0031] Among them, R is a 3×3 matrix, which is the product of the rotation matrices in the X, Y, and Z directions. The rotation angle is provided in real time by the inertial unit, and T is a 3×1 translation matrix;

[0032] S64: Calculate the actual width of the fruit in the camera coordinate system based on the geometric distance between the center point of the fruit and the center point of the left border of the prediction box. According to the above two-dimensional to three-dimensional coordinate analysis, as long as the two-dimensional coordinates of the point are known, the corresponding three-dimensional coordinates in the camera coordinate system can be calculated by the above formula. The three-dimensional coordinates of the center point of the fruit in the camera coordinate system are (X C ,Y C ,Z C ), the three-dimensional coordinates corresponding to the midpoint in the left frame of the prediction box are The actual width of the fruit The calculation method is:

[0033]

[0034] Beneficial effects of the present invention:

[0035] The proposed method uses a ResNet18 network as its backbone feature extraction framework, replacing some residual layers with deformable convolutional neural networks (DCNv2) to better adapt to changes in fruit size and angle. Subsequently, the extended spatial pyramid pooling model (DSPP) and the multi-scale residual aggregation module (FRAM) are integrated to enhance the model's ability to detect objects of varying scales. Finally, the Transformer's attention mechanism is replaced to improve the convergence speed of the overall network model. Finally, binocular vision positioning technology is combined to calculate the three-dimensional coordinates of the center point of similarly colored fruits in an orchard environment and the fruit's width, enabling fruit localization. Compared with the existing technology, the present invention maintains the best balance between detection speed and detection accuracy. Under the premise of maintaining high-precision detection, it can realize the real-time and rapid recognition and three-dimensional positioning of similar-color fruits by the picking robot, and obtain the width information of the fruit at the same time, providing important guarantees for the subsequent fruit size classification. In addition, the present invention overcomes the problems of large parameter volume and calculation amount in the existing technology, has low performance requirements for hardware and takes up less memory, and is very suitable for deployment on embedded devices or mobile devices with limited computing power and memory size, thereby lowering the application threshold of deep learning and meeting the actual needs of deep learning in production practice activities. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a flow chart of a method for detecting and locating similar-colored fruits based on deep learning and binocular vision proposed in the present invention;

[0037] Figure 2 This is a schematic diagram of the network structure of the improved similar color fruit detection model proposed in the present invention;

[0038] Figure 3 This is a schematic diagram of the principle of distance measurement for fruits of similar color proposed in the present invention;

[0039] Figure 4 This is a schematic diagram of the conversion relationship between the pixel coordinate system and the orchard coordinate system proposed in the present invention. DETAILED DESCRIPTION

[0040] The present invention is further described below with reference to the accompanying drawings and specific embodiments. It should be pointed out that the technical solution and design principle of the present invention are described in detail below only with an optimized technical solution, but the protection scope of the present invention is not limited thereto.

[0041] The embodiments described are preferred implementations of the present invention, but the present invention is not limited to the above implementations. Any obvious improvements, substitutions or modifications that can be made by those skilled in the art without departing from the essence of the present invention are within the scope of protection of the present invention.

[0042] like Figure 1The present invention is a method for detecting and locating similar-colored fruits based on deep learning and binocular vision, comprising the following steps:

[0043] S1: Collect samples of green apples with similar colors in the orchard as the original image dataset;

[0044] S2: Use annotation tools to annotate the original image dataset and use the annotated images as the dataset;

[0045] As a preferred embodiment of the present invention, a labeling tool is used to label the green apple samples of similar colors in the original image dataset, a minimum circumscribed rectangular frame is used to frame the position of the fruit in the image, the label category is set to greenapple, the labeled images are used as the dataset, and the dataset is divided into a model training set and a validation set in proportion.

[0046] S3: Build an improved green apple detection model with similar colors, such as Figure 2 The figure shows the overall flow chart of the algorithm;

[0047] As a preferred embodiment of the present invention, the above S3 includes:

[0048] S31: The backbone network of the original DETR network model is replaced with the ResNet18 feature extraction network, and some residual layers are replaced with deformable convolution (DCNv2), so that the model can better adapt to the changes of fruits at different scales and angles;

[0049] S32: Before the ResNet18 backbone network enters the Transformer, the extended spatial pyramid pooling model DSPP and the multi-scale residual aggregation module FRAM are integrated. A DSPP module is designed for each of the four downsampling layers of ResNet18. The feature information output by the four DSPP modules is aggregated and processed by the FRAM module. The processed feature information of the FRAM module is input into the Transformer queue. This helps reduce feature noise and minimize the loss of underlying features during the feature extraction process. The fusion of the two modules enhances the model's detection ability for objects of different scales and improves the accuracy of detecting fruits of similar colors.

[0050] S33: Introducing a multi-scale deformable multi-head attention mechanism in the Transformer part to improve the convergence speed of the overall network model. The formula of the multi-scale deformable multi-head attention mechanism is as follows:

[0051]

[0052] Where m represents the mth attention head in the multi-head attention mechanism, z q Input features for Query, Is the transfer matrix from input feature to Value, W m It is to perform a linear transformation on the result after attention is applied to the value to obtain the output results of different heads. is the input feature map set, x l represents the feature map of layer l, The normalized coordinates of each reference point q represent the relative position on the feature map, m represents the index of the attention head, l represents the input feature level, k represents the sampling point, and Δp mlqk and A mlqk They represent the sampling offset of the kth sampling point in the first feature layer and the attention weight of the mth attention head, respectively, and the scalar attention weight A mlqk Normalized to satisfy This ensures that the sum of the attention weights is 1, which meets the requirements of the attention mechanism. The normalized coordinates are used in the formula Represents the clarity of the scale formula, where the normalized coordinates (0,0) and (1,1) represent the upper left and lower right corners of the image respectively, and the function Used to normalize coordinates The input feature map is rescaled to the lth layer to enable sampling and attention calculation at the correct feature scale. By optimizing the attention module of the Transformer framework, the convergence speed of the overall network model is significantly improved, thereby improving the efficiency of the robot in detecting green apples.

[0053] S4: Use the dataset to train the improved close-color green apple detection model.

[0054] As a preferred embodiment of the present invention, the above-mentioned S4 includes: using the model training set to train the improved close-color fruit detection model to obtain the close-color green apple detection model weights for detecting close-color green apples. The weight file is in .pth format, and the weights include close-color feature texture parameters, color sensitivity layer, surface feature parameters, shape encoder and attention mechanism feature information.

[0055] S5: Use the trained close-color green apple detection model to identify close-color green apples and obtain the two-dimensional plane coordinates (u, v) of the fruit center point in the pixel coordinate system.

[0056] S6: Combine the depth information provided by the binocular camera to obtain the three-dimensional coordinates of the center point of the fruit and obtain the width information of the fruit.

[0057] As a preferred embodiment of the present invention, the above S6 includes:

[0058] S61: According to the binocular camera triangulation positioning principle, the depth information corresponding to the two-dimensional plane coordinates (u, v) of the fruit center point is obtained, that is, the distance Z between the fruit and the binocular camera C ,like Figure 3 As shown, the formula is as follows:

[0059]

[0060] Where f is the focal length of the camera; b is the baseline distance, that is, the distance between the projection centers of the two camera lenses; X L -X R is the disparity, obtained by pixel matching;

[0061] S62: Calculate the three-dimensional coordinates (X, V) of the fruit center point in the pixel coordinate system in the camera coordinate system through the conversion relationship between the pixel coordinate system and the camera coordinate system. C ,Y C ,Z C ), the conversion method between the pixel coordinate system and the camera coordinate system is as follows:

[0062]

[0063] Among them, f x 、f y , u0, v0 are the internal parameters of the binocular camera;

[0064] S63: The three-dimensional coordinates (X C ,Y C ,Z C ) is converted to the three-dimensional coordinates of the orchard coordinate system (X W ,Y W ,Z W ), the conversion method from the camera coordinate system to the orchard coordinate system is:

[0065]

[0066] Among them, R is a 3×3 matrix, which is the product of the rotation matrices in the X, Y, and Z directions. The rotation angle is provided in real time by the inertial unit, and T is a 3×1 translation matrix;

[0067] S64: Calculate the actual width of the fruit in the camera coordinate system based on the geometric distance between the center point of the fruit and the center point of the left border of the prediction box, such as Figure 4 As shown, according to the above three-dimensional coordinate analysis, as long as the two-dimensional coordinates of the point are known, the three-dimensional coordinates of the point in the camera coordinate system can be calculated by the above formula. The three-dimensional coordinates of the fruit center point in the camera coordinate system are (X C ,Y C ,Z C), the three-dimensional coordinates corresponding to the midpoint in the left frame of the prediction box are The actual width of the fruit The calculation method is:

[0068]

Claims

1. A method for detecting and locating close-color fruits based on deep learning and binocular vision, characterized in that: The following steps are involved: S1: Collect samples of green apples with similar colors in the orchard as the original image dataset; S2: Use annotation tools to annotate the original image dataset and use the annotated images as the dataset; S3: Build an improved green apple detection model with similar colors; S4: Use the dataset to train the improved close-color green apple detection model; S5: Use the trained close-color green apple detection model to identify the close-color green apple and obtain the two-dimensional plane coordinates (u, v) of the fruit center in the pixel coordinate system; S6: Combine the depth information provided by the binocular camera to obtain the three-dimensional coordinates of the center point of the fruit and obtain the width information of the fruit.

2. The method for detecting and locating close-color fruits based on deep learning and binocular vision according to claim 1, wherein: The S2 includes: using a labeling tool to label the green apple samples of similar color in the original image dataset, using a minimum circumscribed rectangular frame to frame the position of the fruit in the image, setting the label category to greenapple, using the labeled images as a dataset, and dividing the dataset into a model training set and a validation set in proportion.

3. The method for detecting and locating close-color fruits based on deep learning and binocular vision according to claim 1, wherein: The S3 includes: S31: The backbone network of the original DETR network model is replaced with the ResNet18 feature extraction network, and some residual layers are replaced with deformable convolution DCNv2, so that the model can better adapt to the changes of fruits at different scales and angles; S32: Before the ResNet18 backbone network enters the Transformer, the extended spatial pyramid pooling model DSPP and the multi-scale residual aggregation module FRAM are integrated. A DSPP module is designed for each of the four downsampling layers of ResNet18. The feature information output by the four DSPP modules is aggregated and processed by the FRAM module. The processed feature information of the FRAM module is input into the Transformer queue. This helps reduce feature noise and minimize the loss of underlying features during the feature extraction process. The fusion of the two modules enhances the model's detection ability for objects of different scales and improves the accuracy of detecting fruits of similar colors. S33: Introducing a multi-scale deformable multi-head attention mechanism in the Transformer part to improve the convergence speed of the overall network model. The formula of the multi-scale deformable multi-head attention mechanism is as follows: Where m represents the mth attention head in the multi-head attention mechanism, z q Input features for Query, W, m Is the transfer matrix from input feature to Value, W m It is to perform a linear transformation on the result after attention is applied to the value to obtain the output results of different heads. is the input feature map set, x l represents the feature map of layer l, The normalized coordinates of each reference point q represent the relative position on the feature map, m represents the index of the attention head, l represents the input feature level, k represents the sampling point, and Δp mlqk and A mlqk They represent the sampling offset of the kth sampling point in the first feature layer and the attention weight of the mth attention head, respectively. Used to normalize coordinates Rescale the input feature map to layer l so that sampling and attention computation are performed at the correct feature scale.

4. The method for detecting and locating close-color fruits based on deep learning and binocular vision according to claim 1, wherein: The S4 includes: using the model training set to train the improved close-color fruit detection model to obtain the close-color green apple detection model weights for detecting close-color green apples. The weight file is in .pth format, and the weights include close-color feature texture parameters, color sensitivity layer, surface feature parameters, shape encoder and attention mechanism feature information.

5. The method for detecting and locating close-color fruits based on deep learning and binocular vision according to claim 1, wherein: The S6 includes: S61: According to the binocular camera triangulation positioning principle, the depth information corresponding to the two-dimensional plane coordinates (u, v) of the fruit center point is obtained, that is, the distance Z between the fruit and the binocular camera C , the formula is as follows: Where f is the focal length of the camera; b is the baseline distance, that is, the distance between the projection centers of the two camera lenses; X L -X R is the disparity, obtained by pixel matching; S62: Calculate the three-dimensional coordinates (X, V) of the fruit center point in the pixel coordinate system in the camera coordinate system through the conversion relationship between the pixel coordinate system and the camera coordinate system. C ,Y C ,Z C ), the conversion method between the pixel coordinate system and the camera coordinate system is as follows: Among them, f x 、f y , u0, v0 are the internal parameters of the binocular camera; S63: The three-dimensional coordinates (X C ,Y C ,Z C ) is converted to the three-dimensional coordinates of the orchard coordinate system (X W ,Y W ,Z W ), the conversion method from the camera coordinate system to the orchard coordinate system is: Among them, R is a 3×3 matrix, which is the product of the rotation matrices in the X, Y, and Z directions. The rotation angle is provided in real time by the inertial unit, and T is a 3×1 translation matrix; S64: Calculate the actual width of the fruit in the camera coordinate system based on the geometric distance between the center point of the fruit and the center point of the left border of the prediction box. From the above two-dimensional coordinate to three-dimensional coordinate analysis, as long as the two-dimensional coordinate point is known, the three-dimensional coordinate point corresponding to the camera coordinate system can be calculated by the above formula. The three-dimensional coordinate of the fruit center point in the camera coordinate system is (X C ,Y C ,Z C ), the three-dimensional coordinates corresponding to the midpoint in the left frame of the prediction box are The actual width of the fruit The calculation method is:

Images