A method and device for identifying and locating a point of picking of a mature fruit

Abstract

The application provides a mature fruit recognition and picking point positioning method and device, which comprises the following steps: acquiring a two-dimensional image containing target fruits; acquiring a trained target recognition model, and recognizing the fruits in the two-dimensional image and judging the maturity of the fruits based on the trained target recognition model, and meanwhile, recognizing and determining the picking points of the mature fruits; constructing a three-dimensional algorithm model, and obtaining the three-dimensional space coordinates of the fruit picking point image after coordinate conversion based on the three-dimensional algorithm model and the picking points of the mature fruits in the two-dimensional image. The application can improve the recognition of fruit picking points and the accurate determination of picking point coordinates through deep learning, and greatly improves the picking efficiency.

Description

Technical Field

[0001] This invention belongs to the field of intelligent harvesting and machine image processing technology, and particularly relates to a method and device for identifying mature fruits and locating harvesting points. Background Technology

[0002] Intelligent fruit harvesting can significantly improve harvesting efficiency, and the location of harvesting points is the foundation for realizing intelligent fruit harvesting. Currently, there are two main methods for research on fruit harvesting point location.

[0003] One approach is to use traditional segmentation algorithms, which identify fruits based on thresholding or by combining color information features and edge distance, and further determine the picking point of the fruit to locate the two-dimensional picking point. This method has low generalization ability and is highly dependent on lighting, shooting angle, and background. Its effect on accurate fruit identification and location is not ideal in natural backgrounds.

[0004] Another approach is based on deep learning algorithms, which use object detection algorithms to identify and locate the fruit, and combine this with skeleton extraction to determine the two-dimensional coordinates of the picking point. However, the object detection algorithms used are mostly horizontal bounding boxes, which are not conducive to the accurate identification and location of the fruit picking point. In particular, the subsequent skeleton extraction to determine the picking point greatly increases the task time and has low accuracy.

[0005] Therefore, the present invention provides a method and apparatus for identifying mature fruits and locating picking points to solve the problem of low accuracy in picking point identification and positioning in existing technologies. Summary of the Invention

[0006] In view of this, in order to solve the problem of low accuracy in picking point identification and positioning of existing technical solutions, it is necessary to provide a method and device for identifying mature fruits and locating picking points to improve the accuracy of fruit picking point identification and positioning.

[0007] To achieve the above objectives, the present invention provides the following technical solutions. A method for identifying mature fruit and locating picking points includes: Obtain a two-dimensional image containing the target fruit; A fully trained target recognition model is obtained, and the fruits in the two-dimensional image are identified and their ripeness is judged based on the fully trained target recognition model. At the same time, the picking point of the ripe fruit is identified and determined. A 3D algorithm model is constructed, and the coordinates of the fruit picking point image are obtained by performing coordinate transformation based on the 3D algorithm model and the picking point of the mature fruit in the 2D image.

[0008] In some possible implementations, the target recognition model is a Yolox network model improved based on the Yolox network model; The improved Yolox network model includes a feature extraction backbone module, an attention mechanism module, a feature extraction pyramid module, and an improved feature detection head module.

[0009] In some possible implementations, the fully trained target recognition model is used to identify fruits in a two-dimensional image and determine their ripeness. Simultaneously, the picking points for ripe fruits are identified and determined, including: The two-dimensional image containing the target fruit is input into the feature extraction backbone module to extract features and obtain the first target feature map; A second target feature map with attention information is determined based on the first target feature image and the attention mechanism module; The target feature map is obtained by extracting and fusing features of the second target feature through the feature extraction pyramid module; Based on the target feature map, the feature detection head module identifies the type of fruit in the two-dimensional image and determines the maturity of the fruit, while also determining the picking point of the mature fruit.

[0010] In some possible implementations, the feature extraction backbone module includes a Focus structure and a CSPDarknet structure; the CSPDarknet structure includes: a CBL module, a first CSP1-X module, a second CSP1-X module, a first CSP-res8 module, a second CSP-res8 module, a first Nonlinear mapping module, and a second Nonlinear mapping module; the two-dimensional image containing the target fruit picking point is input into the feature extraction backbone module for feature extraction to obtain a first target feature map, including: After inputting a two-dimensional image of the target fruit picking point into the Focus structure, the first target feature map is extracted pixel by pixel in the row and column directions to form a new feature layer. Each feature map is reorganized into 4 feature layers. Then, the 4 feature layers are stacked and the input channel is expanded by 4 times to output the Focus target feature map. The Focus target feature map is input into the CSPDarknet structure. The Focus target feature map is processed sequentially through the CBL module, the first CSP1-X module, the second CSP1-X module, the first CSP-res8 module, the second CSP-res8 module, the first Nonlinear mapping module, and the second Nonlinear mapping module to obtain the first target feature map.

[0011] In some possible implementations, the attention mechanism module includes a first attention mechanism sub-module, a second attention mechanism sub-module, and the second attention mechanism sub-module is three identical attention mechanism sub-modules. The attention mechanism sub-module includes a global average pooling layer, a Concat function layer, a first 1*1 convolutional layer, a first activation function layer, a second 1*1 convolutional layer, and a second activation function layer connected in sequence. The step of determining the second target feature map with attention information based on the first target feature image and the attention mechanism module includes: Pooling operations are performed based on the global average pooling layer to obtain wide-direction feature maps and high-direction feature maps; The Concat function layer merges the wide-direction feature map and the high-direction feature map to obtain a merged feature map. The merged feature map is reduced in dimension based on the first 1*1 convolutional layer to obtain the dimensionality-reduced merged feature map; The first attention weights for the wide-direction feature map and the high-direction feature map are determined based on the first activation function layer; The second 1*1 convolutional layer restores the feature maps in the wide and high directions to their original dimensions. The second attention weights are determined based on the second activation function to restore the feature maps in the wide and high directions to their original dimensions. The first target feature image is weighted according to the first attention weight and the second attention weight to obtain the second target feature image with attention information.

[0012] In some possible implementations, the feature extraction pyramid module includes an FPN network consisting of three feature layers, namely a first FPN network feature layer, a second FPN network feature layer, and a third FPN network feature layer. The step of extracting and fusing features from the second target feature map using the feature extraction pyramid module to obtain the target feature map includes: The output of the first attention mechanism submodule is used as input to the first FPN network feature layer to obtain the first FPN network features; The output of the second attention mechanism submodule is used as input to the second FPN network feature layer; The output of the third attention mechanism submodule is used as input to the third FPN network feature layer; The target feature map is obtained based on the first FPN network features, the first FPN network features, and the third FPN network features.

[0013] In some possible implementations, the feature detection head module includes a first detection head, a second detection head, and a third detection head; Based on the target feature map, the feature detection head module identifies the type of fruit in the 2D image and determines the ripeness of the fruit. Simultaneously, it determines the picking point for ripe fruit, including: Based on the feature information in the target feature map, the first detection head identifies whether the fruit is ripe. Based on the feature information in the target feature map, the picking point of the mature fruit in the fruit is identified by the second detection head; Based on the feature information contained in the target feature map, the third detection head identifies the position of the fruit on the two-dimensional image and the type of fruit.

[0014] In some possible implementations, a 3D algorithm model is constructed, and the 3D spatial coordinates of the fruit picking point image are obtained by performing coordinate transformation based on the 3D algorithm model and the picking point of the ripe fruit in the 2D image, including: Construct a three-dimensional algorithm model; A coordinate system is established based on the labeled images of ripe fruit picking points in a two-dimensional image to determine the two-dimensional coordinates of the fruit picking points on the two-dimensional image. The three-dimensional coordinates of the fruit picking point are determined based on the aforementioned 3D algorithm model and the two-dimensional coordinates of the fruit picking point on the two-dimensional image.

[0015] In some possible implementations, the stereo algorithm model uses the SGBM algorithm to locate the fruit picking point; The three-dimensional spatial coordinates of the fruit picking point are calculated and determined based on the fruit picking point in the two-dimensional image and the three-dimensional algorithm model, including: Based on the fruit picking point in the two-dimensional image, the two-dimensional coordinates on the fruit picking point image are obtained, and the corresponding coordinate points in the image are matched with the images obtained by the binocular camera to obtain the disparity value. Then, the three-dimensional spatial coordinates of the peach picking point are determined according to the SGBM algorithm.

[0016] In another aspect, the present invention also provides a fruit picking point positioning device, comprising: The image acquisition unit is used to acquire a two-dimensional image containing the target fruit; The fruit and picking point identification unit is used to acquire a fully trained target recognition model, and to identify the fruit in the two-dimensional image and determine the ripeness of the fruit based on the fully trained target recognition model, while identifying the picking point of the ripe fruit and determining the picking point of the ripe fruit. The picking point coordinate positioning unit is used to acquire a two-dimensional image containing the target fruit; A fully trained target recognition model is obtained, and the fruits in the two-dimensional image are identified and their ripeness is judged based on the fully trained target recognition model. At the same time, the picking point of the ripe fruit is identified and determined. A 3D algorithm model is constructed, and the coordinates of the fruit picking point image are obtained by performing coordinate transformation based on the 3D algorithm model and the picking point of the mature fruit in the 2D image.

[0017] Compared with the prior art, the beneficial effects of the present invention include: a method for identifying mature fruit and locating picking points improves the accuracy of fruit identification, fruit maturity identification, and fruit picking point location by using an improved target recognition model based on the Yolox network model to identify and locate the fruit and fruit picking point. At the same time, it improves the accuracy of fruit picking point identification and picking point coordinates by using a three-dimensional algorithm model to predict the three-dimensional coordinates of the picking point based on the two-dimensional coordinates of the picking point, thereby greatly improving the picking efficiency. Attached Figure Description

[0018] Figure 1 A flowchart illustrating an embodiment of a method for identifying ripe fruit and locating picking points provided by this invention; Figure 2 This is a schematic diagram of the model structure of one embodiment of the target recognition model provided in this invention. Figure 3 A schematic diagram of the model structure of an embodiment of the feature detection head module provided in this invention. Figure 4 A diagram illustrating the spatial relationships between the pixel coordinate system, camera coordinate system, image coordinate system, and world coordinate system provided in an embodiment of the present invention. Figure 5 This is a schematic diagram of an embodiment of a fruit picking point positioning device provided by the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0020] It should be understood that the illustrative drawings are not drawn to scale. The flowcharts used in this invention illustrate operations implemented according to some embodiments of the invention. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or performed simultaneously. Furthermore, those skilled in the art, guided by the content of this invention, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0021] Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor systems and / or microcontroller systems.

[0022] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0023] The present invention provides a method for identifying mature fruits and locating picking points, which will be described below.

[0024] Figure 1 A flowchart of an embodiment of a method for identifying ripe fruit and locating picking points provided by this invention includes the following steps: S101. Obtain a two-dimensional image containing the target fruit; S102. Obtain a fully trained target recognition model, and based on the fully trained target recognition model, identify the fruit in the two-dimensional image and determine the ripeness of the fruit. At the same time, identify the picking point of the ripe fruit and determine the picking point of the ripe fruit. S103. Construct a three-dimensional algorithm model, and obtain the three-dimensional spatial coordinates of the fruit picking point image by performing coordinate transformation based on the three-dimensional algorithm model and the picking point of the mature fruit in the two-dimensional image.

[0025] It should be noted that the target recognition model is an improved Yolox network model.

[0026] Compared with existing technologies, a method for identifying mature fruits and locating picking points improves the accuracy of fruit identification, fruit maturity identification, and fruit picking point location by using an improved target recognition model based on the Yolox network model to identify and locate fruits and fruit picking points. At the same time, it improves the accuracy of fruit picking point identification and picking point coordinates by using a 3D algorithm model based on the 2D coordinates of the fruit picking point to predict the 3D coordinates of the picking point, thereby greatly improving the picking efficiency.

[0027] In embodiments of the present invention, such as Figure 2 As shown, Figure 2 This is a schematic diagram of the model structure of one embodiment of the target recognition model provided in this invention. The target recognition model is a Yolox network model improved based on the Yolox network model. The improved Yolox network model includes a feature extraction backbone module, an attention mechanism module, a feature extraction pyramid module, and an improved feature detection head module.

[0028] In a specific embodiment, the feature extraction backbone module used by the Yolox network model is CSPDarknet. In the feature extraction pyramid module, the Yolox network model extracts multiple feature layers for target detection, thus adapting to targets of different sizes. The Yolox network model sets three feature layers, located in different parts of the feature extraction backbone module. All three feature layers are input from the feature extraction backbone module, first through an attention module, and then into the feature extraction pyramid. When the shape of the input backbone network is (640, 640, 3), the shapes of the three feature layers are (80, 80, 256), (40, 40, 512), and (20, 20, 1024), respectively. The feature pyramid structure fuses the features from the three feature layers before outputting them to the feature detection head module.

[0029] It should be noted that the CSPDarknet is a Darknet network with a CSP (Cross Stage Partial) structure.

[0030] It should be further noted that Darknet is a lightweight network architecture.

[0031] In this embodiment of the invention, based on the fully trained target recognition model, fruits in a two-dimensional image are identified and their ripeness is determined. Simultaneously, the picking points of ripe fruits are identified and determined, including: The two-dimensional image containing the target fruit picking point is input into the feature extraction backbone module to extract features and obtain the first target feature map; A second target feature map with attention information is determined based on the first target feature image and the attention mechanism module; The target feature map is obtained by extracting and fusing features of the second target feature through the feature extraction pyramid module; Based on the target feature map, the feature detection head module identifies the type of fruit in the two-dimensional image and determines the maturity of the fruit, while also determining the picking point of the mature fruit.

[0032] In a specific embodiment, the feature extraction backbone module includes a 1x1 convolution and a 3x3 convolution. The feature extraction backbone module also uses the CSPnet network structure, which splits the stacked residual network, with one part maintaining the original structure and the other part being a residual edge structure. In this way, the entire CSP has a larger residual structure. The feature extraction backbone module can alleviate the gradient vanishing problem in the neural network learning process by using skip connections through the residual network, thereby increasing the depth of the network. In turn, the increased network depth is used to improve the prediction accuracy of the neural network.

[0033] The feature extraction backbone module uses a Focus network structure, which extracts pixels at intervals within an image to obtain four feature layers. By stacking these four feature layers, the number of channels is quadrupled. This structure reduces the number of parameters, thereby improving computational speed.

[0034] The attention mechanism module exists to enhance the relationship between network analysis channels, consider more positional information, and thus improve the expressive ability of the detection network to learn features. This module first uses global average pooling on the first target feature map along both width and height dimensions. After obtaining feature maps in both directions, it uses `concat` to merge the first target feature maps. Next, it uses a 1×1 convolutional kernel for dimensionality reduction, performs another convolution operation, and then performs a normalization operation before feeding the feature map into the activation function. Subsequently, it uses 1×1 convolutional kernels to obtain feature maps with unchanged width and height, restoring the original dimensions of both parts. It then uses the Sigmoid activation function again to obtain attention weights in the width and height dimensions. Finally, it performs weighted calculations on the initial feature map, thus encoding information using attention in both width and height dimensions to obtain a second target feature map with attention information.

[0035] The feature pyramid module takes the output of the attention mechanism model as input and fuses the second target feature map with attention information to obtain the target feature map. Based on the target feature map, the feature detection head module identifies the type of fruit in the two-dimensional image and determines the maturity of the fruit, while also determining the picking point of the mature fruit.

[0036] In this embodiment of the invention, the feature extraction backbone module includes a Focus structure and a CSPDarknet structure; the CSPDarknet structure includes: a CBL module, a first CSP1-X module, a second CSP1-X module, a first CSP-res8 module, a second CSP-res8 module, a first Nonlinear mapping module, and a second Nonlinear mapping module; inputting the two-dimensional image containing the target fruit picking point into the feature extraction backbone module for feature extraction to obtain a first target feature map includes: After inputting a two-dimensional image of the target fruit picking point into the Focus structure, the first target feature map is extracted pixel by pixel in the row and column directions to form a new feature layer. Each feature map is reorganized into 4 feature layers. Then, the 4 feature layers are stacked and the input channel is expanded by 4 times to output the Focus target feature map. The Focus target feature map is input into the CSPDarknet structure. The Focus target feature map is processed sequentially through the CBL module, the first CSP1-X module, the second CSP1-X module, the first CSP-res8 module, the second CSP-res8 module, the first Nonlinear mapping module, and the second Nonlinear mapping module to obtain the first target feature map.

[0037] It should be noted that the CBL module consists of a convolutional layer (Conv), a batch normalization layer (BN layer), and a non-linear activation function (Leaky ReLU).

[0038] The CSP1_X module consists of a CBL module, X residual block Res units, a Conv convolutional layer, a Concat layer, a batch normalization layer (BN layer), a Leaky ReLU nonlinear activation function, and the CBL module. The CSP1_X module operates by processing data through two paths: one path sequentially processes the data through the CBL module, the X residual block Res units, and the Conv convolutional layer; the other path processes the data through the Conv convolutional layer. The results from these two paths are concatenated and then sequentially processed through the BN layer, the Leaky ReLU nonlinear activation function, and the CBL module, finally outputting the CSP1_X module's output. The residual block Res unit is obtained by adding the upper input (composed of two CBL modules) to the lower input through an add tensor operation.

[0039] The first CSP1-X module and the second CSP1-X module are two identical CSP1-X modules.

[0040] Furthermore, it should be noted that the first and second nonlinear mapping modules are two identical nonlinear mapping modules; the nonlinear mapping module consists of a convolutional layer (Conv) and a nonlinear activation function (Leaky ReLU); using nonlinear mapping to adjust the dimension of the feature map improves the details of feature fusion, thereby optimizing the detection effect of small targets.

[0041] Furthermore, it should be noted that the CSP_Res8 module is composed of 8 Res unit modules and the CBM component Concate tensor concatenated together.

[0042] The first CSP-res8 module and the second CSP-res8 module are two identical CSP_Res8 modules.

[0043] It's worth noting that the Focus network structure uses four feature layers obtained by taking pixels at intervals within an image. By stacking these four feature layers, the number of channels quadruples. This structure reduces the number of parameters, thereby improving computational speed.

[0044] In this embodiment of the invention, the attention mechanism module includes a first attention mechanism sub-module, a second attention mechanism sub-module, and the second attention mechanism sub-module is three identical attention mechanism sub-modules. The attention mechanism sub-module includes a global average pooling layer, a Concat function layer, a first 1*1 convolutional layer, a first activation function layer, a second 1*1 convolutional layer, and a second activation function layer connected in sequence. The step of determining the second target feature map with attention information based on the first target feature image and the attention mechanism module includes: Pooling operations are performed based on the global average pooling layer to obtain wide-direction feature maps and high-direction feature maps; The Concat function layer merges the wide-direction feature map and the high-direction feature map to obtain a merged feature map. The merged feature map is reduced in dimension based on the first 1*1 convolutional layer to obtain the dimensionality-reduced merged feature map. The first attention weights for the wide-direction feature map and the high-direction feature map are determined based on the first activation function layer; The second 1*1 convolutional layer restores the feature maps in the wide and high directions to their original dimensions. The second attention weights are determined based on the second activation function to restore the feature maps in the wide and high directions to their original dimensions. The first target feature image is weighted according to the first attention weight and the second attention weight to obtain the second target feature image with attention information.

[0045] In this embodiment of the invention, the feature extraction pyramid module includes an FPN network consisting of three feature layers, the FPN network including a first FPN network feature layer, a second FPN network feature layer and a third FPN network feature layer; The step of extracting and fusing features from the second target feature map using the feature extraction pyramid module to obtain the target feature map includes: The output of the first attention mechanism submodule is used as input to the first FPN network feature layer to obtain the first FPN network features; The output of the second attention mechanism submodule is used as input to the second FPN network feature layer; The output of the third attention mechanism submodule is used as input to the third FPN network feature layer; The target feature map is obtained based on the first FPN network features, the first FPN network features, and the third FPN network features.

[0046] It should be noted that the FPN network mentioned is the Feature Pyramid Network (FPN).

[0047] In this embodiment of the invention, the feature detection head module includes a first detection head, a second detection head, and a third detection head; Based on the target feature map, the feature detection head module identifies the type of fruit in the 2D image and determines the ripeness of the fruit. Simultaneously, it determines the picking point for ripe fruit, including: Based on the feature information in the target feature map, the first detection head identifies whether the fruit is ripe. Based on the feature information in the target feature map, the picking point of the mature fruit in the fruit is identified by the second detection head; Based on the feature information contained in the target feature map, the third detection head identifies the position of the fruit on the two-dimensional image and the type of fruit.

[0048] In a specific embodiment, such as Figure 3 , Figure 3 This is a schematic diagram of a model structure of an embodiment of the feature detection head module provided by the present invention. The feature detection head module is divided into three parts, as follows: Figure 3 Cls is the first detection head, Kps is the second detection head, and Reg and Obj are the third detection heads. The target feature map needs to undergo convolution processing before being detected by the three detection heads.

[0049] Cls and Obj share some parameters. Reg can extract the regression parameters of feature points to predict the detection box, i.e., the position of the fruit on the 2D image. The number of convolution channels in the prediction result of Reg is 4, which are the offset of the center point of the prediction box relative to the feature point and the parameters of the width and height of the prediction box relative to the logarithmic exponent. This part can predict the specific location box of the cherry fruit. Obj is used to determine whether the feature point contains an object. The number of convolution channels in the prediction result of Obj is 1, which represents the probability that the prediction box of each feature point contains an object.

[0050] Cls predicts the possible object types at a point, which can be used to determine the ripeness of cherry fruit. The number of convolution channels in the Cls prediction result is 3, and the predicted value represents the probability of which ripeness category the fruit belongs to.

[0051] Kps is used to predict keypoint coordinates. The prediction result of Kps has 6 convolution channels, which means the horizontal and vertical coordinates of three coordinate points, including the coordinates of one picking point and two auxiliary keypoint detection coordinates.

[0052] It should be noted that Conv2D represents two-dimensional convolution, BN represents batch normalization, and SiLU represents the activation function.

[0053] In this embodiment of the invention, a 3D algorithm model is constructed, and the 3D spatial coordinates of the fruit picking point image are obtained by performing coordinate transformation based on the 3D algorithm model and the picking point of the mature fruit in the 2D image, including: Construct a three-dimensional algorithm model; A coordinate system is established based on the labeled images of ripe fruit picking points in a two-dimensional image to determine the two-dimensional coordinates of the fruit picking points on the two-dimensional image. The three-dimensional coordinates of the fruit picking point are determined based on the aforementioned 3D algorithm model and the two-dimensional coordinates of the fruit picking point on the two-dimensional image.

[0054] In this embodiment of the invention, the 3D algorithm model uses the SGBM algorithm to locate the fruit picking point; The three-dimensional spatial coordinates of the fruit picking point are calculated and determined based on the fruit picking point in the two-dimensional image and the three-dimensional algorithm model, including: Based on the fruit picking point in the two-dimensional image, the two-dimensional coordinates on the fruit picking point image are obtained, and the corresponding coordinate points in the image are matched with the images obtained by the binocular camera to obtain the disparity value. Then, the three-dimensional spatial coordinates of the peach picking point are determined according to the SGBM algorithm.

[0055] In a specific embodiment, after obtaining the two-dimensional coordinates on the fruit picking point image, in order to locate the three-dimensional spatial coordinates of the fruit picking point, it is necessary to use the fruit image obtained by the binocular camera to convert the two-dimensional coordinates into three-dimensional coordinates.

[0056] In the specific coordinate transformation process, the coordinate transformation is completed based on a stereo matching algorithm by utilizing the relationship between pixel coordinates, image coordinates, camera coordinates, and world coordinates. The specific coordinate relationships are as follows: Figure 4 , Figure 4 This is a diagram illustrating the spatial relationship between the pixel coordinate system, camera coordinate system, image coordinate system, and world coordinate system provided in an embodiment of the present invention. The pixel coordinate system is established to determine the pixel coordinates on the image, the image coordinate system is used to determine the position of different images, the camera coordinate system is used to determine the position of the camera taking pictures, and the world coordinate system is used to determine the position of the pixel coordinates, image coordinates, and camera coordinates on the world coordinate system.

[0057] Stereo matching algorithms obtain disparity values ​​by matching corresponding points in the left and right images acquired by a camera. The algorithm first preprocesses the image using the horizontal Sobel operator, the specific formula of which is as follows:

[0058] Where P represents the original pixel value of the image, x is the horizontal coordinate of the pixel coordinate, and y is the vertical coordinate of the pixel coordinate.

[0059] The processed pixels are then mapped and calculated using the following mapping function to obtain new coordinate values. The specific mapping function formula is as follows:

[0060] Where P is the pixel value. The pixel values ​​are the values ​​in the new image, and preFilterCap is a constant parameter.

[0061] Furthermore, the new coordinate values ​​are subjected to cost matching using a cost matching formula, the specific cost matching formula of which is shown below: ; In the formula, Lr is the cost function for the sum of paths along the r direction, p is the number of pixels, d is the disparity, and c(p, d) represents the current cost. It is a smoothing penalty, which is generated when the difference between a pixel and its neighboring pixels is extremely small or extremely large. The third item is meaningless; it is only used to eliminate the impact of different path lengths in different directions.

[0062] Finally, the matching value of each pixel is determined based on the matching value. The specific formula for the matching value is shown below:

[0063] In the formula, p represents a pixel, d represents disparity, and Lr is the cost function for summing paths along the r direction. After obtaining the matching value of each pixel, the matching process can be completed and the three-dimensional coordinates of the fruit picking point can be determined through operations such as uniqueness detection, subpixel interpolation, and left-right consistency detection.

[0064] On the other hand, such as Figure 5 , Figure 5 This is a schematic diagram of an embodiment of a fruit picking point positioning device provided by the present invention. The present invention also provides a fruit picking point positioning device 500, comprising: Image acquisition unit 501 is used to acquire a two-dimensional image containing the target fruit; The fruit and picking point identification unit 502 is used to acquire a fully trained target recognition model, and to identify the fruit in the two-dimensional image and determine the ripeness of the fruit based on the fully trained target recognition model, while identifying the picking point of the ripe fruit and determining the picking point of the ripe fruit. The picking point coordinate positioning unit 503 uses a three-dimensional algorithm model to construct a three-dimensional algorithm model and performs coordinate transformation based on the three-dimensional algorithm model and the picking point of the mature fruit in the two-dimensional image to obtain the three-dimensional spatial coordinates of the fruit picking point image.

[0065] This invention provides a method for identifying mature fruits and locating picking points. By using an improved target recognition model based on a Yolox network model, the method identifies and locates fruits and picking points, thereby improving the accuracy of fruit identification, fruit maturity identification, and picking point location. At the same time, by using a 3D algorithm model to predict the 3D coordinates of the picking point based on the 2D coordinates, the method improves the accuracy of picking point identification and picking point coordinates, and greatly improves picking efficiency.

[0066] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0067] The present invention has provided a detailed description of the method and apparatus for identifying mature fruits and locating picking points. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. The above descriptions are only preferred embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the protection scope of the present invention.

Claims

1. A method for identifying mature fruit and locating picking points, characterized in that, include: Obtain a two-dimensional image containing the target fruit; A fully trained target recognition model is obtained, and based on the fully trained target recognition model, the feature detection head module of the target recognition model is used to identify fruits in a two-dimensional image and determine the ripeness of the fruits. At the same time, the picking points of ripe fruits are identified and determined. The feature detection head module includes a first detection head, a second detection head, and a third detection head. According to the feature information carried by the target feature map, the first detection head is used to identify whether the fruit is ripe. According to the feature information carried by the target feature map, the third detection head is used to identify the position of the fruit on the two-dimensional image and the type of fruit. Furthermore, based on the feature information carried by the target feature map, the picking point of the mature fruit in the fruit is identified by the second detection head, wherein the coordinates predicted by the second detection head include a picking point coordinate and two auxiliary key point detection coordinates. A 3D algorithm model is constructed. Based on the 3D algorithm model and the picking points of mature fruits in the 2D image, a coordinate transformation is performed to obtain the 3D spatial coordinates of the fruit picking point image. The 3D algorithm model includes the SGBM algorithm, comprising: By matching corresponding points in the left and right images acquired by the camera, the disparity value is obtained. The horizontal Sobel operator is then used for image preprocessing. The specific formula for the Sobel operator is as follows: Where P represents the original pixel value of the image, x is the horizontal coordinate of the pixel coordinate, and y is the vertical coordinate of the pixel coordinate. The processed pixels are mapped and calculated using the following mapping function to obtain new coordinate values. The specific mapping function formula is as follows: Where P is the pixel value. The pixel values ​​are the values ​​in the new image, and preFilterCap is a constant parameter. The new coordinate values ​​are matched using a cost-matching formula, which is shown below: ； In the formula, Lr is the cost function for the sum of paths along the r direction, p is the number of pixels, d is the disparity, and c(p, d) represents the current cost. It is a smoothing penalty, which occurs when the difference between a pixel and its neighboring pixels is extremely small or extremely large. The third item is meaningless; it is only used to eliminate the impact of different path lengths in different directions. The matching value of each pixel is determined based on the matching value, and the specific formula for the matching value is as follows: In the formula, p is the pixel, d is the disparity, and Lr is the cost function for the sum of paths in the r direction; After obtaining the matching value of each pixel, the matching process is completed through uniqueness detection, subpixel interpolation, and left-right consistency detection to determine the three-dimensional coordinates of the fruit picking point.

2. The method for identifying mature fruit and locating picking points according to claim 1, characterized in that, The target recognition model is an improved Yolox network model based on the Yolox network model; The improved Yolox network model includes a feature extraction backbone module, an attention mechanism module, a feature extraction pyramid module, and an improved feature detection head module.

3. The method for identifying mature fruit and locating picking points according to claim 2, characterized in that, Based on the fully trained target recognition model, the system identifies fruits in a two-dimensional image and determines their ripeness. Simultaneously, it identifies and determines the picking points for ripe fruits, including: The two-dimensional image containing the target fruit is input into the feature extraction backbone module to extract features and obtain the first target feature map; A second target feature map with attention information is determined based on the first target feature image and the attention mechanism module; The target feature map is obtained by extracting and fusing features of the second target feature through the feature extraction pyramid module; Based on the target feature map, the feature detection head module identifies the type of fruit in the two-dimensional image and determines the maturity of the fruit, while also determining the picking point of the mature fruit.

4. The method for identifying mature fruit and locating picking points according to claim 3, characterized in that, The feature extraction backbone module includes a Focus structure and a CSPDarknet structure; the CSPDarknet structure includes: a CBL module, a first CSP1-X module, a second CSP1-X module, a first CSP-res8 module, a second CSP-res8 module, a first Nonlinear mapping module, and a second Nonlinear mapping module; the two-dimensional image containing the target fruit picking point is input into the feature extraction backbone module for feature extraction to obtain a first target feature map, including: After inputting a two-dimensional image of the target fruit picking point into the Focus structure, the first target feature map is extracted pixel by pixel in the row and column directions to form a new feature layer. Each feature map is reorganized into 4 feature layers. Then, the 4 feature layers are stacked and the input channel is expanded by 4 times to output the Focus target feature map. The Focus target feature map is input into the CSPDarknet structure. The Focus target feature map is processed sequentially through the CBL module, the first CSP1-X module, the second CSP1-X module, the first CSP-res8 module, the second CSP-res8 module, the first Nonlinear mapping module, and the second Nonlinear mapping module to obtain the first target feature map.

5. The method for identifying mature fruit and locating picking points according to claim 3, characterized in that, The attention mechanism module includes a first attention mechanism sub-module, a second attention mechanism sub-module, and the second attention mechanism sub-module is three identical attention mechanism sub-modules. The attention mechanism sub-module includes a global average pooling layer, a Concat function layer, a first 1*1 convolutional layer, a first activation function layer, a second 1*1 convolutional layer, and a second activation function layer connected in sequence. The step of determining the second target feature map with attention information based on the first target feature image and the attention mechanism module includes: Pooling operations are performed based on the global average pooling layer to obtain wide-direction feature maps and high-direction feature maps; The Concat function layer merges the wide-direction feature map and the high-direction feature map to obtain a merged feature map. The merged feature map is reduced in dimension based on the first 1*1 convolutional layer to obtain the dimensionality-reduced merged feature map. The first attention weights for the wide-direction feature map and the high-direction feature map are determined based on the first activation function layer; The second 1*1 convolutional layer restores the feature maps in the wide and high directions to their original dimensions. The second attention weights are determined based on the second activation function to restore the feature maps in the wide and high directions to their original dimensions. The first target feature image is weighted according to the first attention weight and the second attention weight to obtain the second target feature image with attention information.

6. The method for identifying mature fruit and locating picking points according to claim 3, characterized in that, The feature extraction pyramid module includes an FPN network consisting of three feature layers, namely a first FPN network feature layer, a second FPN network feature layer, and a third FPN network feature layer. The step of extracting and fusing features from the second target feature map using the feature extraction pyramid module to obtain the target feature map includes: The output of the first attention mechanism submodule is used as input to the first FPN network feature layer to obtain the first FPN network features; The output of the second attention mechanism submodule is used as input to the second FPN network feature layer; The output of the third attention mechanism submodule is used as input to the third FPN network feature layer; The target feature map is obtained based on the first FPN network features, the first FPN network features, and the third FPN network features.

7. The method for identifying mature fruit and locating picking points according to claim 1, characterized in that, A 3D algorithm model is constructed. Based on the 3D algorithm model and the picking point of the mature fruit in the 2D image, a coordinate transformation is performed to obtain the 3D spatial coordinates of the fruit picking point image, including: Construct a three-dimensional algorithm model; A coordinate system is established based on the labeled images of ripe fruit picking points in a two-dimensional image to determine the two-dimensional coordinates of the fruit picking points on the two-dimensional image. The three-dimensional coordinates of the fruit picking point are determined based on the aforementioned 3D algorithm model and the two-dimensional coordinates of the fruit picking point on the two-dimensional image.

8. A fruit picking point positioning device, characterized in that, include: The image acquisition unit is used to acquire a two-dimensional image containing the target fruit; A fruit and picking point recognition unit is used to acquire a fully trained target recognition model, and based on the fully trained target recognition model, to identify fruits in a two-dimensional image and determine the ripeness of the fruits through the feature detection head module of the target recognition model, and to identify and determine the picking points of ripe fruits. The feature detection head module includes a first detection head, a second detection head, and a third detection head. Based on the feature information in the target feature map, the first detection head identifies whether the fruit is ripe; based on the feature information in the target feature map, the third detection head identifies the position of the fruit in the two-dimensional image and the type of fruit. Furthermore, based on the feature information carried by the target feature map, the picking point of the mature fruit in the fruit is identified by the second detection head, wherein the coordinates predicted by the second detection head include a picking point coordinate and two auxiliary key point detection coordinates. A picking point coordinate positioning unit is used to construct a 3D algorithm model. Based on the 3D algorithm model and the picking points of mature fruits in the 2D image, a coordinate transformation is performed to obtain the 3D spatial coordinates of the fruit picking point image. The 3D algorithm model includes the SGBM algorithm, comprising: By matching corresponding points in the left and right images acquired by the camera, the disparity value is obtained. The horizontal Sobel operator is then used for image preprocessing. The specific formula for the Sobel operator is as follows: Where P represents the original pixel value of the image, x is the horizontal coordinate of the pixel coordinate, and y is the vertical coordinate of the pixel coordinate. The processed pixels are mapped and calculated using the following mapping function to obtain new coordinate values. The specific mapping function formula is as follows: Where P is the pixel value. The pixel values ​​are the values ​​in the new image, and preFilterCap is a constant parameter. The new coordinate values ​​are matched using a cost-matching formula, which is shown below: ； In the formula, Lr is the cost function for the sum of paths along the r direction, p is the number of pixels, d is the disparity, and c(p, d) represents the current cost. It is a smoothing penalty, which occurs when the difference between a pixel and its neighboring pixels is extremely small or extremely large. The third item is meaningless; it is only used to eliminate the impact of different path lengths in different directions. The matching value of each pixel is determined based on the matching value, and the specific formula for the matching value is as follows: In the formula, p is the pixel, d is the disparity, and Lr is the cost function for the sum of paths in the r direction; After obtaining the matching value of each pixel, the matching process is completed through uniqueness detection, subpixel interpolation, and left-right consistency detection to determine the three-dimensional coordinates of the fruit picking point.

Images