Outdoor apple identification and positioning method and system based on multi-sensor fusion

Abstract

The invention provides an outdoor apple recognition and positioning method and system based on multi-sensor fusion, and relates to the technical field of target detection, and the method comprises the steps: obtaining the image data of an outdoor apple and laser radar point cloud data, inputting the image data to a target detection model, and obtaining an RGB image; obtaining an ROI depth information area by calibrating the external parameter matrix; extracting image edge features of the RGB image by using an edge detection algorithm to obtain an ROI edge information graph; performing denoising and discontinuous point detection on the ROI depth information region to obtain depth edge data; performing edge feature matching on the depth edge data and the ROI edge information graph by using a double-edge matching algorithm, and generating a smooth edge curve of the apple by using an RANSAC (Random Sample Consensus) fitting algorithm; and constructing a sphere model according to a fitting result, and solving a center coordinate and a radius by minimizing a residual sum of squares to obtain positioning three-dimensional information of the apple.

Description

Technical Field

[0001] The present disclosure relates to the field of target detection technology, and in particular to an outdoor apple recognition and positioning method and system based on multi-sensor fusion. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

[0003] In the automated harvesting of fruits and vegetables, the 3D information recognition technology of the target is crucial to improving the efficiency and accuracy of harvesting. The 3D recognition of apples is particularly challenging because apples are usually hidden by dense leaves and branches on the tree. Their complex surface reflectivity, shape-changing contours, and visibility differences under different lighting conditions all pose difficulties for high-precision 3D positioning and recognition. This makes the ability to obtain reliable 3D information under a variety of lighting environments a key requirement for intelligent harvesting systems.

[0004] Most of the current mainstream 3D recognition methods are based on structured light or binocular cameras, which achieve accurate positioning of the target by acquiring images on the camera plane and combining them with depth information. In these methods, structured light estimates the depth of the scene by projecting special patterns, but in strong lighting environments, the projected light is often interfered by sunlight, resulting in a significant decrease in the depth estimation effect of structured light, or even complete failure. At the same time, binocular cameras rely on stereo matching of two sets of images to calculate depth. However, under strong lighting, light interference and shadow changes often lead to errors in the recognition and positioning of matching points, resulting in inaccurate depth estimation, which is usually unacceptable in picking tasks with high precision requirements. Moreover, the failure of 3D information recognition under complex lighting conditions directly affects the accuracy and stability of apple recognition, which in turn brings unpredictable challenges to the picking task. Summary of the invention

[0005] In order to solve the above problems, the present invention proposes an outdoor apple recognition and positioning method and system based on multi-sensor fusion, which performs multi-modal fusion of the data of two sensors and uses an edge feature matching algorithm to accurately locate the three-dimensional information of apples under complex lighting conditions. At the same time, through the matching of area array laser radar and multi-robotic arms, a more stable detection effect can be achieved with limited recognition hardware cost.

[0006] According to some embodiments, the present disclosure adopts the following technical solutions: The outdoor apple recognition and positioning method based on multi-sensor fusion includes: Obtain outdoor apple image data and lidar point cloud data, and pre-process them; The image data is input into the object detection model to obtain the RGB image of the apple's two-dimensional bounding box. The lidar point cloud is projected into the camera coordinate system through the camera's intrinsic parameters and the multi-sensor calibration extrinsic parameter matrix to obtain the ROI depth information area that matches the two-dimensional bounding box. Use edge detection algorithm to extract image edge features from RGB images, and use inverse distance weighted transformation to assign edge degree to non-edge pixels, extend neighborhood edge information to non-edge pixels, and obtain a complete ROI edge information map; perform denoising and discontinuous point detection on the ROI depth information area to obtain the depth edge data under the denoised ROI point cloud; The double edge matching algorithm is used to match the edge features of the depth edge data with the ROI edge information map, identify the real edge contour of the apple, and use the RANSAC fitting algorithm to generate the smooth edge curve of the apple; A spherical model is constructed based on the fitting results, and the center coordinates and radius are solved by minimizing the residual sum of squares to obtain the three-dimensional positioning information of the apple.

[0007] According to some embodiments, the present disclosure adopts the following technical solutions: The outdoor apple recognition and positioning system based on multi-sensor fusion includes: Data acquisition module, used to acquire outdoor apple image data and LiDAR point cloud data, and pre-process them; The target recognition module is used to input the image data into the target detection model to obtain the RGB image of the apple's two-dimensional bounding box; through the camera intrinsic parameters and the calibration extrinsic parameter matrix of the multi-sensor, the lidar point cloud is projected to the camera coordinate system to obtain the ROI depth information area that matches the two-dimensional bounding box; The edge feature extraction module is used to extract the edge features of the RGB image using the edge detection algorithm, and assign the edge degree to the non-edge pixels using the inverse distance weighted transformation, and extend the neighborhood edge information to the non-edge pixels to obtain a complete ROI edge information map; denoise and detect discontinuous points in the ROI depth information area to obtain the depth edge data under the denoised ROI point cloud; The edge matching module is used to match the edge features of the depth edge data with the ROI edge information map using a dual edge matching algorithm, identify the true edge contour of the apple, and use the RANSAC fitting algorithm to generate a smooth edge curve of the apple; The positioning calculation module is used to build a spherical model based on the fitting results, and to solve the center coordinates and radius by minimizing the residual sum of squares to obtain the three-dimensional positioning information of the apple.

[0008] According to some embodiments, the present disclosure adopts the following technical solutions: A computer program product includes a computer program, and when the computer program is executed by a processor, the outdoor apple recognition and positioning method based on multi-sensor fusion is implemented.

[0009] According to some embodiments, the present disclosure adopts the following technical solutions: A non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the outdoor apple recognition and positioning method based on multi-sensor fusion is implemented.

[0010] According to some embodiments, the present disclosure adopts the following technical solutions: An electronic device includes: a processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to implement the outdoor apple recognition and positioning method based on multi-sensor fusion.

[0011] Compared with the prior art, the present invention has the following beneficial effects: The disclosed outdoor apple identification and positioning method based on multi-sensor fusion and the apple three-dimensional information identification method based on camera and laser radar are used to improve the robustness of identification in strong light interference environment. Laser radar has the characteristic of not being affected by ambient light, and can still maintain accurate depth detection under high-intensity lighting, and combine the image information of the camera to achieve more accurate apple position and shape estimation. By multimodally fusing the data of the two sensors, camera and laser radar, the limitations of a single method under strong light can be effectively overcome, laying a technical foundation for efficient identification and picking of outdoor apple orchards. At the same time, by matching the area array laser radar with multiple robotic arms, a more stable detection effect can be achieved under limited identification hardware costs. This innovative solution not only improves the adaptability of automated picking equipment in complex environments, but also provides new ideas for the application of highly robust target recognition technology in the agricultural field.

[0012] The outdoor apple recognition and positioning method based on multi-sensor fusion disclosed in the present invention uses a double edge matching algorithm to match edge features between depth edge data and ROI edge information graph, identifies the real edge contour of the apple, and uses a RANSAC fitting algorithm to generate a smooth edge curve of the apple. By using a double edge matching method of image edge and depth edge, the real contour information of the target fruit can be obtained more stably under complex outdoor lighting conditions, effectively reducing the overlapping occlusion effects of leaves, branches and other fruits.

[0013] The outdoor apple recognition and positioning method based on multi-sensor fusion disclosed in the present invention constructs a spherical model according to the fitting results, and solves the center coordinates and radius by minimizing the residual sum of squares to obtain the three-dimensional positioning information of the apple. Constructing the spherical model can better establish the three-dimensional position information of the apple, including the physical center of the apple and the three-dimensional contour information of the apple, so as to better realize the subsequent operation of the robot arm to grasp the target. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings constituting a part of the present disclosure are used to provide a further understanding of the present disclosure. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure.

[0015] Figure 1 This is a flow chart of an outdoor apple recognition and positioning method based on multi-sensor fusion according to an embodiment of the present disclosure; Figure 2 The figure is a schematic diagram of a specific implementation process of the outdoor apple recognition and positioning method based on multi-sensor fusion according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0016] The present disclosure is further described below in conjunction with the accompanying drawings and embodiments.

[0017] It should be noted that the following detailed descriptions are all illustrative and are intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present disclosure belongs.

[0018] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0019] Example 1 In one embodiment of the present disclosure, a method for outdoor apple recognition and positioning based on multi-sensor fusion is provided, comprising: Step 1: Obtain outdoor apple image data and lidar point cloud data, and pre-process them; Step 2: Input the image data into the object detection model to obtain the RGB image of the apple's two-dimensional bounding box; project the lidar point cloud to the camera coordinate system through the camera intrinsic parameters and the multi-sensor calibration extrinsic parameter matrix to obtain the ROI depth information area that matches the two-dimensional bounding box; Step 3: Use edge detection algorithm to extract image edge features from RGB images, and use inverse distance weighted transform to assign edge degree to non-edge pixels, extend neighborhood edge information to non-edge pixels, and obtain a complete ROI edge information map; perform denoising and discontinuous point detection on the ROI depth information area to obtain the depth edge data under the denoised ROI point cloud; Step 4: Use the double edge matching algorithm to match the edge features of the depth edge data with the ROI edge information map, identify the real edge contour of the apple, and use the RANSAC fitting algorithm to generate a smooth edge curve of the apple; Step 5: Build a spherical model based on the fitting results, solve the center coordinates and radius by minimizing the residual sum of squares, and obtain the three-dimensional positioning information of the apple.

[0020] As an embodiment, the specific implementation process of an outdoor apple recognition and positioning method based on multi-sensor fusion disclosed in the present invention is as follows: Step 1: Obtain outdoor apple image data and lidar point cloud data, and pre-process them; Specifically, the calibration results (camera internal and external parameters) are used to perform distortion correction to eliminate lens distortion. The point cloud data is filtered using statistical outlier removal (SOR) to remove outliers, reduce data volume, and improve processing efficiency.

[0021] Step 2: Input the image data into the object detection model to obtain the RGB image of the apple's two-dimensional bounding box; project the lidar point cloud to the camera coordinate system through the camera intrinsic parameters and the multi-sensor calibration extrinsic parameter matrix to obtain the ROI depth information area that matches the two-dimensional bounding box; Specifically, through the image captured by the camera, a deep learning target detection model such as Yolo is used to obtain the two-dimensional bounding box (Bbox) of the apple and record its pixel coordinate information. The pixel coordinates of the identified Bbox are obtained as follows: .

[0022] Furthermore, the lidar point cloud is projected to the camera coordinate system through the camera intrinsic parameters and the calibrated extrinsic matrix of the multi-sensor. Assume that the extrinsic matrix is , camera intrinsic parameter matrix K The LiDAR point cloud Mapped to the camera plane, and then cut to form a ROI depth information area that matches the Bbox.

[0023] Specifically, in order to map the LiDAR point cloud to the camera plane, the geometric calibration and projection process between multiple sensors must be completed. The camera intrinsic parameter matrix is ​​obtained through camera calibration tools (such as OpenCV or MATLAB calibration method) K , and correct the image distortion,

[0024] Among them, the camera calibration is obtained from the projection matrix It refers to extracting the camera's internal parameters from the camera's projection matrix, including the focal length and Optical Center calibration parameters.

[0025] Then use the PNP method to obtain the external parameter matrix of the area array laser radar and a camera. The PNP (Perspective-n-Point) method is a technology that solves the camera's posture (i.e., rotation matrix R and translation vector t) through known three-dimensional points and their two-dimensional points projected on the image plane. Specifically, the input data 3D point set (LiDAR coordinate system): the known three-dimensional point set P={(X,Y,Z)} is usually obtained by calibrating objects (such as a chessboard), and (X,Y,Z) is the three-dimensional coordinate position of the point cloud in the coordinate system.

[0026] Corresponding 2D pixel points: The projection points p={(u,v)} of these 3D points on the image plane are usually obtained through image processing. Camera intrinsic parameter matrix K As described above. Then calculate according to the projection formula,

[0027] in,[ R ∣ t ] is the position of the camera relative to the laser radar coordinate system. S is the scaling factor, and the depth Z related. K is the camera intrinsic matrix. By minimizing the error (such as reprojection error), the camera rotation matrix and translation vector are iteratively solved. (X, Y, Z) are the three-dimensional coordinates of the point cloud in the coordinate system, and X, Y, and Z are the specific positions of the X, Y, and Z axes of the coordinate system respectively. (u, v) are the projection point coordinates of the three-dimensional point on the image plane.

[0028] Finally, the external parameter matrix is ​​obtained .

[0029] in, R is the rotation matrix and t is the translation vector.

[0030] Step 3: Use the edge detection algorithm to extract the image edge features from the RGB image, and use the inverse distance weighted transform to assign edge degrees to non-edge pixels, extend the neighborhood edge information to non-edge pixels, and obtain a complete ROI edge information map; perform denoising and discontinuous point detection on the ROI depth information area to obtain the depth edge data under the denoised ROI point cloud; Specifically, 1) the edge detection algorithm is used to extract the edge features of the RGB image, and the inverse distance weighted transform is used to assign edge degrees to non-edge pixels, and the neighborhood edge information is extended to non-edge pixels. The process of obtaining a complete ROI edge information map is as follows: Due to different fields of view, the LiDAR point cloud will have deep interference noise caused by leaves, branches or fruits, which cannot be eliminated during conversion and will be brought into the ROI area. In order to eliminate the interference of non-apple objects (such as leaves and branches), the RGB image is first grayscaled, and then the edge detection algorithm such as Canny or Sobel is used to extract the image edge to obtain the binary edge map T.

[0031] At each edge pixel ( i , j ), the edge degree is calculated by measuring the maximum absolute value of the surrounding 8 neighborhood pixels. M ( i , j ), in the specific calculation, first obtain the target pixel and its 8-neighborhood pixels (8-neighborhood is defined as the pixel point ( i , j ) The absolute value of the grayscale value difference of the adjacent pixels in the horizontal, vertical and diagonal directions, and then take the maximum value of these differences as the edge degree of the pixel The larger the edge degree, the more significant the grayscale difference between the pixel and the surrounding pixels, and the more likely it is to be located in the edge area of ​​the image.

[0032] Furthermore, the inverse distance weighted transform is used to assign edge degrees to non-edge pixels. , the formula is:

[0033] in, is the edge degree of the pixel, T is a binary edge map, where each pixel value represents the edge value at that location. β The distance weighting coefficient of each non-edge point to other position points is larger the closer the point is, and γ It is set to 0.5, which represents the weighted sum of the maximum radiation of each non-edge pixel at other locations of that location. is the coordinate of the target pixel, that is, the pixel point whose edge degree is currently being calculated. Represents the coordinates of other pixel points, which are other points in the entire image that participate in the calculation. Represents the coordinate position The edge value of the pixel at the position. The radiation is related to the edge degree and distance of other positions. This assignment method extends the neighborhood edge information to non-edge pixels to obtain a complete ROI edge information map. At the same time, a threshold is set to remove points that do not conform to the edge, providing a more robust edge description for subsequent feature matching.

[0034] 2) De-noise and detect discontinuous points in the ROI depth information area to obtain the depth edge data under the denoised ROI point cloud, including: In the point cloud data, in order to remove the interference of leaves and branches, find the points where the depths of adjacent points are discontinuous in the ROI depth information area. i Point p， Calculate the distance difference between it and its neighboring points and calculate its depth discontinuity value ,

[0035] in, d Represents the depth of the point, which is the z value after the point cloud obtained by the lidar is converted to the camera coordinate system (x, y, z). The physical meaning is the distance from the point to the origin of the camera coordinate system in the z-axis direction. γ is a weight parameter used to control the influence of the distance difference on the depth discontinuity calculation, which is set to 0.5 here.

[0036] Filter all points whose depth discontinuity is less than the threshold (that is, continuous points) to obtain the depth edge data under the denoised ROI point cloud for further analysis.

[0037] Step 4: Use the double edge matching algorithm to match the edge features of the depth edge data with the ROI edge information map, identify the real edge contour of the apple, and use the RANSAC fitting algorithm to generate a smooth edge curve of the apple; Specifically, the deep edge data under the filtered and denoised ROI point cloud is matched with the ROI edge information map to identify the real edge of the apple. This includes: Get the image edge point set: Get the edge contour set of the two-dimensional image of the ROI edge information map ; Depth discontinuity point set: depth discontinuity point set of lidar point cloud , contains M discontinuities, representing potential edges that are considered as interference in the depth map.

[0038] The distance between edges is calculated by Chamfer matching method, and the edge area with the highest similarity is found to filter out interference items.

[0039] in, Represents the edge points of the image , and the set of depth discontinuities The Euclidean distance of the nearest point in . For each d Click and find its The point with the closest distance is summed up and averaged. Through this distance metric, if the Chamfer distance is small, it means that the edge feature matches well, and it can be considered that the found area is the edge contour set of the target apple. and .

[0040] Furthermore, after obtaining the filtered edge data, the RANSAC fitting algorithm is used to generate a smooth edge curve of the apple. The RANSAC method fits the edge curve by random sampling and minimizing the residual.

[0041] Let the fitted edge curve be C, then the edge point set and The sum of the distance residuals to the fitted curve is defined as,

[0042] in, and Represents the projection point of curve C at point d and x position. Represents Euclidean distance calculation, and by minimizing Error, the best fitting curve can be found. C , thus generating a smooth apple edge curve. This curve can be used as a basis for subsequent apple positioning and shape estimation.

[0043] Step 5: Construct a spherical model based on the fitting results, solve the center coordinates and radius by minimizing the residual sum of squares, and obtain the three-dimensional positioning information of the apple.

[0044] Specifically, assuming that the apple is spherical or approximately spherical, multiple evenly distributed sample points are selected using the inner depth point of the fitting curve to calculate the depth value of each point. The center position of the apple and its three-dimensional information are calculated by fitting the sample depth in the point cloud using the least squares spherical fitting method. N Evenly distributed point cloud depth points, expressed as (where i=1,2,…,N). The goal is to find a sphere model so that these points are as close to the surface of the sphere as possible.

[0045] The mathematical model of a sphere can be expressed as,

[0046] in, Represents the center of the sphere.

[0047] Solve for the center coordinates and radius by minimizing the residual sum of squares r , the residual sum of squares is defined as,

[0048] After expanding the above formula, we can get and r In order to minimize the Error, the nonlinear equations can be solved by the least squares method or a numerical optimization algorithm (such as the Levenberg-Marquardt algorithm). The result of the solution is the three-dimensional information of the apple accurately positioned under complex lighting conditions.

[0049] Through the above-mentioned multi-sensor fusion and edge feature matching algorithm, the present invention can accurately locate the three-dimensional information of apples under complex lighting conditions, thereby improving the robustness and accuracy of the picking system.

[0050] Example 2 An embodiment of the present disclosure provides an outdoor apple recognition and positioning system based on multi-sensor fusion, including: Data acquisition module, used to acquire outdoor apple image data and LiDAR point cloud data, and pre-process them; The target recognition module is used to input the image data into the target detection model to obtain the RGB image of the apple's two-dimensional bounding box; through the camera intrinsic parameters and the calibration extrinsic parameter matrix of the multi-sensor, the lidar point cloud is projected to the camera coordinate system to obtain the ROI depth information area that matches the two-dimensional bounding box; The edge feature extraction module is used to extract the edge features of the RGB image using the edge detection algorithm, and assign the edge degree to the non-edge pixels using the inverse distance weighted transformation, and extend the neighborhood edge information to the non-edge pixels to obtain a complete ROI edge information map; denoise and detect discontinuous points in the ROI depth information area to obtain the depth edge data under the denoised ROI point cloud; The edge matching module is used to match the edge features of the depth edge data with the ROI edge information map using a dual edge matching algorithm, identify the true edge contour of the apple, and use the RANSAC fitting algorithm to generate a smooth edge curve of the apple; The positioning calculation module is used to build a spherical model based on the fitting results, and to solve the center coordinates and radius by minimizing the residual sum of squares to obtain the three-dimensional positioning information of the apple.

[0051] Example 3 In one embodiment of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor, implements the outdoor apple recognition and positioning method based on multi-sensor fusion.

[0052] Example 4 In one embodiment of the present disclosure, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by a processor, the outdoor apple recognition and positioning method based on multi-sensor fusion is implemented.

[0053] Example 5 In one embodiment of the present disclosure, an electronic device is provided, including: a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device executes the outdoor apple recognition and positioning method based on multi-sensor fusion.

[0054] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0055] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0056] Although the above describes the specific implementation methods of the present disclosure in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present disclosure. Technical personnel in the relevant field should understand that on the basis of the technical solution of the present disclosure, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present disclosure.

Claims

1. An outdoor apple recognition and positioning method based on multi-sensor fusion is characterized by: include: Obtain outdoor apple image data and lidar point cloud data, and pre-process them; The image data is input into the object detection model to obtain the RGB image of the apple's two-dimensional bounding box. The lidar point cloud is projected into the camera coordinate system through the camera's intrinsic parameters and the multi-sensor calibration extrinsic parameter matrix to obtain the ROI depth information area that matches the two-dimensional bounding box. Use edge detection algorithm to extract image edge features from RGB images, and use inverse distance weighted transformation to assign edge degree to non-edge pixels, extend neighborhood edge information to non-edge pixels, and obtain a complete ROI edge information map; perform denoising and discontinuous point detection on the ROI depth information area to obtain the depth edge data under the denoised ROI point cloud; The double edge matching algorithm is used to match the edge features of the depth edge data with the ROI edge information map, identify the real edge contour of the apple, and use the RANSAC fitting algorithm to generate the smooth edge curve of the apple; A spherical model is constructed based on the fitting results, and the center coordinates and radius are solved by minimizing the residual sum of squares to obtain the three-dimensional positioning information of the apple.

2. The outdoor apple recognition and positioning method based on multi-sensor fusion as claimed in claim 1, characterized in that: The YOLO target detection model is used to obtain the apple image captured by the camera, obtain the two-dimensional bounding box Bbox of the apple and record its pixel coordinate information; the lidar point cloud is projected to the camera coordinate system through the camera intrinsic parameter matrix and the calibrated extrinsic parameter matrix of the multi-sensor; the camera intrinsic parameter matrix maps the lidar point cloud to the camera plane, and then intercepts it to form a ROI depth information area that matches the two-dimensional bounding box Bbox.

3. The outdoor apple recognition and positioning method based on multi-sensor fusion as claimed in claim 1, characterized in that: The RGB image is grayed out, and the edge detection algorithm is used to extract the image edge to obtain a binary edge map. At each edge pixel, the edge degree is measured by the maximum absolute value of the surrounding 8 neighborhood pixels, and the inverse distance weighted transform is used to assign the edge degree to the non-edge pixels, which is: in, T is a binary edge map, where each pixel value represents the edge value at that location. β The distance weighting coefficient of each non-edge point to other position points is larger the closer the point is, and γ Set to 0.5, representing the weighted sum of the maximum radiation of each non-edge pixel at that position. i , j ) is the coordinate of the target pixel, that is, the pixel whose edge degree is currently being calculated, (x, y) represents the coordinate of other pixels, which are other points in the entire image involved in the calculation, and the subscripts x and y of T represent the position.

4. The outdoor apple recognition and positioning method based on multi-sensor fusion as claimed in claim 1, characterized in that: In the point cloud data, remove the interference of various noises, find the points with discontinuous depth between adjacent points in the ROI depth information area, and set the first i Point p , calculate the distance difference between it and its neighboring points, and calculate its depth discontinuity value , in, d is the distance from the point to the origin of the camera coordinate system in the z-axis direction, γ is the weight parameter; Filter all points whose depth discontinuity is less than the set threshold to obtain the depth edge data under the denoised ROI point cloud.

5. The outdoor apple recognition and positioning method based on multi-sensor fusion as claimed in claim 1, characterized in that: A set of pixel points of the two-dimensional image is extracted from the ROI edge information map, and a set of depth discontinuous points of the lidar point cloud is obtained from the depth edge data. The distance between edges is calculated by the Chamfer matching method. Through distance measurement, if the Chamfer distance is small, it means that the edge feature matches well, and the found area is considered to be the edge contour of the target apple. After obtaining the filtered edge contour data, the RANSAC fitting algorithm is used to generate a smooth edge curve of the apple.

6. The outdoor apple recognition and positioning method based on multi-sensor fusion as claimed in claim 1, characterized in that: Construct a spherical model of an apple with a spherical or spherical structure. Use the inner depth points of the fitting curve to select multiple evenly distributed sample points, calculate the depth value of each point, and calculate the center position of the apple and its three-dimensional information by fitting the sample depth in the point cloud using the least squares spherical fitting method. The mathematical model of the sphere is expressed as: Among them, multiple evenly distributed point cloud depth points are selected inside the fitting curve and expressed as , r represents the radius of the sphere, c The center point of the sphere.

7. The outdoor apple recognition and positioning system based on multi-sensor fusion is characterized by: include: Data acquisition module, used to acquire outdoor apple image data and LiDAR point cloud data, and pre-process them; The target recognition module is used to input the image data into the target detection model to obtain the RGB image of the apple's two-dimensional bounding box; through the camera intrinsic parameters and the calibration extrinsic parameter matrix of the multi-sensor, the lidar point cloud is projected to the camera coordinate system to obtain the ROI depth information area that matches the two-dimensional bounding box; The edge feature extraction module is used to extract the edge features of the RGB image using the edge detection algorithm, and assign the edge degree to the non-edge pixels using the inverse distance weighted transformation, and extend the neighborhood edge information to the non-edge pixels to obtain a complete ROI edge information map; denoise and detect discontinuous points in the ROI depth information area to obtain the depth edge data under the denoised ROI point cloud; The edge matching module is used to match the edge features of the depth edge data with the ROI edge information map using a dual edge matching algorithm, identify the true edge contour of the apple, and use the RANSAC fitting algorithm to generate a smooth edge curve of the apple; The positioning calculation module is used to build a spherical model based on the fitting results, and to solve the center coordinates and radius by minimizing the residual sum of squares to obtain the three-dimensional positioning information of the apple.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the outdoor apple recognition and positioning method based on multi-sensor fusion described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by the processor, the outdoor apple identification and positioning method based on multi-sensor fusion as described in any one of claims 1-6 is implemented.

10. An electronic device, characterized in that: include: A processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to implement the outdoor apple recognition and positioning method based on multi-sensor fusion as described in any one of claims 1-6.

Images