Target positioning method and system based on RGB and point cloud information fusion

Abstract

The invention relates to the technical field of three-dimensional image analysis, and specifically to a target positioning method and system based on fusion of RGB and point cloud information. In order to solve the technical problem of low positioning and identification accuracy of Agaricus bisporus in the prior art, the invention first obtains several mushroom bed area depth point cloud sets based on several mushroom bed area depth maps on the surface of an Agaricus bisporus mushroom bed, and splices them after registration to obtain a spliced ​​mushroom bed depth point cloud model that is convenient for accurately and intuitively reflecting the actual shape and depth change of the mushroom bed; then, based on the Agaricus bisporus pixel information reflected by the RGB map of the mushroom bed, the background is removed and then the occlusion identification of Agaricus bisporus is performed to obtain a set of Agaricus bisporus identification areas that can be picked and meet the picking conditions; finally, the background in the spliced ​​mushroom bed depth point cloud model is removed based on the depth information, and the point cloud data corresponding to the Agaricus bisporus identification area that can be picked is marked to obtain accurate point cloud positioning data of Agaricus bisporus that can be picked, thereby improving the Agaricus bisporus positioning and identification accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional image analysis, and in particular to a target object positioning method and system based on fusion of RGB and point cloud information. Background Art

[0002] With the development of modern cultivation technology, factory production has become a trend. After my country adopted factory production of Agaricus bisporus, the output has increased year by year. At present, the planting range is all over the country, and the annual output and annual export volume are among the highest in the world. At present, the picking of Agaricus bisporus is manual operation, which is inefficient and affects the mushroom house environment, which is not conducive to improving production efficiency and product quality. With the increasing shortage of labor, it is imperative to design advanced picking and sorting robots.

[0003] Currently, Agaricus bisporus is often located and identified based on two-dimensional images to determine whether Agaricus bisporus can be picked. However, two-dimensional images are easily affected by lighting changes, complex backgrounds, and occlusion by clusters of Agaricus bisporus. This results in low accuracy in the final positioning and identification of Agaricus bisporus, which affects the quality of Agaricus bisporus picked by sorting robots. Summary of the invention

[0004] The purpose of the present invention is to provide a method and system for locating an object based on the fusion of RGB and point cloud information.

[0005] The technical solution of the present invention is as follows:

[0006] A target positioning method based on RGB and point cloud information fusion includes the following operations:

[0007] S1. Obtain RGB images and depth images of several areas on the surface of the Agaricus bisporus mushroom bed to obtain several RGB images and depth images of the mushroom bed area; obtain respective depth point cloud sets of the mushroom bed area based on the several depth images of the mushroom bed area; after registering the several depth point cloud sets of the mushroom bed area, splice the registered depth point cloud sets of the mushroom bed areas in adjacent areas to obtain a spliced ​​mushroom bed depth point cloud model;

[0008] S2, several RGB images of mushroom bed areas are spliced ​​to obtain a spliced ​​mushroom bed RGB image; the background in the spliced ​​mushroom bed RGB image is removed to obtain an Agaricus bisporus image; a connected domain analysis is performed on the Agaricus bisporus image to obtain an initial Agaricus bisporus identification area; based on the connected domain area and the connected domain aspect ratio, the Agaricus bisporus overlapping area in the initial Agaricus bisporus identification area is marked; the Agaricus bisporus overlapping area is subjected to Hough transform processing to obtain an Agaricus bisporus segmentation and identification area at the overlap; the Agaricus bisporus segmentation and identification area at the overlap and the non-Agaricus bisporus overlapping area in the initial Agaricus bisporus identification area form an Agaricus bisporus identification area set; based on the standard Agaricus bisporus area, the Agaricus bisporus identification area set is screened to obtain a harvestable Agaricus bisporus identification area set;

[0009] S3. Remove the background in the spliced ​​mushroom bed depth point cloud model to obtain the Agaricus bisporus depth point cloud model; mark the point cloud corresponding to the harvestable Agaricus bisporus identification area in the Agaricus bisporus depth point cloud model to obtain the harvestable Agaricus bisporus point cloud positioning data.

[0010] The operations of obtaining the mushroom bed area depth point cloud set based on the mushroom bed area depth map in S1 are specifically as follows: based on the camera matrix, the coordinate value and the corresponding depth value of each pixel point in the mushroom bed area depth map are converted into point coordinates in three-dimensional space to obtain an initial depth point cloud set; the initial depth point cloud set is subjected to downsampling processing to obtain a downsampled depth point cloud set; the normal of the downsampled depth point cloud set is obtained, and the downsampled depth point cloud set is subjected to least squares processing and nearest neighbor interpolation processing based on the normal to obtain a mushroom bed area depth point cloud set.

[0011] The specific operations of the registration processing in S1 are as follows: select several point clouds from the previous registered mushroom bed area depth point cloud set as feature point clouds; obtain the point clouds closest to the several feature point clouds in the current mushroom bed area depth point cloud set, and form point cloud pairing with the corresponding feature point clouds; obtain the current rotation and translation matrix based on the point cloud pairing; perform rotation and translation transformation on the current mushroom bed area depth point cloud set based on the current rotation and translation matrix to obtain the current round of updated depth point cloud set; determine whether the current round of updated depth point cloud set is consistent with the previous registered mushroom bed area depth point cloud set. Whether the transformation error of the point cloud set is less than the change error threshold; if not, the current round of updated depth point cloud set and the previous registered mushroom bed area depth point cloud set perform several point cloud pairing acquisition, rotation and translation matrix acquisition and rotation and translation transformation operations until the transformation error between the updated depth point cloud set and the previous registered mushroom bed area depth point cloud set is less than the change error threshold, and the current registered mushroom bed area depth point cloud set is obtained, and the stitching operation is performed; if less than, the current round of updated depth point cloud set is used as the current registered mushroom bed area depth point cloud set, and the stitching operation is performed.

[0012] The transformation error is obtained by the following formula:

[0013] ,

[0014] Depth point cloud collection for the current mushroom bed area s Depth point cloud set of the mushroom bed area aligned with the previous one s-1 The transformation error, N is the total number of point clouds in the depth point cloud set of the current mushroom bed area, Depth point cloud collection for the last registered mushroom bed area s- 1 Middle j The location of the point cloud, The depth point cloud in the current mushroom bed area s Middle jThe point cloud closest to the point cloud i location, is the initial rotation and translation matrix.

[0015] The operation of removing the background in the spliced ​​mushroom bed RGB image in S2 is specifically as follows: converting the spliced ​​mushroom bed RGB image into the HSV space and performing normalization processing to obtain a normalized mushroom bed image; binarizing the normalized mushroom bed image to retain the corresponding pixels of Agaricus bisporus to obtain an Agaricus bisporus image.

[0016] The specific operation of removing the background in the spliced ​​mushroom bed depth point cloud model to obtain the Agaricus bisporus depth point cloud model in S2 is: several mushroom bed area depth maps are spliced ​​to obtain a spliced ​​mushroom bed depth map; the pixel points in the spliced ​​mushroom bed depth map with depth values ​​less than the depth threshold are used as background points; the point cloud corresponding to the background points in the spliced ​​mushroom bed depth point cloud model is deleted to obtain the initial screening mushroom bed depth point cloud model; the initial screening mushroom bed depth point cloud model is filtered to obtain the Agaricus bisporus depth point cloud model.

[0017] A target positioning system based on RGB and point cloud information fusion, used to implement the above-mentioned target positioning method based on RGB and point cloud information fusion, comprising:

[0018] The spliced ​​mushroom bed depth point cloud model generation module is used to obtain the RGB images and depth images of several areas of the Agaricus bisporus mushroom bed surface, and obtain several mushroom bed area RGB images and mushroom bed area depth images; based on the several mushroom bed area depth images, obtain respective mushroom bed area depth point cloud sets; after registering the several mushroom bed area depth point cloud sets, splice the mushroom bed area registered depth point cloud sets of adjacent areas to obtain a spliced ​​mushroom bed depth point cloud model;

[0019] The module for generating a set of harvestable Agaricus bisporus identification regions is used for splicing a number of RGB images of mushroom bed regions to obtain a spliced ​​mushroom bed RGB image; removing the background in the spliced ​​mushroom bed RGB image to obtain an Agaricus bisporus image; performing connected domain analysis on the Agaricus bisporus image to obtain an initial Agaricus bisporus identification region; marking the Agaricus bisporus overlapping region in the initial Agaricus bisporus identification region based on the connected domain area and the connected domain aspect ratio; performing Hough transform processing on the Agaricus bisporus overlapping region to obtain an Agaricus bisporus segmentation identification region at the overlap; the Agaricus bisporus segmentation identification region at the overlap and the non-Agaricus bisporus overlapping region in the initial Agaricus bisporus identification region form an Agaricus bisporus identification region set; based on the standard Agaricus bisporus area, the Agaricus bisporus identification region set is screened to obtain an Agaricus bisporus identification region set that can be picked;

[0020] The module for generating point cloud positioning data of harvestable Agaricus bisporus is used to remove the background in the spliced ​​mushroom bed depth point cloud model to obtain the Agaricus bisporus depth point cloud model; the point cloud corresponding to the harvestable Agaricus bisporus identification area is marked in the Agaricus bisporus depth point cloud model to obtain the point cloud positioning data of harvestable Agaricus bisporus.

[0021] A target positioning device based on fusion of RGB and point cloud information comprises a processor and a memory, wherein the processor implements the above-mentioned target positioning method based on fusion of RGB and point cloud information when executing a computer program stored in the memory.

[0022] A computer-readable storage medium is used to store a computer program, wherein when the computer program is executed by a processor, the above-mentioned target positioning method based on fusion of RGB and point cloud information is implemented.

[0023] The beneficial effects of the present invention are:

[0024] The present invention provides a target positioning method based on the fusion of RGB and point cloud information. First, based on a plurality of mushroom bed area depth maps on the surface of an Agaricus bisporus mushroom bed, a plurality of mushroom bed area depth point cloud sets are obtained, which are spliced ​​after registration to obtain a spliced ​​mushroom bed depth point cloud model that is convenient for accurately and intuitively reflecting the actual shape and depth change of the mushroom bed; then, based on the Agaricus bisporus pixel information reflected by the RGB map of the mushroom bed, the background is removed and then the Agaricus bisporus occlusion recognition is performed to obtain a set of Agaricus bisporus identification areas that can be picked that meet the picking conditions; finally, the background in the spliced ​​mushroom bed depth point cloud model is removed based on the depth information, and the point cloud data corresponding to the Agaricus bisporus identification area that can be picked is marked to obtain accurate Agaricus bisporus point cloud positioning data that can be picked, thereby improving the Agaricus bisporus positioning and recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] By reading the detailed description of the preferred embodiment below, the scheme and advantages of the present application will become clear to those skilled in the art. The accompanying drawings are only for the purpose of illustrating the preferred embodiment and are not to be considered as limiting the present invention.

[0026] In the attached picture:

[0027] Figure 1 This is a depth point cloud model diagram of the spliced ​​mushroom bed in this embodiment;

[0028] Figure 2 is a marking diagram of the Agaricus bisporus that can be picked in the embodiment;

[0029] Figure 3 : is a point cloud image of a single Agaricus bisporus after removing the background in this embodiment;

[0030] Figure 4 : is a depth diagram of the spliced ​​mushroom bed in this embodiment. DETAILED DESCRIPTION

[0031] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings.

[0032] This embodiment provides a target positioning method based on RGB and point cloud information fusion, including the following operations:

[0033] S1. Obtain RGB images and depth images of several areas on the surface of the Agaricus bisporus mushroom bed to obtain several RGB images and depth images of the mushroom bed area; obtain respective depth point cloud sets of the mushroom bed area based on the several depth images of the mushroom bed area; after registering the several depth point cloud sets of the mushroom bed area, splice the registered depth point cloud sets of the mushroom bed areas in adjacent areas to obtain a spliced ​​mushroom bed depth point cloud model;

[0034] S2, several RGB images of mushroom bed areas are spliced ​​to obtain a spliced ​​mushroom bed RGB image; the background in the spliced ​​mushroom bed RGB image is removed to obtain an Agaricus bisporus image; a connected domain analysis is performed on the Agaricus bisporus image to obtain an initial Agaricus bisporus identification area; based on the connected domain area and the connected domain aspect ratio, the Agaricus bisporus overlapping area in the initial Agaricus bisporus identification area is marked; the Agaricus bisporus overlapping area is subjected to Hough transform processing to obtain an Agaricus bisporus segmentation and identification area at the overlap; the Agaricus bisporus segmentation and identification area at the overlap and the non-Agaricus bisporus overlapping area in the initial Agaricus bisporus identification area form an Agaricus bisporus identification area set; based on the standard Agaricus bisporus area, the Agaricus bisporus identification area set is screened to obtain a harvestable Agaricus bisporus identification area set;

[0035] S3. Remove the background in the spliced ​​mushroom bed depth point cloud model to obtain the Agaricus bisporus depth point cloud model; mark the point cloud corresponding to the harvestable Agaricus bisporus identification area in the Agaricus bisporus depth point cloud model to obtain the harvestable Agaricus bisporus point cloud positioning data.

[0036] S1. Obtain RGB images and depth images of several areas on the surface of an Agaricus bisporus mushroom bed to obtain several RGB images and depth images of the mushroom bed area; based on the several depth images of the mushroom bed area, obtain respective depth point cloud sets of the mushroom bed area; after registering the several depth point cloud sets of the mushroom bed area, splice the registered depth point cloud sets of the mushroom bed areas in adjacent areas to obtain a spliced ​​mushroom bed depth point cloud model.

[0037] Based on several depth maps of the Agaricus bisporus mushroom bed area on the surface of the mushroom bed, a spliced ​​mushroom bed depth point cloud model is established to accurately and intuitively reflect the actual shape and depth changes of the mushroom bed, which is conducive to the subsequent positioning and identification of Agaricus bisporus.

[0038] First, use several depth cameras (preferably Intel realsense D435 cameras) at different positions above the Agaricus bisporus mushroom bed to shoot the Agaricus bisporus mushroom bed, obtain RGB images and depth images of several areas on the surface of the Agaricus bisporus mushroom bed, and obtain several RGB images of the mushroom bed area that can reflect the pixel distribution characteristics of the mushroom bed, and the depth images of the mushroom bed area that can reflect the depth information characteristics of the mushroom bed. The total shooting range of the several depth cameras covers the Agaricus bisporus mushroom bed.

[0039] Then, based on several mushroom bed area depth maps, the respective mushroom bed area depth point cloud sets are obtained, which is convenient for intuitively analyzing the depth information of the mushroom bed area. Specifically, based on the camera matrix, the coordinate value and corresponding depth value of each pixel point in the mushroom bed area depth map are converted into point coordinates in three-dimensional space to obtain the initial depth point cloud set; the initial depth point cloud set is processed by downsampling to obtain the downsampled depth point cloud set; the normal of the downsampled depth point cloud set is obtained, and the downsampled depth point cloud set is processed by least squares method and nearest neighbor interpolation based on the normal to smooth the point cloud data, reduce the point cloud fluctuation caused by measurement error or local noise, and restore some detail information that may be lost in the downsampling process, so that the point cloud of the mushroom bed is more complete and accurate, and can better reflect the actual shape and depth change of the mushroom bed. The depth point cloud set of the mushroom bed area is obtained. All mushroom bed area depth maps are processed by point coordinate conversion, downsampling, least squares method processing and nearest neighbor interpolation respectively to obtain their respective mushroom bed area depth point cloud sets, providing more accurate data for subsequent mushroom bed analysis.

[0040] Finally, after registering several mushroom bed area depth point cloud sets, the registered depth point cloud sets of mushroom bed areas in adjacent areas are spliced ​​to obtain a spliced ​​mushroom bed depth point cloud model. The effect can be seen in Figure 1 .

[0041] Taking the previous registered mushroom bed area depth point cloud set and the current mushroom bed area depth point cloud set as an example, the registration processing operation is specifically as follows.

[0042] Step 1: Randomly select several point clouds from the previous registered mushroom bed area depth point cloud set as feature point clouds; obtain the point clouds closest to several feature point clouds in the current mushroom bed area depth point cloud set, and form point cloud pairing with the corresponding feature point clouds.

[0043] Step 2: Based on all point cloud pairings, the current rotation and translation matrix is ​​obtained; specifically, based on all point cloud pairings, it is required that there are at least three non-collinear point cloud pairings in all point cloud pairings, and a linear equation group corresponding to the rigid body transformation is constructed. The linear equation group is solved by the least squares method to obtain the rotation and translation; based on the rotation and translation, a homogeneous transformation matrix is ​​constructed to obtain the current rotation and translation matrix; then, based on the current rotation and translation matrix, a rotation and translation transformation is performed on the current mushroom bed area depth point cloud set, so that the distance and shape features of the current mushroom bed area depth point cloud set are closer to the previous registered mushroom bed area depth point cloud set, and the current round of updated depth point cloud set is obtained.

[0044] The third step is to determine whether the transformation error between the current round of updated depth point cloud set and the previous registered mushroom bed area depth point cloud set is less than the change error threshold.

[0045] If it is not less than, it means that the registration accuracy of the current round of updated depth point cloud set and the previous registered mushroom bed area depth point cloud set does not meet the standard, then the current round of updated depth point cloud set is used as the new depth point cloud set, and several point cloud pairing acquisition, rotation and translation matrix acquisition and rotation and translation transformation operations are performed with the previous registered mushroom bed area depth point cloud set until the transformation error between the updated depth point cloud set and the previous registered mushroom bed area depth point cloud set is less than the change error threshold, and the corresponding round of updated depth point cloud set is used as the current registered mushroom bed area depth point cloud set, and subsequent stitching operations are performed.

[0046] If it is less than, it means that the registration accuracy of the current round of updated depth point cloud set and the previous registered mushroom bed area depth point cloud set meets the standard. The current round of updated depth point cloud set is used as the current registered mushroom bed area depth point cloud set to perform subsequent stitching operations.

[0047] By analogy, after the above operations are performed on all mushroom bed area depth point cloud sets, several mushroom bed area registered depth point cloud sets are obtained for subsequent stitching operations.

[0048] The above transformation error is obtained by the following formula:

[0049] ,

[0050] Depth point cloud collection for the current mushroom bed area s Depth point cloud set of the mushroom bed area aligned with the previous one s-1 The transformation error, N is the total number of point clouds in the depth point cloud set of the current mushroom bed area, Depth point cloud collection for the last registered mushroom bed area s- 1 Middle j The location of the point cloud, The depth point cloud in the current mushroom bed area sMiddle j The point cloud closest to the point cloud i location, is the initial rotation and translation matrix.

[0051] S2. Several RGB images of mushroom bed areas are spliced ​​to obtain a spliced ​​mushroom bed RGB image; the background in the spliced ​​mushroom bed RGB image is removed to obtain an Agaricus bisporus image; a connected domain analysis is performed on the Agaricus bisporus image to obtain an initial Agaricus bisporus recognition area; based on the connected domain area and the connected domain aspect ratio, the Agaricus bisporus overlapping area in the initial Agaricus bisporus recognition area is marked; the Agaricus bisporus overlapping area is subjected to Hough transform processing to obtain an Agaricus bisporus segmentation recognition area at the overlap; the Agaricus bisporus segmentation recognition area at the overlap and the non-Agaricus bisporus overlapping area in the initial Agaricus bisporus recognition area form an Agaricus bisporus recognition area set; based on the standard Agaricus bisporus area, the Agaricus bisporus recognition area set is screened to obtain a harvestable Agaricus bisporus recognition area set.

[0052] Based on the pixel information of Agaricus bisporus reflected in the RGB image of the mushroom bed, the background is removed first, and then the occlusion recognition of Agaricus bisporus is performed to obtain the identification area set of Agaricus bisporus that can be picked and meets the picking conditions.

[0053] First, several RGB images of the mushroom bed area are spliced ​​to obtain a spliced ​​mushroom bed RGB image that can reflect the pixel distribution characteristics of the entire mushroom bed, and the background in the spliced ​​mushroom bed RGB image is removed to obtain an Agaricus bisporus image. Specifically, the spliced ​​mushroom bed RGB image is converted to the HSV space, and the difference between Agaricus bisporus and the background is highlighted, and then normalized to obtain a normalized mushroom bed image; the normalized mushroom bed image is binarized, and the pixels corresponding to Agaricus bisporus are white, and the pixels corresponding to the background are black, and the pixels corresponding to Agaricus bisporus are retained to obtain an Agaricus bisporus image.

[0054] Then, the connected domain analysis is performed on the Agaricus bisporus map, the area where the Agaricus bisporus is located is circled, and a preliminary positioning is performed to obtain the initial Agaricus bisporus recognition area.

[0055] Next, based on the connected domain area and the connected domain aspect ratio, the Agaricus bisporus overlapping area in the initial Agaricus bisporus recognition area is marked. Specifically, the Agaricus bisporus where the connected domain area is within the standard blocked Agaricus bisporus area range and the connected domain aspect ratio is within the standard connected domain aspect ratio range is marked as the Agaricus bisporus overlapping area. The connected domain aspect ratio is the ratio of the longest distance (length) between two points in the closed area of ​​the connected domain to the shortest distance (length) between the two points in the direction perpendicular to the longest distance between the two points.

[0056] Subsequently, the overlapping areas of Agaricus bisporus are processed by Hough transform to obtain the Agaricus bisporus segmentation and recognition areas at the overlap. Therefore, the Agaricus bisporus segmentation and recognition areas at the overlap and the non-Agaricus bisporus overlapping areas in the initial Agaricus bisporus recognition areas form an Agaricus bisporus recognition area set.

[0057] Finally, based on the standard pickable Agaricus bisporus area range, the Agaricus bisporus identification area set is screened to obtain the pickable Agaricus bisporus identification area set. Specifically, the areas in the Agaricus bisporus identification area set where the Agaricus bisporus identification area is smaller than the standard Agaricus bisporus area are deleted, that is, the Agaricus bisporus identification areas where the Agaricus bisporus area is too small to meet the picking conditions are deleted, and the non-Agaricus bisporus overlapping areas in the initial Agaricus bisporus identification area are processed by Hough transform, and the non-Agaricus bisporus overlapping areas are marked with circle marks to obtain the pickable Agaricus bisporus identification area set. The effect is shown in the figure. Figure 2 A labeled diagram of Agaricus bisporus that can be picked.

[0058] S3. Remove the background in the spliced ​​mushroom bed depth point cloud model to obtain the Agaricus bisporus depth point cloud model; mark the point cloud corresponding to the harvestable Agaricus bisporus identification area in the Agaricus bisporus depth point cloud model to obtain the harvestable Agaricus bisporus point cloud positioning data.

[0059] Based on the depth information, the background in the spliced ​​mushroom bed depth point cloud model is removed, and the point cloud data corresponding to the harvestable Agaricus bisporus identification area is marked to obtain accurate point cloud positioning data of harvestable Agaricus bisporus.

[0060] First, remove the background in the spliced ​​mushroom bed depth point cloud model to obtain the Agaricus bisporus depth point cloud model. The effect can be seen in Figure 3 A single Agaricus bisporus point cloud after removing the background. Specifically, several mushroom bed area depth maps are spliced ​​to obtain a spliced ​​mushroom bed depth map. Figure 4 ; The pixel points with depth values ​​less than the depth threshold in the spliced ​​mushroom bed depth map are used as background points; the point cloud corresponding to the background points in the spliced ​​mushroom bed depth point cloud model is deleted to obtain the initial screening mushroom bed depth point cloud model; the initial screening mushroom bed depth point cloud model is filtered to smooth the depth point cloud model data to obtain the Agaricus bisporus depth point cloud model.

[0061] Finally, the point cloud data corresponding to the harvestable Agaricus bisporus identification area is marked in the Agaricus bisporus deep point cloud model to obtain the point cloud positioning data of the harvestable Agaricus bisporus.

[0062] This embodiment further provides a target positioning system based on RGB and point cloud information fusion, which is used to implement the above-mentioned target positioning method based on RGB and point cloud information fusion, including:

[0063] The spliced ​​mushroom bed depth point cloud model generation module is used to obtain the RGB images and depth images of several areas of the Agaricus bisporus mushroom bed surface, and obtain several mushroom bed area RGB images and mushroom bed area depth images; based on the several mushroom bed area depth images, obtain respective mushroom bed area depth point cloud sets; after registering the several mushroom bed area depth point cloud sets, splice the mushroom bed area registered depth point cloud sets of adjacent areas to obtain a spliced ​​mushroom bed depth point cloud model;

[0064] The module for generating a set of harvestable Agaricus bisporus identification regions is used for splicing a number of RGB images of mushroom bed regions to obtain a spliced ​​mushroom bed RGB image; removing the background in the spliced ​​mushroom bed RGB image to obtain an Agaricus bisporus image; performing connected domain analysis on the Agaricus bisporus image to obtain an initial Agaricus bisporus identification region; marking the Agaricus bisporus overlapping region in the initial Agaricus bisporus identification region based on the connected domain area and the connected domain aspect ratio; performing Hough transform processing on the Agaricus bisporus overlapping region to obtain an Agaricus bisporus segmentation identification region at the overlap; the Agaricus bisporus segmentation identification region at the overlap and the non-Agaricus bisporus overlapping region in the initial Agaricus bisporus identification region form an Agaricus bisporus identification region set; based on the standard Agaricus bisporus area, the Agaricus bisporus identification region set is screened to obtain an Agaricus bisporus identification region set that can be picked;

[0065] The module for generating point cloud positioning data of harvestable Agaricus bisporus is used to remove the background in the spliced ​​mushroom bed depth point cloud model to obtain the Agaricus bisporus depth point cloud model; the point cloud corresponding to the harvestable Agaricus bisporus identification area is marked in the Agaricus bisporus depth point cloud model to obtain the point cloud positioning data of harvestable Agaricus bisporus.

[0066] This embodiment also provides a target positioning device based on fusion of RGB and point cloud information, including a processor and a memory, wherein the processor implements the above-mentioned target positioning method based on fusion of RGB and point cloud information when executing a computer program stored in the memory.

[0067] This embodiment also provides a computer-readable storage medium for storing a computer program, wherein when the computer program is executed by a processor, the above-mentioned target positioning method based on fusion of RGB and point cloud information is implemented.

[0068] The present embodiment provides a target positioning method based on the fusion of RGB and point cloud information. First, based on several mushroom bed area depth maps on the surface of the Agaricus bisporus mushroom bed, several mushroom bed area depth point cloud sets are obtained, which are spliced ​​after alignment to obtain a spliced ​​mushroom bed depth point cloud model that is convenient for accurately and intuitively reflecting the actual shape and depth changes of the mushroom bed; then, based on the Agaricus bisporus pixel information reflected by the RGB map of the mushroom bed, the background is removed and then the Agaricus bisporus occlusion recognition is performed to obtain a set of Agaricus bisporus identification areas that can be picked and meet the picking conditions; finally, the background in the spliced ​​mushroom bed depth point cloud model is removed based on the depth information, and the point cloud data corresponding to the Agaricus bisporus identification area that can be picked is marked to obtain accurate Agaricus bisporus point cloud positioning data that can be picked, thereby improving the Agaricus bisporus positioning and recognition accuracy.

Claims

1. A target positioning method based on RGB and point cloud information fusion, characterized in that: The following operations are included: S1. Obtain RGB images and depth images of several areas on the surface of the Agaricus bisporus mushroom bed, and obtain several RGB images and depth images of the mushroom bed areas; Based on several mushroom bed area depth maps, the depth point cloud sets of the respective mushroom bed areas are obtained; Specifically, based on the camera matrix, the coordinate value and corresponding depth value of each pixel point in the depth map of the mushroom bed area are converted into point coordinates in three-dimensional space to obtain an initial depth point cloud set; the initial depth point cloud set is subjected to downsampling processing to obtain a downsampled depth point cloud set; the normal of the downsampled depth point cloud set is obtained, and the downsampled depth point cloud set is subjected to least squares processing and nearest neighbor interpolation processing based on the normal to obtain a mushroom bed area depth point cloud set; After registering several mushroom bed area depth point cloud sets, the registered mushroom bed area depth point cloud sets of adjacent areas are spliced ​​to obtain a spliced ​​mushroom bed depth point cloud model; the registration processing operation is specifically as follows: selecting several point clouds from the last registered mushroom bed area depth point cloud set as feature point clouds; In the current mushroom bed area depth point cloud set, point clouds closest to several feature point clouds are obtained, and point cloud pairing is formed with the corresponding feature point clouds; Based on the point cloud pairing, the current rotation and translation matrix is ​​obtained; based on the current rotation and translation matrix, the current mushroom bed area depth point cloud set is subjected to rotation and translation transformation to obtain the current round of updated depth point cloud set; it is determined whether the transformation error between the current round of updated depth point cloud set and the previous registered mushroom bed area depth point cloud set is less than the change error threshold; If it is not less than, the current round of updated depth point cloud set and the previous registered mushroom bed area depth point cloud set perform several point cloud pairing acquisition, rotation and translation matrix acquisition and rotation and translation transformation operations until the transformation error between the updated depth point cloud set and the previous registered mushroom bed area depth point cloud set is less than the change error threshold, and the current registered mushroom bed area depth point cloud set is obtained, and the splicing operation is performed; if it is less than, the current round of updated depth point cloud set is used as the current registered mushroom bed area depth point cloud set, and the splicing operation is performed; S2, several RGB images of the mushroom bed area are spliced ​​to obtain a spliced ​​mushroom bed RGB image; Remove the background from the spliced ​​mushroom bed RGB image to obtain the Agaricus bisporus image; perform connected domain analysis on the Agaricus bisporus image to obtain the initial Agaricus bisporus recognition area; Based on the connected domain area and the connected domain aspect ratio, the Agaricus bisporus overlapping area in the initial Agaricus bisporus recognition area is marked; the Agaricus bisporus overlapping area is processed by Hough transform to obtain the Agaricus bisporus segmentation recognition area at the overlapping area; The overlapping Agaricus bisporus segmentation recognition area and the non-Agaricus bisporus overlapping area in the initial Agaricus bisporus recognition area form an Agaricus bisporus recognition area set; based on the standard Agaricus bisporus area, the Agaricus bisporus recognition area set is screened to obtain the Agaricus bisporus recognition area set that can be picked; S3. Remove the background in the spliced ​​mushroom bed depth point cloud model to obtain the Agaricus bisporus depth point cloud model; mark the point cloud corresponding to the harvestable Agaricus bisporus identification area in the Agaricus bisporus depth point cloud model to obtain the harvestable Agaricus bisporus point cloud positioning data.

2. The target positioning method based on RGB and point cloud information fusion according to claim 1, characterized in that: The transformation error is obtained by the following formula: ， Depth point cloud collection for the current mushroom bed area s Depth point cloud set of the mushroom bed area aligned with the previous one s-1 The transformation error, N is the total number of point clouds in the depth point cloud set of the current mushroom bed area, Depth point cloud collection for the last registered mushroom bed area s-1 Middle j The location of the point cloud, The depth point cloud in the current mushroom bed area s Middle j The point cloud closest to the point cloud i location, is the initial rotation and translation matrix.

3. The target positioning method based on RGB and point cloud information fusion according to claim 1, characterized in that: In S2, the operation of removing the background in the spliced ​​mushroom bed RGB image is as follows: The spliced ​​mushroom bed RGB image is converted into HSV space and then normalized to obtain a normalized mushroom bed image; the normalized mushroom bed image is binarized to retain the corresponding pixels of Agaricus bisporus to obtain an Agaricus bisporus image.

4. The target positioning method based on RGB and point cloud information fusion according to claim 1, characterized in that: In S2, the operation of removing the background in the spliced ​​mushroom bed depth point cloud model to obtain the Agaricus bisporus depth point cloud model is as follows: Several mushroom bed area depth maps are spliced ​​to obtain a spliced ​​mushroom bed depth map; pixels in the spliced ​​mushroom bed depth map whose depth values ​​are less than a depth threshold are used as background points; The point cloud corresponding to the background point in the spliced ​​mushroom bed depth point cloud model is deleted to obtain the initial screening mushroom bed depth point cloud model; the initial screening mushroom bed depth point cloud model is filtered to obtain the Agaricus bisporus depth point cloud model.

5. A target positioning system based on RGB and point cloud information fusion, used to implement the target positioning method based on RGB and point cloud information fusion according to claim 1, characterized in that: include: The spliced ​​mushroom bed depth point cloud model generation module is used to obtain the RGB images and depth images of several areas of the Agaricus bisporus mushroom bed surface, and obtain several mushroom bed area RGB images and mushroom bed area depth images; based on the several mushroom bed area depth images, obtain respective mushroom bed area depth point cloud sets; after registering the several mushroom bed area depth point cloud sets, splice the mushroom bed area registered depth point cloud sets of adjacent areas to obtain a spliced ​​mushroom bed depth point cloud model; A module for generating a set of harvestable Agaricus bisporus identification regions is used to obtain a spliced ​​mushroom bed RGB image by splicing several mushroom bed area RGB images; The background of the spliced ​​mushroom bed RGB image is removed to obtain the Agaricus bisporus image; the connected domain analysis is performed on the Agaricus bisporus image to obtain the initial Agaricus bisporus recognition area; based on the connected domain area and the connected domain aspect ratio, the Agaricus bisporus overlapping area in the initial Agaricus bisporus recognition area is marked; the Agaricus bisporus overlapping area is processed by Hough transform to obtain the Agaricus bisporus segmentation recognition area at the overlap; The overlapping Agaricus bisporus segmentation recognition area and the non-Agaricus bisporus overlapping area in the initial Agaricus bisporus recognition area form an Agaricus bisporus recognition area set; based on the standard Agaricus bisporus area, the Agaricus bisporus recognition area set is screened to obtain the Agaricus bisporus recognition area set that can be picked; The module for generating point cloud positioning data of harvestable Agaricus bisporus is used to remove the background in the spliced ​​mushroom bed depth point cloud model to obtain the Agaricus bisporus depth point cloud model; the point cloud corresponding to the harvestable Agaricus bisporus identification area is marked in the Agaricus bisporus depth point cloud model to obtain the point cloud positioning data of harvestable Agaricus bisporus.

6. A target positioning device based on RGB and point cloud information fusion, characterized in that: It comprises a processor and a memory, wherein when the processor executes the computer program stored in the memory, the target positioning method based on RGB and point cloud information fusion as described in any one of claims 1 to 4 is implemented.

7. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein when the computer program is executed by a processor, the target positioning method based on RGB and point cloud information fusion as described in any one of claims 1 to 4 is implemented.

Images