Three-dimensional point cloud clustering recognition method and system based on image recognition

Abstract

The present disclosure provides a method and system for three-dimensional point cloud clustering recognition based on image recognition, comprising acquiring image data of an area to be recognized and its corresponding point cloud data; preprocessing the image data and point cloud data; identifying target instances based on the image data and a pre-trained deep learning model; acquiring depth data of the target instances and determining the projected position of the target instances in the point cloud based on the depth data; and visually segmenting the point cloud data at the projected position based on a clustering algorithm to obtain cluster recognition results. This solution achieves cluster recognition of three-dimensional point clouds by integrating optical sensors and lidar, taking advantage of the optical sensor's ease of data acquisition, and employing a deep learning-based visual recognition and depth positioning solution, thereby improving the accuracy and stability of cluster recognition.

Description

Technical Field

[0001] The present disclosure belongs to the technical field of target recognition based on multi-sensor fusion, and in particular relates to a three-dimensional point cloud clustering recognition method and system based on image recognition. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

[0003] With the rapid development of high-precision sensor technology and computer vision, image-based object detection and recognition are becoming increasingly mature. However, in practical applications such as autonomous driving and robotic deployment, since point clouds are closer to the original representation of objects and obtaining only 2D information from images has significant limitations, people are more concerned with the semantic information and location coordinates of point clouds in 3D maps. Therefore, recognition and clustering of 3D point clouds has become very necessary.

[0004] Point cloud clustering is to divide the original point cloud group into a series of point cloud clusters according to certain features. Traditional point cloud clustering methods include the partition-based K-means algorithm, the fuzzy c-means algorithm, the density-based DBSCAN algorithm, the OPTICS algorithm, and the distance-based Euclidean clustering method. The inventors found that traditional clustering methods run fast and can be used to process large amounts of data, but lack semantic information of point clouds, require different thresholds to be set for different environments, and cannot guarantee the same effect, resulting in poor versatility. In the field of three-dimensional deep learning, point cloud clustering methods mainly include PointNet, PointNet++, VoxelNet, etc. However, these methods require a large amount of data to be obtained in advance for training, but the acquisition of point cloud data and the production of data sets are relatively difficult, and the detection accuracy is limited; therefore, point cloud clustering methods based on deep learning are not widely used. Summary of the Invention

[0005] In order to solve the above problems, the present disclosure provides a three-dimensional point cloud clustering recognition method and system based on image recognition. The solution integrates optical sensors and lidars, takes advantage of the fact that optical sensors can easily acquire data, and adopts a deep learning-based visual recognition and depth positioning solution to realize clustering recognition of three-dimensional point clouds, thereby improving the accuracy and stability of clustering recognition.

[0006] According to a first aspect of an embodiment of the present disclosure, a three-dimensional point cloud clustering recognition method based on image recognition is provided, comprising:

[0007] Obtain image data of the area to be identified and its corresponding point cloud data;

[0008] Preprocessing the image data and point cloud data;

[0009] Identify target instances based on the image data and a pre-trained deep learning model;

[0010] Acquire depth data of the target instance, and determine a projection position of the target instance in the point cloud based on the depth data;

[0011] At the projection position, the point cloud data is visually segmented based on a clustering algorithm to obtain a cluster recognition result.

[0012] Furthermore, the image of the area to be identified and its corresponding point cloud data are obtained by using a laser radar and a binocular camera with specific relative positions.

[0013] Furthermore, the pre-trained deep learning model adopts the YOLACT network model, and its training process is: obtaining monocular image data from a binocular camera, and annotating the monocular image data with target instances to form a training set; using the training set to train the deep learning model, and using the trained deep learning model to identify target instances.

[0014] Furthermore, the determination of the projection position is specifically as follows: determining the position information of the target instance relative to the acquisition device based on the depth data of the target instance, and determining the position of the target instance projected into the point cloud data based on the position information.

[0015] Furthermore, the recognition result of the target instance using the deep learning model includes the location information and semantic information of the target instance in the image.

[0016] According to a second aspect of an embodiment of the present disclosure, a three-dimensional point cloud clustering recognition system based on image recognition is provided, comprising:

[0017] A data acquisition unit, which is used to acquire image data of the area to be identified and its corresponding point cloud data;

[0018] a preprocessing unit, configured to preprocess the image data and point cloud data;

[0019] a target instance recognition unit, configured to recognize target instances based on the image data and a pre-trained deep learning model;

[0020] a projection position determining unit, configured to obtain depth data of a target instance and determine a projection position of the target instance in the point cloud based on the depth data;

[0021] A cluster recognition unit is used to perform visual segmentation on the point cloud data at the projection position based on a clustering algorithm to obtain a cluster recognition result.

[0022] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising a memory, a processor, and a computer program stored and running on the memory, wherein when the processor executes the program, a three-dimensional point cloud clustering recognition method based on image recognition is implemented.

[0023] According to a fourth aspect of an embodiment of the present disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, a three-dimensional point cloud clustering recognition method based on image recognition is implemented.

[0024] Compared with the prior art, the present invention has the following advantages:

[0025] (1) The present disclosure provides a three-dimensional point cloud clustering recognition method and system based on image recognition. The scheme integrates optical sensors and lidars, takes advantage of the fact that optical sensors can easily acquire data, and adopts a deep learning-based visual recognition and depth positioning scheme to achieve clustering recognition of three-dimensional point clouds, thereby improving the accuracy and stability of clustering recognition.

[0026] (2) The proposed solution makes comprehensive use of cameras and lidar to cluster the three-dimensional point cloud of the instance, which significantly reduces the amount of computation compared to clustering methods that directly process point clouds. The recognition part calls the YOLACT (You Only LookAt Coefficients) network and adopts the KD-Tree structure for clustering, which significantly improves the real-time performance.

[0027] (3) The proposed scheme processes only the point cloud within the camera's field of view and clusters only the target instances identified in the image. Compared with traditional clustering methods, the clustering accuracy is improved and semantic information that traditional clustering methods do not have is provided.

[0028] (4) The scheme uses the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) method to cluster point clouds at specific locations without specifying the shape of the clusters, thereby improving the ability to resist noise. The method has good robustness in complex scenarios.

[0029] Advantages of additional aspects of the present disclosure will be given in part in the following description and in part will become apparent from the following description or learned through practice of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings, which constitute a part of the present disclosure, are used to provide a further understanding of the present disclosure. The exemplary embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation to the present disclosure.

[0031] Figure 1 This is an overall flow chart of the three-dimensional point cloud clustering recognition method based on image recognition described in Example 1 of the present disclosure.

[0032] Figure 2 This is a flow chart of measuring the distance of a target instance by a binocular camera as described in the first embodiment of the present disclosure.

[0033] Figure 3 This is a conversion relationship diagram of the actual position of the target instance described in the first embodiment of the present disclosure from the camera to the radar point cloud.

[0034] Figure 4 This is a schematic diagram of the layout of the laser radar and binocular camera described in Example 1 of the present disclosure.

[0035] Among them, 1. Top heat sink; 2. LiDAR; 3. Bottom heat sink; 4. Binocular camera; 5. Fixed bracket. DETAILED DESCRIPTION

[0036] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

[0037] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present disclosure belongs.

[0038] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0039] In the absence of conflict, the embodiments of the present disclosure and the features thereof may be combined with each other.

[0040] Example 1:

[0041] The purpose of this embodiment is to provide a three-dimensional point cloud clustering recognition method based on image recognition.

[0042] A three-dimensional point cloud clustering recognition method based on image recognition, comprising:

[0043] Obtain image data of the area to be identified and its corresponding point cloud data;

[0044] Preprocessing the image data and point cloud data;

[0045] Identify target instances based on the image data and a pre-trained deep learning model;

[0046] Acquire depth data of the target instance, and determine a projection position of the target instance in the point cloud based on the depth data;

[0047] At the projection position, the point cloud data is visually segmented based on a clustering algorithm to obtain a cluster recognition result.

[0048] Furthermore, the acquisition of images of the area to be identified and its corresponding point cloud data utilizes a laser radar and a binocular camera with specific relative positions. Specifically, the laser radar and binocular camera are fixed relative to each other, maintaining a fixed relative pose. The laser radar requires a multi-line laser radar, and the binocular camera must be capable of capturing RGB images. The binocular camera is calibrated to obtain its intrinsic parameters and distortion coefficients. The binocular camera and laser radar are then jointly calibrated to obtain their initial extrinsic parameters.

[0049] Furthermore, the pre-trained deep learning model adopts a network model, and its training process is as follows: obtain monocular image data from a binocular camera, and annotate the monocular image data with target instances to form a training set; use the training set to train the deep learning model, and use the trained deep learning model to identify target instances. Specifically, the left (or right) image data of the binocular camera is used to identify target instances, and a data acquisition system is formed between the calibrated sensors. Obtain enough images of instances to be clustered for point cloud, annotate the dataset according to the form of the COCO dataset, use the YOLACT network to train the images of the instances to be identified, and obtain a training model.

[0050] Furthermore, the determination of the projection position is specifically as follows: determining the position information of the target instance relative to the acquisition device based on the depth data of the target instance, and determining the position of the target instance projected into the point cloud data based on the position information.

[0051] Furthermore, the recognition result of the target instance using the deep learning model includes the location information and semantic information of the target instance in the image.

[0052] Specifically, for ease of understanding, the solution disclosed herein is described in detail below with reference to the accompanying drawings and specific examples:

[0053] This disclosure uses optical sensors to obtain external image information and lidar to acquire point cloud information. Optical sensors have the advantages of small size, easy deployment, and low cost, while lidar has the advantages of high resolution, strong anti-interference ability, and unaffected by light. By integrating lidar and optical sensors for target detection, a three-dimensional point cloud clustering recognition method based on image recognition is proposed, including preparatory work and recognition process, wherein the preparatory work is as follows:

[0054] Fix the laser radar and binocular camera at a certain relative position so that they have a fixed relative posture. Use a black and white checkerboard to calibrate the binocular camera to obtain the left and right intrinsic parameter matrix K of the camera. l , K r , distortion coefficients (k1, k2, p1, p2, k3) L 、(k1,k2,p1,p2,k3) R , focal length f; perform joint calibration on the binocular camera and lidar to obtain the external parameter matrix of the lidar relative to the left and right objects of the binocular camera, which are recorded as Use the left (or right) image data of the binocular camera to perform instance recognition, and the calibrated sensors form a data acquisition device; use the YOLACT network to train the image of the instance to be recognized to obtain the trained model weights.

[0055] The identification process mainly includes:

[0056] Step (1): Obtain image data collected by the monocular camera in the binocular camera, and perform dedistortion processing based on the pre-calibrated camera internal parameters, distortion coefficients, and distortion model to obtain a corrected image.

[0057] Specifically, the image data collected by one of the binocular cameras is obtained, and the rectified image is obtained by dedistortion processing based on the pre-calibrated camera intrinsic parameters, distortion coefficients, and distortion model: the dedistorted image data is represented by (u, v), and the dedistorted image data is represented by (u′, v′). The dedistorted image can be obtained from the distortion model. The specific formula is as follows:

[0058]

[0059] Step (2): Obtain the original point cloud data of the lidar. In order to reduce the amount of calculation, geometric constraints are applied to the point cloud, and only the point cloud within the camera's field of view is retained. The RANSAC segmentation method is used to remove the ground points to obtain the filtered point cloud.

[0060] Specifically, the original point cloud data of the laser radar is obtained. In order to reduce the amount of calculation, only the point cloud within the field of view of the camera is retained: the point cloud in the laser radar coordinate system is represented as P L, then the point can be expressed in the camera coordinate system as The position in the image is By judging (x u ,y v ) is in the image range (u′, v′) to know whether the point is within the camera's field of view.

[0061] The ground point cloud is initially filtered out according to the installation height of the lidar, and then the RANSAC algorithm is used to remove the ground points: set the confidence level P, the number of iterations k, and the distance threshold d, and randomly select three points p1(x1, y1, z1), p2(x2, y2, z2), and p3(x3, y3, z3) from the point cloud after the initial screening, and calculate a plane model Ax+By+Cz+D=0. The distance from any point p(x, y, z) to the plane is When the point is considered to be in the plane, all points are traversed, and the probability of the point in the plane can be expressed as P', and the number of update iterations k = log 1-P (1-P′ 3 ), the plane with the most points among all planes is considered to be the ground point. When the number of iterations is greater than k, it exits and obtains the filtered point cloud.

[0062] Step (3): Input the corrected image into the trained YOLACT network, import the trained model weights, and obtain the segmentation instance through semantic segmentation; at the same time, in order to ensure real-time performance, deploy graphics card acceleration during image processing, and display the semantic information and 2D detection box of the detected target on the image.

[0063] Step (4): The target instance to be detected is passed into the depth detection program, and the average value is obtained through multiple frames of images to obtain the average depth of the target instance. The relative position of the target instance relative to the acquisition system is obtained, and the relative distance is displayed on the image with the detection frame.

[0064] Specifically, the pixel position of the target instance image to be detected is passed to the depth calculation program: the underlying driver is called to obtain one of the segmented instances, and an index value is obtained for it. The depth of each pixel in the segmented instance area is calculated, and the sum is averaged. According to the index value, the depth of the segmented instance is calculated by continuously calculating multiple frames of images, and the average value is obtained to obtain the average depth of the segmented instance. The position coordinate of the center point of the target instance in the camera coordinate system is obtained and marked as P C =(x C ,y C , z C ) and displays the relative distance onto the image with the detection box.

[0065] Step (5): Project the target instance position into the point cloud, cluster the point cloud at this position using the DBSCAN clustering method to form a 3D detection box, display the identified semantic information in the 3D detection box, and output the clustering result.

[0066] Specifically, the target instance position is projected into the point cloud. The position of the instance identified by the image in the LiDAR coordinate system can be obtained by The density-based DBSCAN method is used to visually segment the point cloud at this location:

[0067] To improve processing speed, L The nearest neighbor query algorithm based on KD-Tree is used: set the distance threshold d max The point cloud is three-dimensional data. The values ​​of n points in the x, y, and z directions are arranged from small to large to obtain the set {K1, K2, ..., K n} x , {K1, K2, ..., K n} y , {K1, K2, ..., K n} z , taking the x direction as an example, the division threshold m can be obtained by the following formula:

[0068]

[0069] By P L Start by comparing from the root node of KD-Tree. L =(x, y, z), when x < m, visit the left subtree, otherwise visit the right subtree until reaching the leaf node, at which point the leaf node is the current nearest neighbor, which is the same as P L The distance is d min , visit all search paths, and find P in the process L Distance less than d min The nodes are updated to the nearest neighbor points until all paths are passed. Use KD-Tree to find k nodes away from P L The nearest points are clustered using DBSCAN density. A minimum number of points in a cluster, MinPts, is set. These k points are traversed. If a point has more points than MinPts within the radius Eps, it is considered a core point. A new cluster is created and all neighboring points are added to the cluster. At the same time, for the core point in the neighborhood, its neighboring points are recursively added to the cluster. The clustering process is exited until there are no more points to add to the cluster. A 3D preselection box of the point cloud is formed, and the semantic information identified in step (3) is annotated in the 3D preselection box. The clustering result is output.

[0070] Example 2:

[0071] The purpose of this embodiment is to provide a three-dimensional point cloud clustering recognition system based on image recognition.

[0072] A three-dimensional point cloud clustering recognition system based on image recognition, comprising:

[0073] A data acquisition unit, which is used to acquire image data of the area to be identified and its corresponding point cloud data;

[0074] a preprocessing unit, configured to preprocess the image data and point cloud data;

[0075] a target instance recognition unit, configured to recognize target instances based on the image data and a pre-trained deep learning model;

[0076] a projection position determining unit, configured to obtain depth data of a target instance and determine a projection position of the target instance in the point cloud based on the depth data;

[0077] The cluster recognition unit is used to perform visual segmentation on the point cloud data based on a clustering algorithm to obtain a cluster recognition result.

[0078] In further embodiments, there is also provided:

[0079] An electronic device includes a memory and a processor, and computer instructions stored in the memory and executed by the processor. When the computer instructions are executed by the processor, the method described in Example 1 is performed. For the sake of brevity, no further details are given here.

[0080] It should be understood that in this embodiment, the processor may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), off-the-shelf field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0081] The memory may include a read-only memory and a random access memory, and provides instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0082] A computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by a processor, the method described in embodiment 1 is performed.

[0083] The method in Example 1 can be directly implemented as being executed by a hardware processor, or by a combination of hardware and software modules within the processor. The software module can be located in a storage medium well-established in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not given here.

[0084] Those skilled in the art will appreciate that the units, i.e., algorithm steps, of the various examples described in conjunction with this embodiment can be implemented using electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0085] The three-dimensional point cloud clustering recognition method and system based on image recognition provided in the above embodiment can be implemented and has broad application prospects.

[0086] The foregoing description is merely a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Those skilled in the art will readily appreciate that various modifications and variations are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present disclosure shall be included within the scope of protection of the present disclosure.

Claims

1. A three-dimensional point cloud clustering recognition method based on image recognition, characterized in that: include: Obtain image data of the area to be identified and its corresponding point cloud data; The image data and point cloud data are preprocessed; wherein, the preprocessing includes performing RANSAC segmentation on the point cloud data to remove ground points to obtain filtered point cloud data; specifically, the ground point cloud data is preliminarily filtered out according to the installation of the lidar, and the ground points are removed using the RANSAC algorithm, that is, the confidence level P, the number of iterations k, and the distance threshold d are set, and three points are randomly selected from the preliminarily filtered point cloud data. , , , calculate a plane model , take any Distance to plane When the selected point is considered to be in the plane, all points are traversed and the probability of the point in the plane is expressed as , update the number of iterations , the plane with the most points among all planes is the ground point. When the number of iterations is greater than k, exit and obtain the filtered point cloud data; Based on the image data and a pre-trained deep learning model, target instances are identified; wherein the pre-trained deep learning model adopts a YOLACT network model, and its training process is as follows: obtaining monocular image data from a binocular camera, and annotating the monocular image data with target instances to form a training set; using the training set to train the deep learning model, and using the trained deep learning model to identify target instances; Acquire the depth data of the target instance, and determine the projection position of the target instance in the point cloud based on the depth data; acquire the depth data of the target instance based on the depth detection method of the target instance, and average the depth of multiple frames of images to obtain the average depth of the target instance; pass the pixel position of the target instance image to be detected into the depth calculation program: call the underlying driver to obtain one of the segmented instances, mark it to obtain an index value; calculate the depth of each pixel in the segmented instance area, sum and average it; continuously calculate multiple frames of images according to the index value to obtain the depth of the segmented instance, obtain the average value, obtain the average depth of the segmented instance, obtain the position coordinates of the center point of the target instance in the camera coordinate system, and display the relative distance from the detected target instance to the camera on the image with the detection frame; At the projection position, visually segmenting the point cloud data based on a clustering algorithm to obtain a cluster recognition result; The determination of the projection position is specifically as follows: determining the position information of the target instance relative to the acquisition device based on the depth data of the target instance, and determining the position of the target instance projected into the point cloud data based on the position information; wherein the position of the target instance is projected into the point cloud, and the position of the target instance is determined by It is concluded that, is the position coordinate of the center point of the target instance in the camera coordinate system; The nearest neighbor query algorithm based on KD-Tree is used to set the distance threshold , point cloud is three-dimensional data, arrange the values ​​of n points in x, y, and z directions from small to large, and get the set , taking the x direction as an example, the division threshold m can be obtained by the following formula: Depend on Start by comparing from the root node of KD-Tree. , When , visit the left subtree, otherwise visit the right subtree, until reaching the leaf node, at this time the leaf node is the current nearest neighbor, and its The distance is , visiting all search paths, and finding Distance less than The node is updated to the nearest neighbor until all paths are passed; k nodes are found through KD-Tree. The nearest point is clustered using DBSCAN density among k points. A minimum number of points for a cluster, MinPts, is set. The k points are traversed. If a point has more points than MinPts within the radius Eps, the point is considered to be a core point. A new cluster is created and all neighboring points are added to the cluster. At the same time, for the core point among the neighboring points, the neighboring points of the core point are recursively added to the cluster until no points can be added to the cluster. The clustering process is exited to form a 3D pre-selected box of the point cloud. The recognized semantic information is marked in the 3D pre-selected box and the clustering result is output.

2. The three-dimensional point cloud clustering recognition method based on image recognition according to claim 1, characterized in that: The image of the area to be identified and its corresponding point cloud data are obtained by using a laser radar and a binocular camera with relative positions.

3. The three-dimensional point cloud clustering recognition method based on image recognition according to claim 1, characterized in that: The preprocessing also includes performing dedistortion processing on the obtained image data to obtain a corrected image.

4. The three-dimensional point cloud clustering recognition method based on image recognition according to claim 1, characterized in that: The recognition results of the target instance using the deep learning model include the location information and semantic information of the target instance in the image.

5. A three-dimensional point cloud clustering recognition system based on image recognition, characterized in that: include: A data acquisition unit, which is used to acquire image data of the area to be identified and its corresponding point cloud data; A preprocessing unit is used to preprocess the image data and point cloud data; wherein, the preprocessing includes performing a RANSAC segmentation method on the point cloud data to remove ground points to obtain filtered point cloud data; specifically, the ground point cloud data is preliminarily filtered out according to the installation of the laser radar, and the ground points are removed using the RANSAC algorithm, that is, the confidence level P, the number of iterations k, and the distance threshold d are set, and three points are randomly selected from the preliminarily filtered point cloud data. , , , calculate a plane model , take any Distance to plane When the selected point is considered to be in the plane, all points are traversed and the probability of the point in the plane is expressed as , update the number of iterations , the plane with the most points among all planes is the ground point. When the number of iterations is greater than k, exit and obtain the filtered point cloud data; A target instance recognition unit, which is used to recognize target instances based on the image data and a pre-trained deep learning model; wherein the pre-trained deep learning model adopts a YOLACT network model, and its training process is as follows: obtaining monocular image data from a binocular camera and annotating the monocular image data with target instances to form a training set; using the training set to train the deep learning model, and using the trained deep learning model to recognize target instances; A projection position determination unit is used to obtain depth data of a target instance and determine the projection position of the target instance in the point cloud based on the depth data; the acquisition of the depth data of the target instance is based on a depth detection method for the target instance, and the depth of multiple frames of images is averaged to obtain the average depth of the target instance; the pixel position of the target instance image to be detected is passed into the depth calculation program: the underlying driver is called to obtain one of the segmented instances, and it is annotated to obtain an index value; the depth of each pixel in the segmented instance area is calculated, and the sum is obtained after averaging; the depth of the segmented instance is continuously calculated over multiple frames of images according to the index value, and the average value is obtained to obtain the average depth of the segmented instance, and the position coordinates of the center point of the target instance in the camera coordinate system are obtained, and the relative distance from the detected target instance to the camera is displayed on the image with the detection frame; A cluster recognition unit, configured to perform visual segmentation on the point cloud data at the projection position based on a clustering algorithm to obtain a cluster recognition result; The determination of the projection position is specifically as follows: determining the position information of the target instance relative to the acquisition device based on the depth data of the target instance, and determining the position of the target instance projected into the point cloud data based on the position information; wherein the position of the target instance is projected into the point cloud, and the position of the target instance is determined by It is concluded that is the position coordinate of the center point of the target instance in the camera coordinate system; The nearest neighbor query algorithm based on KD-Tree is used to set the distance threshold , point cloud is three-dimensional data, arrange the values ​​of n points in x, y, and z directions from small to large, and get the set , taking the x direction as an example, the division threshold m can be obtained by the following formula: Depend on Start by comparing from the root node of KD-Tree. , When , visit the left subtree, otherwise visit the right subtree, until reaching the leaf node, at this time the leaf node is the current nearest neighbor, and its The distance is , visiting all search paths, and finding Distance less than The node is updated to the nearest neighbor until all paths are passed; k nodes are found through KD-Tree. The nearest point is clustered using DBSCAN density among k points. A minimum number of points for a cluster, MinPts, is set. The k points are traversed. If a point has more points than MinPts within the radius Eps, the point is considered to be a core point. A new cluster is created and all neighboring points are added to the cluster. At the same time, for the core point among the neighboring points, the neighboring points of the core point are recursively added to the cluster until no points can be added to the cluster. The clustering process is exited to form a 3D pre-selected box of the point cloud. The recognized semantic information is marked in the 3D pre-selected box and the clustering result is output.

6. An electronic device comprising a memory, a processor, and a computer program stored and running on the memory, characterized in that: When the processor executes the program, it implements the three-dimensional point cloud clustering recognition method based on image recognition as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the three-dimensional point cloud clustering recognition method based on image recognition as described in any one of claims 1 to 4 is implemented.

Images