A 3D object recognition and localization method for complex scenes by associating and fusing point cloud clustering with image texture

Abstract

The present invention discloses a three-dimensional target recognition and positioning method for complex scenes by associating and fusing point cloud clustering and image texture. An image texture data set and a three-dimensional point cloud data set are acquired through a three-dimensional vision sensor; the collected image texture data is preprocessed, and a deep learning training-inference method is used to label, train, and infer the image texture data set to obtain the image position data of the region of interest; using the position of the component in the image as an index, the 3D point cloud data of the component is searched and segmented; the obtained 3D point cloud data is filtered and denoised; based on the current component point cloud, it is judged whether there is a large point cloud cluster around, and the spatial position where the point cloud cluster exists is output. The present invention mainly aims at complex scenes, and adopts technical means of point cloud spatial information clustering, image texture detail extraction, and the association and fusion of the two to achieve high-precision and fast-computing three-dimensional target recognition and positioning.

Description

Technical Field

[0001] The present invention relates to the technical field, and in particular to a three-dimensional object recognition and positioning method for complex scenes that associates and fuses point cloud clustering and image texture. Background Art

[0002] In recent years, with the continuous rapid development and in-depth exploration of the field of computer vision, a series of innovative and efficient object detection methods have emerged in an endless stream, greatly enriching the research connotation of this field and promoting its extensive expansion in practical applications. These methods show remarkable diversity and flexibility in terms of input data modalities, mainly covering three categories: First, object detection based on single-modal images, which makes full use of the rich information in two-dimensional images and realizes the accurate recognition and positioning of target objects through complex algorithm models; Second, single-modal point cloud object detection, which focuses on three-dimensional point cloud data and uses the accurate coordinate information of the point cloud in three-dimensional space to realize the three-dimensional reconstruction and accurate detection of target objects; Third, multi-modal object detection that fuses image and point cloud information, which cleverly fuses two-dimensional images and three-dimensional point cloud data and makes full use of the complementarity of their feature representations, thereby further improving the accuracy and robustness of object detection.

[0003] Single-modal image object detection methods mainly perform object detection and positioning based on 2D or 3D images. Visual features of the image are used to detect target objects. Single-modal image object detection can be further divided into two major categories. One is represented by the Fast Region-based Convolutional Neural Network (Fast R-CNN), which uses a candidate region extractor to generate potential target regions and then classifies the objects in the candidate box regions; the other is represented by YOLO (You Only Look Once), which completes the positioning and classification tasks in one step.

[0004] Single-modal point cloud object detection can be divided into three categories: (1) directly inputting the original point cloud data into the network for end-to-end recognition, such as PointNet performing feature extraction and aggregation on the direct point cloud data; (2) converting the point cloud into an image and then inputting it into the network, such as PointPillars applying traditional image processing techniques and convolutional neural networks on the bird's-eye view; (3) converting the point cloud into a voxel format and then inputting it into the network, such as Voxelnet dividing the point cloud data into regular three-dimensional voxel grids and applying convolutional neural networks on the voxel grids.

[0005] However, in the face of complex environmental perception tasks, a single sensor often fails to meet all requirements. Although binocular cameras can provide rich color and texture information, the depth information is not accurate enough; lidar can provide accurate distance measurement and motion detection, with good anti-interference ability and the ability to adapt to complex environments. However, it is relatively weak in obtaining details such as colors and textures. Multimodal fusion methods fuse the data of different sensors such as cameras and lidar, make full use of the advantages of each sensor, and make up for each other's deficiencies, so as to provide more comprehensive and accurate environmental perception information.

[0006] Multimodal fusion methods can be divided into three categories: (1) Data-level fusion, which directly fuses the raw data of multiple sensors, such as using vector fusion of laser scanner and two short-range radar data; (2) Feature-level fusion, which fuses the feature representations of images and point clouds, and can achieve feature alignment and fusion by sharing weights, such as MV3D, AVOD; (3) Decision-level fusion, which fuses the prediction results of images and point clouds, such as using the Hungarian algorithm to fuse the detection results of YOLOv3 and PointRCNN algorithms.

[0007] Multimodal fusion methods fuse the data of different sensors such as cameras and lidar, make full use of the advantages of each sensor, and make up for each other's deficiencies, so as to provide more comprehensive and accurate environmental perception information. However, images and point clouds have different data structures and feature representation methods. How to perform effective fusion is the key issue of multimodal fusion methods. Summary of the Invention

[0008] Object of the Invention: Aiming at the above problems, the present invention proposes a 3D object recognition and localization method for complex scenes that associates point cloud clustering with image texture fusion.

[0009] Technical Solution:

[0010] The present invention provides a 3D object recognition and localization method for complex scenes that associates point cloud clustering with image texture fusion, including the following steps: Step S1, collecting image data and point cloud data through sensors to form an image data set and a 3D point cloud data set;

[0011] Step S2, processing the collected image data, and using deep learning training-inference methods to label, train, and infer the 2D image data set to obtain the image position data of the region of interest;

[0012] Step S3, using the position of the component in the image as an index to search and segment the 3D point cloud data of the component;

[0013] Step S4, performing filtering and denoising processing on the obtained 3D point cloud data to eliminate the noise and clutter therein;

[0014] Step S5: Starting from an arbitrarily selected random point as the starting point, perform corresponding processing according to different component features;

[0015] Step S6: Judging whether there are point cloud cluster components around with the current component point cloud, and outputting the spatial positions where the point cloud clusters exist.

[0016] Preferably, the specific steps of Step S2 include: first annotating the region of interest through annotation software, converting the annotated image file into a file in the dataset format required for model training, and then using this file as input data to train it through the deep learning training-inference method to obtain a model adapted to the data; inferring the region of interest of the two-dimensional image data based on this model, performing binary processing on this region of interest to obtain a binary image;

[0017] Performing dilation and erosion processing on the binary image. The specific representation of erosion is as follows:

[0018] ;

[0019] ;

[0020] where C represents the erosion operation, (x, y) is the pixel position in image I, and (i, j) is the relative position in the structuring element S;

[0021] The specific representation of dilation is as follows:

[0022] ;

[0023] ;

[0024] where E represents the dilation operation, (x, y) is the pixel position in image I, and (i, j) is the relative position in the structuring element S;

[0025] Traversing the processed binary image from top to bottom, finding the pixel points with a value of 255, arranging the component regions formed by the qualified points one by one according to the eight-neighborhood algorithm, obtaining the specific number of components in the image and the corresponding component numbers, and obtaining the set of position point coordinates and the corresponding position point indices of each component; removing those with the set of position point coordinates less than a certain threshold, so as to obtain the total number of components in the image and the index set of the two-dimensional coordinates of the pixel points corresponding to each component.

[0026] Preferably, in step S3, an index set of the two-dimensional coordinates of the components in the image obtained through step S2 is used as an index to obtain and search for a segmented 3D point cloud data set. There are two optional methods. One is to convert the points in the pixel coordinate system to the world coordinate system to achieve the conversion between the image and the 3D point cloud. The other is to perform a filtering operation on the corresponding point cloud space through the position point index pointID to obtain the point cloud data of the components.

[0027] Preferably, the conversion between the image and the 3D point cloud by converting the points in the pixel coordinate system to the world coordinate system specifically includes: Step S3.1: Convert the obtained two-dimensional coordinate set from the pixel coordinate system to the image coordinate system, where (ui, vi) represents the coordinates of the i-th point in the position point coordinate set pointSet corresponding to the pixel in the v-th row and u-th column of the image; Step S3.2: Convert from the image coordinate system to the camera coordinate system; Step S3.3: Convert from the camera coordinate system to the world coordinate system;

[0028] Through steps S3.1 to S3.3, the conversion between the pixel coordinate system and the world coordinate system is completed through the following formula:

[0029] ;

[0030] where, is the point coordinate in the pixel coordinate system, represents the point where the camera optical center is perpendicular to the point in the image coordinate system, and are the focal lengths of the camera, representing the horizontal and vertical focal lengths respectively, and are the rotation matrix and translation matrix from the camera coordinate system to the world coordinate system respectively, is the 3D point cloud data coordinate in the world coordinate system.

[0031] Preferably, the filtering operation on the corresponding point cloud space through the position point index pointID to obtain the point cloud data of the component specifically includes: Step S3.1: Place the three-dimensional point cloud acquisition device and the two-dimensional image acquisition device in the same space at the same time for data acquisition, and acquire the image data and point cloud data of the area containing the component in the same space at the same time; Step S3.2: Process the image data according to Step S2 to obtain the position point index pointID of the component; Step S3.3: Use the ExtractIndices filter to extract the point cloud subset pointcloudSet according to the original point cloud collected in Step S3.1 and the point index pointID obtained in Step S3.2. The specific steps include: 1) Perform downsampling using the voxel grid downsampling method; 2) Extract the point cloud data subset that conforms to the geometric model using the SAC plane parameter model; 3) Use the negative variable to extract the remaining point cloud of the opposite point cloud set; 4) Use the remaining point cloud as the point cloud to be processed, return to Step 2, and enter the next round of loop.

[0032] Preferably, in Step S4, the 3D point cloud dataset of the component is obtained through Step S3. The obtained 3D point cloud data is subjected to statistical outlier filtering, and a statistical analysis is performed on the neighborhood of each obtained point, and those points that do not meet certain criteria are trimmed, so as to achieve the effect of removing the outliers in the point cloud data and obtain the processed 3D point cloud dataset; The specific steps of performing statistical outlier filtering on the obtained 3D point cloud data include: 1) Traverse the point cloud and find its k-nearest neighbors for each point in the point cloud; 2) Calculate the average distance from this point to the k-nearest neighbor points ; 3) Calculate the average value of the average distance ; and standard deviation ; 4) Set the distance threshold , and its expression is , where is the proportionality factor of the distance threshold, is the standard deviation multiplier, and is a non-zero natural number; 5) Compare the set distance threshold with , and the points exceeding the threshold are marked as outliers and removed, while the points below the threshold are marked as inliers.

[0033] Preferably, in step S5, based on the 3D point cloud data set filtered in step S4, starting from an arbitrarily selected random point, corresponding processing or no processing is performed according to different component features; for those components presenting obvious spherical features, starting from the starting point, using the sphere fitting algorithm, the contour of the sphere is accurately depicted, and its center and radius are determined accordingly; based on this information, the sphere is segmented, cropped or processed in other forms; if the point cloud data contains components with cylindrical features, starting from the starting point, using the cylinder fitting algorithm, the axis and radius of the cylinder are accurately captured, and then the positions of its two ends are determined; based on this information, the cylinder is cut, adjusted or other necessary processing is performed; for components with more complex shapes and more diverse features, machine learning algorithms are used for feature classification and recognition, and refined processing is performed by combining methods such as shape analysis and geometric transformation.

[0034] Preferably, in step S6, calculate the two points with the farthest distance on the transverse section of the current component point cloud, and record this distance as , and synthesize a dynamic sphere with as the diameter, where δ is the range to be searched near the current component point cloud, is greater than 0, use the center point cloud coordinate of the same plane of the component as the center of the sphere of the dynamic sphere, continuously change the position of the center of the dynamic sphere, if it is found that there is a point cloud cluster outside and inside , output the spatial position where the point cloud cluster exists.

[0035] Preferably, the specific steps of step S6 are as follows: 1) Calculate the two points with the farthest distance on the transverse section of the current component point cloud; the component point cloud is P, which contains n points, that is , where ; find the two points with the farthest distance on the transverse section and , such that ;

[0036] 2) Determine the diameter and search range of the dynamic sphere; where ;

[0037] ;

[0038] Synthesize a dynamic sphere with as the diameter, and its radius is R, where is the range to be searched near the current component point cloud;

[0039] 3) Determine the center of the dynamic sphere; use the center point cloud coordinate of the same plane of the component as the center of the sphere of the dynamic sphere :

[0040] ;

[0041] wherein is the mean value of the z coordinate or a certain fixed value;

[0042] 4) Search for the surrounding point cloud clusters; if it is found that outside the there is a point cloud cluster, output the spatial position where the point cloud cluster exists: for a certain center position of the sphere and radius , if there is a point cloud p that satisfies:

[0043] ;

[0044] then output the spatial position where the point cloud cluster exists.

[0045] The present invention has the following beneficial effects compared with the prior art: The present invention mainly aims at complex scenarios, and adopts technical means of clustering point cloud spatial information, extracting image texture details, and associating and fusing the two to achieve high-precision and fast-computing three-dimensional target recognition and positioning; The present invention filters 3D point cloud data through a statistical outlier filtering method, trimming off points that do not meet certain criteria, so as to achieve the effect of removing outliers in the point cloud data; The present invention performs different processes according to different component features, and can capture the complex shapes and internal structures of components more accurately, providing a more refined and reliable data basis for subsequent applications; The present invention is based on technologies in the fields of computer vision, pattern recognition, machine learning, etc., combines the original data of point clouds and image data collected by sensors, uses image texture details to quickly extract the target of interest, filters out unnecessary data information, reduces the amount of point cloud data, and improves the computing speed. This method can achieve precise positioning, shape analysis and fault diagnosis of components in complex scenarios, thereby improving the reliability and safety of components. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a flowchart of a method for three-dimensional target recognition and positioning in a complex scenario by associating and fusing point cloud clustering and image texture according to the present invention;

[0047] Figure 2 is a flowchart of step S2 of a method for three-dimensional target recognition and positioning in a complex scenario by associating and fusing point cloud clustering and image texture according to the present invention;

[0048] Figure 3 is an alternative flowchart of step S3 of a method for three-dimensional target recognition and positioning in a complex scenario by associating and fusing point cloud clustering and image texture according to the present invention;

[0049] Figure 4It is another alternative flowchart of step S3 of the 3D object recognition and positioning method for fusing point cloud clustering and image texture in the complex scene of the present invention;

[0050] Figure 5 It is a flowchart of filtering 3D point cloud data by the statistical outlier filtering method of the present invention;

[0051] Figure 6 It is a flowchart of step S5 of the 3D object recognition and positioning method for fusing point cloud clustering and image texture in the complex scene of the present invention;

[0052] Figure 7 It is a flowchart of the implementation of the actual application of the present invention, where (a) is the collected image data; (b) is the binary image after binarization processing; (c) is the binary image after closing operation; (d) is the contour extraction map; (e) is the original 3D point cloud data; (f) is the fusion result map; (g) is the result map after 3D point cloud filtering; (h) is the map of the search for point cloud clusters;

[0053] Figure 8 It is a data result map of the actual application of the present invention;

[0054] Figure 9 It is a principle block diagram of the 3D object recognition and positioning method for fusing point cloud clustering and image texture in the complex scene of the present invention. Detailed implementation manners

[0055] Many modifications and variations made by those skilled in the art based on the purpose of the present invention fall within the protection scope of the present invention.

[0056] Those skilled in the art of this technology can understand that unless specifically stated, the singular forms "a", "an", "the" and "said" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present invention means the presence of the stated features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or their groups. It should be understood that when an element or component is "connected" to another element or component, it can be directly connected to other elements or components, or there may also be intermediate elements or components. The term "and / or" used herein includes any and all combinations of one or more of the associated listed items.

[0057] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are only a part rather than all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0058] As Figure 1 shown, an embodiment of the present invention provides a method for three-dimensional target recognition and positioning in complex scenes with point cloud clustering and image texture correlation fusion, including:

[0059] Step S1: Collect image data and point cloud data through a sensor to form an image data set and a three-dimensional point cloud data set;

[0060] Step S2: Process the collected image data, and use a deep learning training-inference method to label, train, and infer the 2D image data set to obtain the image position data of the region of interest;

[0061] Step S3: Use the position of the component in the image as an index to search for and segment the 3D point cloud data of the component;

[0062] Step S4: Perform filtering and denoising processing on the obtained 3D point cloud data to eliminate the noise and miscellaneous points therein;

[0063] Step S5: Select an arbitrary random point as the starting point and perform corresponding processing according to different component features;

[0064] Step S6: Use the current component point cloud to determine whether there is a large point cloud cluster (component) around, and output the spatial position where the point cloud cluster exists.

[0065] As Figure 9 shown, it is a principle block diagram of the method for three-dimensional target recognition and positioning in complex scenes with point cloud clustering and image texture correlation fusion of the present invention.

[0066] Further, in step S1, the implementation environment includes a three-dimensional point cloud acquisition device, a two-dimensional image acquisition device, and a computer device. Among them, the three-dimensional point cloud acquisition device can be a lidar scanning device or a binocular camera device, etc., and the two-dimensional image acquisition device can be an optical camera device. It is necessary to ensure that the three-dimensional point cloud acquisition device and the two-dimensional image acquisition device are horizontally installed on the same bracket, and the three-dimensional point cloud acquisition device is installed directly above the two-dimensional image acquisition device, and the three-dimensional point cloud acquisition device and the two-dimensional image acquisition device need to be kept horizontal. The three-dimensional point cloud acquisition device and the two-dimensional image acquisition device can also be any devices integrated in the same device that can realize three-dimensional information scanning or two-dimensional information acquisition. The present invention does not impose any restrictions on this. Relevant personnel can obtain the three-dimensional point cloud data and two-dimensional image information of the target scene including the target object through the three-dimensional point cloud acquisition device and the two-dimensional image acquisition device. According to actual needs, the collected two-dimensional image information is cleaned to remove blurred, repeated or unqualified images, and the images are preprocessed, such as operations of cropping, scaling, rotating, denoising, etc., to improve the image quality. The collected three-dimensional point cloud data is also preprocessed, such as operations of denoising, registration, filtering, segmentation, etc., to remove redundant information and improve the data quality. The preprocessed three-dimensional point cloud data and two-dimensional image information are stored in a specified format and order, thereby constituting an image data set and a three-dimensional point cloud data set.

[0067] Further, in step S2, as Figure 2 shown, its steps include:

[0068] Step S2.1: First, the region of interest is labeled through labeling software, and the labeled image file is converted into a file in a specific data set format required for model training. Then, using this file as input data, it is trained through a deep learning training-inference method to obtain a model adapted to the data. Based on this model, the region of interest of the two-dimensional image data is inferred, and the region of interest is binarized. The gray value of the background region is 0 and it is displayed as black, and the gray value of the region of interest is 255 and it is displayed as white, and it is segmented from the two-dimensional image data to obtain a binary image.

[0069] Specifically, in step S2.1, the deep learning training-inference method implements the Deeplabv3+ inference algorithm based on pytorch. The main body of its Encoder is a DCNN with dilated convolutions, and a common classification network can be adopted. Then there is a spatial pyramid pooling module with dilated convolutions, mainly to introduce multi-scale information. Compared with DeepLabv3, Deeplabv3+ introduces a Decoder module, which further fuses the low-level features and high-level features to improve the segmentation boundary accuracy. Deeplabv3+ introduces the idea of EcoderDecoder based on DilatedFCN. For DilatedFCN, mainly the later blocks of the classification network are modified, and the downsampling layer with stride = 2 is replaced with dilated convolutions. The image dataset of the components is segmented into a training set and a test set in a specific training format. The trained model is loaded, and the vscode programming environment and the python language are used for programming inference. Through inference, a binary image after binarization processing and the pixel positions of the components are obtained.

[0070] Step S2.2: Dilate and erode the binary image. Through dilation, the defective areas in the regions of interest in the binary image are filled, and through erosion, the burrs around the regions of interest are removed. By combining the two, eroding first and then dilating removes the burrs in the regions of interest and retains the original image size; dilating first and then eroding bridges the narrow discontinuities and slender gullies in the regions of interest, eliminates small holes, fills in breaks, and retains the original image size. The specific representation of erosion is as follows:

[0071] ;

[0072] ;

[0073] Among them, C represents the erosion operation, (x, y) is the pixel position in the image, and (i, j) is the relative position in the structuring element S. For each pixel position (x, y) in the image I, the erosion operation finds the minimum value among all the pixels covered by the structuring element S and assigns this minimum value to the pixel at the corresponding position (x, y) in the result image.

[0074] The specific representation of dilation is as follows:

[0075] ;

[0076] ;

[0077] Similarly, E represents the dilation operation, (x, y) is the pixel position in the image, and (i, j) is the relative position in the structuring element S. For each pixel position (x, y) in the image I, the dilation operation finds the maximum value among all the pixels covered by the structuring element S and assigns this maximum value to the pixel at the corresponding position (x, y) in the result image.

[0078] Step S2.3: Traverse the processed binary image from top to bottom to find the pixel points with a value of 255. Arrange the component regions formed by the points that meet the requirements one by one according to the eight-neighborhood algorithm to obtain the specific number of components in the image and the corresponding component numbers. According to the following formula

[0079] ;

[0080] ;

[0081] Obtain the position point index pointID of each component and obtain the set of position point coordinates pointSet of each component. pointSet contains the X coordinate pointX and Y coordinate pointY of each point, where v represents each row of the image, u represents each column of the image, and width represents the width of the image.

[0082] Specifically, in step S2.3, the eight-neighborhood algorithm takes the first pixel point with a gray value of 255 found as the center point. After traversing the eight adjacent pixel points clockwise, it checks the gray value difference between the current pixel point and the pixel points in its eight-neighborhood. If there is a gray value difference, it is considered that the pixel point may be on the edge. Repeat the above process, and use the edge tracking algorithm to connect the detected edge points to form a continuous edge contour. The edge tracking algorithm expands the edge contour by checking whether there are other edge points in the eight-neighborhood of the current edge point. Fill the inside of the contour region according to the obtained contour. The specific method is to uniformly fill the internal region surrounded by the contour with a gray value of 255 (i.e., white) in this way to clearly distinguish and identify each component region in the image.

[0083] Step S2.4: Process each formed pointSet, that is, process each time a component region in the image is identified. When the number of elements in pointSet (i.e., the number of pixel points pointNum) is less than a certain number of pixel points T, regard this pointSet as non-component data and remove this pointSet. Specifically expressed as follows:

[0084] ;

[0085] Step S2.5: By setting boundary conditions, the area formed by pointSet is used as the first component area on the image. Record the component number lineID = 1, and combine pointSet, pointID, and lineID to form the first element of the component two-dimensional coordinate set inspectUnit.

[0086] Specifically, in Step S2.5, the boundary conditions are expressed as follows:

[0087] ;

[0088] Among them, represents the eight pixels adjacent to the central pixel point, represents the gray value of the i-th pixel. The above formula means that when the sum of the gray values of the eight pixel points in the eight-neighborhood is 0, the search for the first component area is completed.

[0089] Repeat Steps S2.3 to S2.5, arrange the component areas formed by pointSet in different regions on the image in sequence, attach the corresponding lineID to each component, form the index set inspectUnit of the two-dimensional coordinates of each component in the 2D graph, and obtain the image position data of each component.

[0090] Furthermore, in Step S3, based on the two-dimensional coordinate set and point index of the component obtained in Step S2, there are two optional methods to search and segment the point cloud data set of the region of interest, as described below.

[0091] One optional method is as Figure 3 shown, and the specific steps are as follows:

[0092] Step S3.1: Since the obtained pointSet is a two-dimensional coordinate set in the pixel coordinate system, first convert the obtained two-dimensional coordinate set from the pixel coordinate system to the image coordinate system, where (u i , v i ) represents the pixel at the v-th row and u-th column in the image corresponding to the i-th point in the two-dimensional coordinate set pointSet, and pointNum is the number of points;

[0093] Specifically, the conversion between the two coordinate systems can be completed through the following formula:

[0094] ;

[0095] Use the matrix representation of homogeneous coordinates:

[0096] ;

[0097] Among them and represent the physical sizes of each pixel in the u-axis and v-axis directions, represent the coordinates in the two-dimensional coordinate set pointSet, represent the point where the optical center is perpendicular to the image coordinate system, represent the coordinates in the image coordinate system corresponding to the pixel points in the two-dimensional coordinate set;

[0098] Step S3.2: Convert from the image coordinate system to the camera coordinate system;

[0099] Specifically, through Step S3.1, the coordinates of the pixel points in the image coordinate system can be obtained , and the conversion from the image coordinate system to the camera coordinate system can be completed using the following formula:

[0100] ;

[0101] where, represents the position of the pixel point in the camera coordinate system, represents the focal length of the camera;

[0102] Step S3.3: Convert from the camera coordinate system to the world coordinate system;

[0103] Specifically, when transforming from the camera coordinate system to the world coordinate system, the object does not deform, and only rotation and translation are required. The conversion from the camera coordinate system to the world coordinate system can be completed using the following formula:

[0104] ;

[0105] where, and are respectively the point in the camera coordinate system and the 3D point cloud data coordinates in the world coordinate system, and R and T are respectively the rotation matrix and translation matrix from the camera coordinate system to the world coordinate system;

[0106] It can be seen from Steps S3.1 to S3.3 that the conversion between the two coordinate systems from the pixel coordinate system to the world coordinate system can be completed through the following formula:

[0107] ;

[0108] where, is the point coordinate in the pixel coordinate system, represents the point where the camera optical center is perpendicular to the image coordinate system, and are the focal lengths of the camera, representing the horizontal and vertical focal lengths respectively, and R and T are respectively the rotation matrix and translation matrix from the camera coordinate system to the world coordinate system, are the 3D point cloud data coordinates in the world coordinate system. According to this formula, the two-dimensional coordinate set obtained in step S2 can be converted into a 3D point cloud data set

[0109] ;

[0110] where i represents the point cloud coordinates at the i-th position, and pointcloudNum represents the number of point clouds.

[0111] Another optional method is as Figure 4 shown, and the specific steps are as follows:

[0112] In step S3, the point cloud data of the corresponding image is obtained through step S1. Since the point index pointID of the component on the binary image is obtained in step S2, the corresponding point cloud space can be filtered by the point index pointID to obtain the point cloud data of the component.

[0113] Step S3.1: Place the three-dimensional point cloud acquisition device and the two-dimensional image acquisition device in the same space at the same time for data acquisition, and acquire the image data and point cloud data of the area containing the component in the same space at the same time;

[0114] Specifically, in step S3.1, a three-dimensional point cloud acquisition device, such as a lidar (LiDAR) or a structured light scanner, emits a laser beam or a structured light pattern. These beams intersect with the surface of the target object and reflect back. By calculating parameters such as the time difference or phase difference of these reflected lights, the device can accurately measure the three-dimensional coordinates of each reflection point, thereby constructing a three-dimensional point cloud map of the target area. These point cloud data depict in detail the shape, size, and spatial position relationship of the object. At the same time, a two-dimensional image acquisition device, such as a high-definition camera, also takes pictures of the same area at the same moment, capturing the two-dimensional image information of the area. These image data not only contain visual features such as the color and texture of the object but also can reflect the layout and relative position of the object on the two-dimensional plane. By synchronously acquiring these two types of data, we can obtain a comprehensive data set that contains both depth information (through point cloud data) and rich texture features (through two-dimensional image data).

[0115] Step S3.2: Process the image data in step S2 to obtain the point index pointID of the component;

[0116] Step S3.3: Filter the obtained point cloud data with pointID to obtain the point cloud data of the component;

[0117] Specifically, in step S3.3, a point cloud subset is extracted by using the ExtractIndices filter based on the original point cloud collected in step S3.1 and the point indices pointID obtained in step S3.2 , and the specific steps for the ExtractIndices filter to extract the point cloud subset are as follows:

[0118] 1) Downsampling is performed using the voxel grid downsampling method:

[0119] First, the original point cloud data is processed by voxel grid downsampling. The purpose of this step is to reduce the number of point cloud data and improve the speed and efficiency of subsequent processing. The voxel grid downsampling method divides the point cloud data into multiple small three-dimensional cubes (voxels) and selects representative points within each voxel to generate the downsampled point cloud data.

[0120] Principle of the voxel grid downsampling method:

[0121] A. Determine the side length of the minimum bounding box :

[0122] ;

[0123] Among them, and are the maximum and minimum values of the minimum bounding box in the x, y, and z directions respectively, is a parameter for correcting the side length.

[0124] B. Calculation formula for the side length d of the voxel grid:

[0125] ;

[0126] ;

[0127] Among them, V is the volume of the small voxel, is a scale factor used to adjust the voxel side length, s is the scale coefficient, and N represents the total number of points in the point cloud data.

[0128] C. Calculate the centroid :

[0129] ;

[0130] Among them, is the coordinate of the i-th point in the voxel grid, and n is the total number of points in the grid. Select the point closest to the centroid and keep it, and delete the other points in the grid to achieve point cloud downsampling.

[0131] 2) The SAC plane parameter model extracts a subset of point cloud data that conforms to this geometric model:

[0132] Use the SAC (Random Sample Consensus) plane parameter model to extract a subset of point cloud data that conforms to this geometric model. The SAC algorithm fits a plane model by randomly selecting a set of points in the point cloud and calculates the parameters of the fitted plane. Then, according to the set threshold, the point cloud data that conforms to this plane model is extracted to form a subset of point cloud data that conforms to the geometric plane. During this process, segmentation algorithms (such as region growing, Euclidean clustering, etc.) can be used to further extract the subset of point cloud data that conforms to the geometric plane, improving the accuracy and integrity of the extraction.

[0133] 3) Use the negative variable to extract the remaining point cloud of the opposite point cloud set:

[0134] By setting the value of the negative variable to True or False, it is possible to select to extract the point cloud data that conforms to the model (negative = False) or does not conform to the model (negative = True).

[0135] 4) Use the remaining point cloud as the point cloud to be processed, return to step 2, and enter the next round of loop:

[0136] Take the remaining point cloud as the new point cloud to be processed, and return to step 2 for the next round of loop processing. In this round of loop, different geometric models (such as spheres, cylinders, etc.) can be selected or the parameters of the SAC algorithm can be adjusted to extract a subset of point cloud data that conforms to the new geometric model. Through multiple loop processes, different geometric features in the original point cloud data can be gradually extracted to form multiple point cloud subsets.

[0137] Furthermore, in step S4, a 3D point cloud dataset of the component is obtained through step S3. The obtained 3D point cloud data is subjected to statistical outlier filtering, and a statistical analysis is performed on the neighborhood of each obtained point, and those points that do not meet certain criteria are trimmed, so as to achieve the effect of removing outliers in the point cloud data and obtain the processed 3D point cloud dataset. The specific implementation is as follows:

[0138] Step S4.1: Perform statistical outlier filtering on the 3D point cloud data obtained in step S3, perform a statistical analysis on the neighborhood of each obtained point, and trim those points that do not meet certain criteria.

[0139] Specifically, in step S4.1, the obtained 3D point cloud data is subjected to statistical outlier filtering, and a statistical analysis is performed on the neighborhood of each point. Based on the distance distribution characteristics from the point to all neighboring points, some outliers that do not meet the requirements are filtered out. As Figure 5 shown, the principle of statistical outlier filtering method for filtering 3D point cloud data:

[0140] 1) Traverse the point cloud and find the k-nearest neighbors for each point in the point cloud;

[0141] 2) Calculate the average distance from this point to its k-nearest neighbor points ;

[0142] 3) Calculate the average value and the standard deviation of the average distance; ;

[0143] The expression for the mean value is

[0144]

[0145] The expression for the standard deviation is

[0146]

[0147] 4) Set the distance threshold , and its expression is

[0148] ;

[0149] In the formula, is the proportionality factor of the distance threshold, is the standard deviation multiplier, which is a non-zero natural number. In the point cloud denoising process, the proportionality factor is introduced to achieve a balance between the denoising effect and the preservation of local features of the point cloud model. When the proportionality factor increases, the distance threshold will increase accordingly, and then the data volume of the point cloud model after denoising will increase; on the contrary, when decreases, the distance threshold will shrink accordingly, resulting in a decrease in the data volume of the point cloud model after denoising. By flexibly adjusting the value of , according to actual needs, while removing noise, the degree of reduction of the point cloud data can be controlled to meet the requirements of different application scenarios;

[0150] 5) Compare the set distance threshold with . Points exceeding the threshold are marked as outliers and removed, while points below the threshold are marked as inliers.

[0151] Further, in step S5, based on the 3D point cloud data set filtered in step S4, starting from an arbitrarily selected random point, corresponding processing or no processing is performed according to different component features. For those components presenting obvious sphere features, starting from the starting point, using the sphere fitting algorithm, the contour of the sphere is accurately depicted, and its center and radius are determined accordingly. With this information, the sphere can be segmented, cropped, or otherwise processed according to actual needs. If the point cloud data contains components with cylinder features, starting from the starting point, using the cylinder fitting algorithm, the axis and radius of the cylinder are accurately captured, and then the positions of its two ends are determined. With this information, the cylinder can be cut, adjusted, or otherwise processed according to actual needs. For components with more complex shapes and diverse features, more flexible processing methods are adopted. This includes using machine learning algorithms for feature classification and recognition, as well as combining methods such as shape analysis and geometric transformation for refined processing. These methods can more accurately capture the complex shapes and internal structures of the components, providing a more refined and reliable data basis for subsequent applications. There is an example that is based on the 3D point cloud data set filtered in step S4, starting from an arbitrarily selected random point, and performing corresponding processing according to different component features, as described below.

[0152] One example is based on the 3D point cloud data of cylinder features. Taking the wires in the converter valve of a substation as an example, if the wire layout design inside the power converter valve is improper, especially when the distance between wires is too close, it is extremely easy to cause discharge phenomena. This will not only damage the equipment insulation, reduce the system efficiency, but also may trigger more serious safety accidents such as short circuits, fires, and even equipment explosions, posing a major threat to personnel safety and the environment. In order to ensure the safety of staff and the safe operation of the power converter valve, safety distance detection is required. Removing the point cloud data at the head and tail of the wire (insulating glue part or non-wire part) can more accurately describe the shape and position of the wire, as Figure 6 shown, and the specific implementation method is as follows:

[0153] Step S5.1: Based on the 3D point cloud data set of the wire filtered in step S4, let the 3D point cloud data set of the filtered wire be , where each point represents a three-dimensional coordinate . Select an arbitrary random point as the starting search point, and search for all points within the range of the neighborhood radius r through the starting search site, which is specifically expressed as follows:

[0154] 1) Randomly select the starting search point:

[0155] ;

[0156] 2) Given the domain radius:

[0157] ;

[0158] 3) Search for points within the neighborhood:

[0159] Define a set to represent all points within a spherical domain centered at with radius r, i.e.:

[0160] ;

[0161] where represents the Euclidean distance, i.e.:

[0162] ;

[0163] Step S5.2: According to the radial characteristics of the wire, find the point p1 that is farthest from the current initial search site among all points within the neighborhood radius of r. Record the position coordinates of point p1, connect the current point to the farthest point p1 to calculate the direction d1. If the direction d1 is along the upward radial characteristic of the wire, then determine the direction d1 as the target direction; otherwise, continue to search for points within the domain radius r from far to near whose directions conform to the upward radial characteristic of the wire. Specifically, it is expressed as follows:

[0164] 1) Find the farthest point p1:

[0165] Among the farthest p1 can be expressed as:

[0166]

[0167] 2) Record the position coordinates of point p1;

[0168] 3) Calculate the direction d1:

[0169] The direction d1 is the vector from to P1 and can be expressed as:

[0170]

[0171] 4) Determine whether the direction d1 is along the upward radial characteristic of the wire. For a function that accepts a direction vector as input and returns a boolean value indicating whether the direction is along the upward radial characteristic of the wire. If is true, then d1 is the target direction; otherwise, continue to search for points within that conform to the upward radial characteristic of the wire.

[0172] Step S5.3: Then, find another point p2 in the opposite direction of d1 from far to near, record the position coordinates of point p2, and calculate the direction d2 based on connecting the current point and point p2. If the direction d2 is along the radial feature of the wire downward, then determine the direction d2 as the target direction; otherwise, continue to find another point in the domain radius r from far to near that is in the opposite direction of d1 and whose point direction conforms to the radial feature of the wire downward. The specific representation is as follows:

[0173] 1) Find the point p2 in the opposite direction of d1. By calculating the angle between the vectors formed by each point in the set and and -d1, and select the point with the smallest angle to achieve:

[0174] A. Calculate the vector set formed by all points in the set and :

[0175] ;

[0176] ;

[0177] where represents the vector formed by the i-th point in the set and .

[0178] B. Calculate the angle set between the vectors and -d1:

[0179] ;

[0180] C. Select the point with the smallest angle (or closest to 180 degrees) as the p2 point:

[0181] ;

[0182] where is a function that accepts the direction vector d1 and the point cloud set and returns the point p2 that is roughly opposite to d1.

[0183] 2) Calculate the direction d2:

[0184] ;

[0185] 3) Determine whether the direction d2 is along the radial feature of the wire downward. Similar to step S5.2.

[0186] Step S5.4: Then, use p1 as the initial search site to conduct a search. Search for all points within a neighborhood radius of r with p1 as the origin. Find the point p3 that has the same target direction as the one being searched for and is the farthest in position from near to far. Update the position coordinate information of point p1 to p3, and calculate the direction d3. If the direction d3 is upward along the radial feature of the wire, then determine the direction d3 as the updated target direction; otherwise, continue to search for points within the neighborhood radius r from far to near whose directions conform to the upward radial feature of the wire. The specific representation is as follows:

[0187] 1) Search for points within the neighborhood:

[0188] ;

[0189] 2) Find the point p3 with the same target direction:

[0190] Similar to step S5.3, select the point with the smallest included angle (or closest to 0 degrees, indicating the same direction) and the farthest position as p3.

[0191] ;

[0192] Among them, is a function that accepts the direction vector d1 and the point cloud set and returns the point p3 that is roughly the same as d1;

[0193] 3) Calculate the direction d3:

[0194] ;

[0195] 4) Update the position coordinate information of point p1:

[0196] ;

[0197] 5) Determine whether the direction d3 is upward along the radial feature of the wire. Similar to step S5.2.

[0198] Step S5.5: Similarly, search for all points within a neighborhood radius of r with p2 as the origin, and determine the farthest point p4 in the same direction as d2. Update the position coordinate information of point p2 to p4, and calculate a new direction d4 by connecting the current point and the farthest point p4. If the direction d4 is downward along the radial feature of the wire, then determine the direction d4 as the target direction; otherwise, continue to find points within the neighborhood radius r from far to near whose directions conform to the downward radial feature of the wire. The specific representation is as follows:

[0199] 1) Search for points within the neighborhood:

[0200] ;

[0201] 2) Find the point p4 with the same target direction:

[0202] Similar to step S5.4, select the point with the smallest included angle (or closest to 0 degrees, indicating the same direction) and the farthest position as p4.

[0203] ;

[0204] Among them, is the included angle between direction d2 and vector , is a preset included angle threshold for determining the standard of the same direction.

[0205] 3) Calculate direction d4:

[0206] ;

[0207] 4) Update the position coordinate information of point p1:

[0208] ;

[0209] 5) Determine whether direction d4 is downward along the radial feature of the wire. Similar to step S5.2.

[0210] Step S5.6: Repeat the above steps until no points are added, and output the position coordinate information stored in p1 and p2 to obtain the cloud position of the head and tail of the wire.

[0211] Step S5.7: Obtain the cloud position of the head and tail of the wire through step S5.6, and within a certain safety threshold range L, remove the data points at the head and tail of the wire to obtain the cloud data with the head and tail of the wire removed.

[0212] Specifically, according to the cloud position of the head and tail of the wire obtained in step S5.6, with the cloud position of the head of the wire as the origin and a certain safety threshold range L as the radius, filter the data within the radius L; similarly, with the cloud position of the tail of the wire as the origin and a certain safety threshold range L as the radius, filter the data within the radius L, which is specifically expressed as follows:

[0213] Let , the safety threshold range is L, and for any point in the point cloud dataset P, then:

[0214] ;

[0215] If , then the point is within the safety threshold range and should be filtered out.

[0216] Furthermore, in step S6, judge whether there is a large point cloud cluster around the current component point cloud and output the detection result;

[0217] Specifically, calculate the two points with the farthest distance on the horizontal cross-section of the current part point cloud, and record this distance as , and synthesize a dynamic sphere with as the diameter, where is the range to be searched near the current part point cloud, must be greater than 0. Take the center point cloud coordinates of the same plane of the part as the center of the sphere, and continuously change the position of the center of the dynamic sphere. If it is found that there is a point cloud cluster inside outside , output the spatial position of the existing point cloud group, which is specifically expressed as follows:

[0218] 1) Calculate the two points with the farthest distance on the horizontal cross-section of the current part point cloud:

[0219] The part point cloud is P, which contains n points, that is , where . Find the two points with the farthest distance and on the horizontal cross-section, such that

[0220] ;

[0221] 2) Synthesize a dynamic sphere:

[0222] ;

[0223] ;

[0224] Take as the diameter to synthesize a dynamic sphere with a radius of R, where is the range to be searched near the current part point cloud.

[0225] 3) Determine the initial center of the sphere:

[0226] Take the center point cloud coordinates of the same plane of the part as the center of the sphere :

[0227] ;

[0228] where is the average value of the z coordinate or a certain fixed value.

[0229] 4) Search for the surrounding point cloud groups; if it is found that there is a point cloud cluster inside outside , output the spatial position of the existing point cloud group:

[0230] For a certain center position and radius , if there exists a point cloud p that satisfies:

[0231] ;

[0232] Then output the spatial position where the point cloud clusters exist.

[0233] It should be noted that the technical solution of the present invention can be further illustrated by the following experiments.

[0234] 1. Experimental background:

[0235] As a key component of the smart grid, the substation undertakes the core functions of power conversion and power transmission. If the wire layout design inside the power change valve is improper, especially when the distance between wires is too close, it is extremely easy to cause discharge phenomena, which will not only damage the equipment insulation, reduce the system efficiency, but also may trigger more serious safety accidents, such as short circuits, fires, and even equipment explosions, posing a major threat to personnel safety and the environment. In order to ensure the safety of staff and the safe operation of the power change valve, safety distance detection is required. Therefore, the method in this article can be used to carry out target detection research on substation wires.

[0236] 2. Experimental conditions:

[0237] This experiment uses python 3.8.18 and the C++ programming language, and is carried out on a computer equipped with the Windows10 system. The operating environment is implemented using Anaconda and PyTorch 2.1.2, and at the same time, the graphics card of NVIDIA GeForce RTX 3080 and CUDA 12.1 are used for computing acceleration, and it is carried out on the VSCode and VS coding debugging platforms.

[0238] 3. Experimental content:

[0239] Experiment 1 uses a device that integrates a three-dimensional point cloud acquisition device and a two-dimensional image acquisition device on the same device to achieve three-dimensional information scanning and two-dimensional information acquisition. Through this device, the original image data and the original three-dimensional point cloud data in the same space and at the same time can be obtained, and these data constitute an image data set and a three-dimensional point cloud data set. The collected image data is as shown in Figure 7 (a), and the three-dimensional point cloud data is as shown in Figure 7 (e).

[0240] Experiment 2, Deep learning training - The inference method is to implement the Deeplabv3+ inference algorithm based on pytorch. The model is trained using the SGD optimizer, with the initial learning rate set to 0.01, the momentum set to 0.9, and the weight decay set to 0.0005. The training ran for 20,000 iterations, and the model was validated every 2,000 iterations. The labelme software was used to annotate the wires in the collected images, and the annotated image data was stored in the VOC data format, divided into a training set and a test set. The image data processed as above was used as the input for the deep learning training - inference method to carry out the training phase of the training and inference process. The trained model was tested using the pictures in the test set. The aAcc (overall accuracy) of the test was 97.65, the mAcc (average accuracy) was 87.46, and the test time was 0.1889s. The test calculation results can verify the feasibility of the technology of using the DeepLabv3+ network for wire pixel - level segmentation. Loading the trained model, using the vscode programming environment and programming in the python language for inference, the binary image after binarization processing and the pixel positions of the wires were obtained through inference, as Figure 7 shown in (b).

[0241] Closing operation was performed on the binary image. First, dilation was carried out, and then erosion was carried out, which bridged the narrow discontinuities and slender gullies of the wires, eliminated small holes, filled in the breaks, and retained the original image size, as Figure 7 shown in (c). By using the eight - neighborhood algorithm, with the first pixel of gray value 255 as the center, the gray - level differences in its eight - neighborhood were checked clockwise to determine the edge points. The edge points were connected using edge tracking to form a contour, and the inside of the contour was filled with a gray value of 255 to distinguish and identify each wire in the image, as Figure 7 shown in (d).

[0242] Experiment 3, for each wire coordinate set, operations were performed to obtain the position indices of the corresponding coordinates in the original three - dimensional point - cloud data. The ExtractIndices filter was used to fuse the point cloud according to the wire indices. The fusion result is as Figure 7 shown in (f).

[0243] Experiment 4, through Experiment 3, the 3D point - cloud data set of the wires was obtained. The obtained 3D point - cloud data was subjected to statistical outlier filtering. A statistical analysis was carried out on the neighborhood of each obtained point, and those points that did not meet certain criteria were trimmed, so as to achieve the effect of removing the outliers in the point - cloud data and obtain the processed 3D point - cloud data set. The filtered result is as Figure 7 shown in (g).

[0244] Experiment 5: Select any random point as the starting point, search for the cloud positions of the head and tail of the wire based on the radial characteristics of the wire, and remove the data points at the head and tail of the wire. Determine whether there are large cloud clusters around based on the current wire cloud. The search for cloud clusters is as shown in Figure 7 (h), and the spatial positions where cloud clusters exist are output as shown in Figure 8 .

[0245] The applicant of the present invention has made a detailed description and illustration of the embodiments of the present invention in combination with the accompanying drawings of the specification. However, those skilled in the art should understand that the above embodiments are only the preferred implementation schemes of the present invention, and the detailed description is only to help readers better understand the spirit of the present invention, rather than a limitation on the protection scope of the present invention. On the contrary, any improvement or modification made based on the spirit of the present invention should fall within the protection scope of the present invention.

Claims

1. A three-dimensional target recognition and positioning method for complex scenes that associates and fuses point cloud clustering and image texture, characterized in that Including the following steps: Step S1: Collect image data and point cloud data through sensors to form an image data set and a three-dimensional point cloud data set; Step S2: Process the collected image data, and use the deep learning training-inference method to perform annotation training and inference on the 2D image data set to obtain the image position data of the region of interest; Step S3: Using the position of the component in the image as an index, search and segment the 3D point cloud data of the component; Step S4: Perform filtering and denoising processing on the obtained 3D point cloud data to eliminate the noise and miscellaneous points therein; Step S5: Select an arbitrary random point as the starting point and perform corresponding processing according to different component features; Step S6: Judging whether there is a point cloud cluster component around according to the current component point cloud, and outputting the spatial position where the point cloud cluster exists; In step S6, calculate the two points with the farthest distance on the transverse section of the current component point cloud, and record this distance as d max , with dia max = d max + δ as the diameter to synthesize a dynamic sphere, where δ is the range to be searched near the current component point cloud, δ > 0, using the center point cloud coordinates of the same plane of the component as the center of the sphere, continuously changing the position of the center of the dynamic sphere. If it is found that there is a point cloud cluster within δ outside d max , output the spatial position of the point cloud cluster The specific steps of step S6 are as follows: 1) Calculate the two points with the farthest distance on the horizontal section of the current component point cloud; the component point cloud is P, which contains n points, that is, P = P{p1, p2,..., p n}, where p i =(x i , y i , z i ); find the two points p max and p min with the farthest distance on the horizontal section, such that 2) Determine the diameter and search range of the dynamic sphere; where dia max = d max + δ, δ > 0 Using dia max synthesize a dynamic sphere with a diameter, where the radius is R, and δ is the range to be searched near the current component point cloud; 3) Determine the center of the dynamic sphere; use the center point cloud coordinates of the same plane of the component as the center of the sphere c0: where mean(z i ) is the mean of the z coordinate or some fixed value; 4) Search for the surrounding point cloud clusters; If it is found that there is a point cloud cluster within δ outside the farthest distance d on the horizontal cross-section of the current part point cloud, output the spatial position where the point cloud cluster exists: For a certain center position c = (x max , y c , z c ) and radius c , if there is a point cloud p that satisfies: If there is a point cloud p, satisfy: ||p - c|| > d max Then output the spatial position where the point cloud cluster exists.

2. The 3D object recognition and localization method for complex scenes with point cloud clustering and image texture association and fusion according to claim 1, characterized in that, The specific content of step S2 includes: first annotating the region of interest through annotation software, converting the annotated image file into a file in the data set format required for model training, and then using this file as input data to perform training on it through the deep learning training-inference method to obtain a model adapted to the data; based on this model, infer the region of interest of the two-dimensional image data, perform binary processing on the region of interest to obtain a binary image; Perform dilation and erosion processing on the binary image. The specific representation of erosion is as follows: Where C represents the erosion operation, (x, y) is the pixel position in the image I, and (i, j) is the relative position in the structuring element S; The specific representation of dilation is as follows: Where E represents the dilation operation, (x, y) is the pixel position in the image I, and (i, j) is the relative position in the structuring element S; Traverse the processed binary image from top to bottom, find the pixel points with a value of 255, arrange the component regions formed by the points that meet the requirements one by one according to the eight-neighborhood algorithm, obtain the specific number of components in the image and the corresponding component numbers, and obtain the position point coordinate set and the corresponding position point index of each component; remove those with the position point coordinate set less than a certain threshold, so as to obtain the total number of components in the image and the index set of the two-dimensional coordinates of the pixel points corresponding to each component.

3. A three-dimensional object recognition and localization method for complex scenes with point cloud clustering and image texture association and fusion according to claim 2, characterized in that In step S3, through the index set of the two-dimensional coordinates of the component obtained in step S2, using this index set of coordinates as an index, obtain the searched and segmented 3D point cloud data set; there are two optional methods. One is to convert the points in the pixel coordinate system to the world coordinate system, so as to realize the conversion between the image and the 3D point cloud; The other is to perform a filtering operation on the corresponding point cloud space through the position point index pointID to obtain the point cloud data of the component.

4. A three-dimensional object recognition and positioning method for complex scenes with point cloud clustering and image texture correlation fusion according to claim 3, characterized in that The conversion between the image and the 3D point cloud by converting the points in the pixel coordinate system to the world coordinate system specifically includes: Step S3.1: Convert the obtained two-dimensional coordinate set pointSet = {(u i , v i ) | i = 1, 2, … pointNum} from the pixel coordinate system to the image coordinate system, where (u i , v i ) represents the pixel at the v-th row and the u-th column in the image corresponding to the i-th point in the position point coordinate set pointSet, and pointNum represents the total number of pixel points; Step S3.2: Convert from the image coordinate system to the camera coordinate system; Step S3.3: Convert from the camera coordinate system to the world coordinate system; Through steps S3.1 to S3.3, the conversion between the two coordinate systems from the pixel coordinate system to the world coordinate system is completed by the following formula: Among them, Z C represents a point on the Z-axis in the camera coordinate system, (u, v) are the point coordinates in the pixel coordinate system, d x and d y represent the physical sizes of each pixel in the u-axis and v-axis directions; f represents the focal length of the camera; (u0, v0) represents the point on the image coordinate system perpendicular to the camera optical center, f x and f y represent the horizontal and vertical focal lengths of the camera respectively, R and T are the rotation matrix and translation matrix from the camera coordinate system to the world coordinate system, (X w , Y w , Z w ) are the 3D point cloud data coordinates in the world coordinate system.

5. A three-dimensional object recognition and localization method for complex scenes with point cloud clustering and image texture association and fusion according to claim 3, characterized in that The filtering operation on the corresponding point cloud space through the position point index pointID to obtain the point cloud data of the component specifically includes: Step S3.1: Place the three-dimensional point cloud acquisition device and the two-dimensional image acquisition device in the same space at the same time for data acquisition, and acquire the image data and point cloud data of the area containing the component in the same space at the same time; Step S3.2: Process the image data according to Step S2 to obtain the position point index pointID of the component; Step S3.3: Use the ExtractIndices filter to extract the point cloud subset pointcloudSet according to the original point cloud collected in Step S3.1 and the point index pointID obtained in Step S3.

2.

6. A three-dimensional object recognition and localization method for complex scenes with point cloud clustering and image texture association and fusion, characterized in that, The specific content of Step S3.3 includes: 1) Perform downsampling using the voxel grid downsampling method; 2) Extract the point cloud data subset that conforms to the geometric model using the SAC plane parameter model; 3) Use the negative variable to extract the remaining point cloud of the opposite point cloud set; 4) Use the remaining point cloud as the point cloud to be processed, return to Step 2, and enter the next round of loop.

7. A three-dimensional target recognition and positioning method for complex scenes with point cloud clustering and image texture association and fusion according to claim 1, characterized in that In step S4, a 3D point cloud dataset of the component is obtained through step S3. The obtained 3D point cloud data is subjected to statistical outlier filtering. A statistical analysis is performed on the neighborhood of each obtained point, and those points that do not meet the standards are trimmed to obtain a processed 3D point cloud dataset. The specific steps of performing statistical outlier filtering on the obtained 3D point cloud data include: 1) Traverse the point cloud dataset and find the k-nearest neighbors for each point in the point cloud dataset; 2) Calculate the average distance d from this point to its k-nearest neighbors i ; 3) Calculate the mean μ and standard deviation σ of the average distance d i ; 4) Set a distance threshold thresh_d, and its expression is thresh_d = β(μ ± k·σ), where β is the proportional factor of the distance threshold, k is the standard deviation multiplier, and k is a non-zero natural number; 5) Compare the set distance threshold thresh_d with d i , and the points exceeding the threshold are marked as outliers and removed, while the points below the threshold are marked as inliers.

8. A three-dimensional target recognition and positioning method for complex scenes that associates and fuses point cloud clustering and image texture, characterized in that In Step S5, based on the 3D point cloud data set filtered in Step S4, starting from an arbitrarily selected random point, corresponding processing is performed according to different component characteristics; for those components presenting obvious spherical characteristics, starting from the starting point, using the sphere fitting algorithm, accurately depict the contour of the sphere, and accordingly determine its center and radius; through this information, perform segmentation and cropping processing on the sphere; if the point cloud data contains components with cylindrical characteristics, starting from the starting point, use the cylinder fitting algorithm to accurately capture the axis and radius of the cylinder, and then determine the positions at both ends; through this information, perform cutting and adjustment processing on the cylinder; for components with more complex shapes and more diverse characteristics, use machine learning algorithms for feature classification and recognition, and combine shape analysis and geometric transformation methods for processing.

Images