An intelligent detection method for facade protrusions based on multi-sensor tight coupling

Abstract

The present invention discloses an intelligent detection method for facade protrusions based on multi-sensor tight coupling. Multimodal data including point cloud and image are collected for the facade to be detected, and then the image is panoramically segmented to obtain semantic information, which is transferred to the point cloud. The point cloud is downsampled, and density clustering is performed in combination with semantic information to calculate the projection confidence vector. The random sample consensus algorithm is used to calculate the plane equation of the facade, and the bounding box of the protrusion is calculated to complete the rough estimation of the protrusion based on the point cloud. The surface equation of the protrusion is calculated using the point cloud of the protrusion, and the surface equation of the protrusion is transformed into the camera coordinate system. Using the pixel position of the protrusion in the image and combining with the facade equation, the boundary of the protrusion in space is fitted. According to the projection confidence vector, the rough estimation result of the point cloud and the boundary fitted based on the image are adjusted, and the two types of results are combined and filtered for coupling to complete the precise detection of the protrusion and obtain the protrusion queue.

Description

Technical Field

[0001] The present invention relates to the field of intelligent environment perception, and particularly to an intelligent detection method for facade protrusions based on multi-sensor tight coupling. Background Art

[0002] At present, the cleaning of tunnel walls mainly adopts the manual cleaning method, which requires a large amount of time and human resources, and there are great safety hazards. Compared with traditional manual cleaning, the wall washing vehicle can complete the work more efficiently and quickly. The manual operation of the wall washing vehicle is mainly completed by a team consisting of a driver and a worker. The driver is responsible for driving the wall washing vehicle, and the worker sits in the operation room and manipulates the cleaning boom of the wall washing vehicle through a remote control to complete the work of cleaning the stains on the outer surface of the facade. However, due to the complex operating environment of the wall washing vehicle, there are often various protrusions on the tunnel and sound insulation screen facades, which bring great difficulties and risks to the cleaning operation. If the cleaning worker is inattentive or makes an operation error, it may cause the end of the boom to collide with the facade, thereby causing impact damage to the boom system and the facade, and even causing a car accident in severe cases.

[0003] In automated cleaning work, currently, a cleaning equipment based on the combination of lidar and camera imaging is considered. It can automatically adjust the cleaning strategy by perceiving the distance and shape information of the surrounding environment, avoid hitting protrusions, and ensure the cleaning effect. This combination can not only improve the cleaning efficiency and quality, but also greatly reduce the cleaning cost, reduce the labor intensity and safety risks of cleaning workers. However, due to the different characteristics of facades in different scenarios, such as different curvatures in the horizontal and vertical directions, different transparencies of facades; when the size of the protrusion is small and the light reflectivity of the color and material is low, the information of the protrusion obtained by the lidar is weak, and it is more sensitive to environmental changes; for facades with high transparency such as sound insulation screens, most of the laser beams emitted by the lidar will penetrate the glass, resulting in the return of objects behind the sound insulation screen or no return signal; the water mist sprayed during vehicle operation will move forward in the direction of the vehicle under the drive of the rotary brush and enter the field of view of the lidar, and the dense water mist will reflect the radar signal or float around the protrusion. The camera is sensitive to texture information, difficult to use in scenarios such as tunnels, and unable to obtain spatial position information. Summary of the Invention

[0004] In view of the above problems, the present invention proposes an intelligent detection method for facade protrusions based on multi-sensor tight coupling. This method can accurately identify various facade protrusions under different scenarios and thin water mist conditions. Applying it to the facade cleaning equipment can obtain reliable and stable detection results of facade protrusions;

[0005] In order to achieve the above technical objectives, the technical solution of the present invention is:

[0006] An intelligent detection method for facade protrusions based on multi-sensor tight coupling, comprising the following steps:

[0007] S1. Collect multi-modal data including point cloud and image for the facade to be detected, then use a deep learning network to perform panoramic segmentation on the image to obtain semantic information, and transfer the semantic information to the point cloud;

[0008] S2. Downsample the point cloud, perform density clustering in combination with semantic information, calculate the projection confidence vector, use the random sample consensus algorithm to calculate the plane equation of the facade, calculate the bounding box for the protrusion, and complete the rough estimation of the protrusion based on the point cloud;

[0009] S3. Use the point cloud to calculate the surface equation of the protrusion, transform the surface equation of the protrusion into the camera coordinate system, use the pixel position of the protrusion in the image, and combine with the facade equation to fit the boundary of the protrusion in space;

[0010] S4. Adjust the rough estimation result of the point cloud and the boundary fitted based on the image according to the projection confidence vector, perform combined filtering on the two types of results for coupling, complete the precise detection of the protrusion, and obtain the protrusion queue.

[0011] In the method described above, in step S1, the step of using a deep learning network to perform panoramic segmentation on the image to obtain semantic information includes:

[0012] S101. First, correct the distortion of the collected image according to the internal parameter matrix of the camera used to collect the image, then use the pre-trained deep learning network to perform panoramic segmentation on the image, and classify the pixels according to the segmentation result;

[0013] S102. Assign semantic information to each class obtained by segmentation according to the characteristics of the facade working scene, find the classes where the facade and the facade protrusions are located, and perform dilation operation on the pixels of the protrusions; then label the semantic information of the class for each pixel within each class, and determine the position of the pixels of each class in the image.

[0014] In the method described above, in step S102, assigning semantic information to each class obtained by segmentation according to the characteristics of the facade working scene includes:

[0015] 1) Statistically calculate the upper, lower, left, and right boundaries of the pixels of each class to divide the bounding box, and then calculate the area of the bounding box;

[0016] 2) Select the classes with the bounding box area greater than the area threshold, where the area threshold is preset based on historical classification experience, and then, according to the preset ground threshold, sky threshold, and facade threshold, mark the class with the upper boundary lower than the ground threshold as the ground, the class with the lower boundary higher than the sky threshold as the sky, and the class with the upper and lower boundaries within the facade threshold as the facade;

[0017] 3) After the marking is completed, select the classes with an area smaller than the preset protrusion threshold and whose bounding boxes are inside the facade class, and mark them as the protrusion class.

[0018] In the method described above, in step S1, transmitting semantic information to the point cloud includes:

[0019] S103, using the pre-calibrated relative position matrix of the lidar and the camera as the external parameter matrix, project the classes where the protrusions and facades are located from the image to the point cloud to endow the point cloud with semantic information.

[0020] In the method described above, in step S2, the rough estimation method of facade protrusions based on lidar point cloud includes:

[0021] S201, perform voxel grid downsampling on the point cloud, and then use half of the voxel grid side length as the clustering radius. Combining with semantic information, perform density clustering on the point clouds corresponding to each protrusion and facade to obtain point cloud clusters;

[0022] S202, use the random sample consensus algorithm for the facade to solve the facade equation, and then solve the distance from each point in the protrusion point cloud cluster to the facade. If the distance from the wall is less than the predetermined threshold, then take this point as an exclusion point and exclude it from the protrusion cluster;

[0023] S203, next, count the maximum and minimum values of the three-dimensional coordinates in each point cloud cluster to form the outer bounding anchor box of each point cloud cluster, and then use the outer bounding anchor box as the boundary to form the rough estimation result of the point cloud, and form a protrusion queue;

[0024] S204, according to the position information of the exclusion points in each point cloud cluster obtained in S202, combined with the information of the protrusion cluster where the exclusion points are located, calculate the projection confidence vector of each protrusion as the matching feature for adjusting the image-coupled point cloud.

[0025] In the method described above, in step S201, the density clustering includes the following steps:

[0026] ① Traverse the point cloud, search for neighboring points in the neighborhood with a radius of half of the clustering radius for each point. If the semantic information of the neighboring points is the same, mark them as the same class and continue to recursively search with the same-class neighboring points as the center; when traversing the points of the protrusion class, if there are no points of the same class in the neighborhood, then modify the semantic mark of this point to a non-protrusion point;

[0027] ② After the traversal is completed, perform secondary density clustering with the same clustering radius and the same clustering method to form the point cloud clusters of each class.

[0028] In the method described above, in step S204, the steps of calculating the projection confidence vector include:

[0029] 1. Calculate the distances from the removed points in each point cloud cluster to the bounding box of the protrusion in three spatial dimensions, and combine the three distances into an offset vector. If a removed point is within the bounding box in a certain dimension, the distance in this dimension is zero;

[0030] 2. Count the number of offset vectors, vectorially add each offset vector to obtain the total offset vector, and then divide it by the total number of points in the point cloud cluster to obtain the unit total offset vector of each point;

[0031] 3. Take the negative of the scaled unit total offset vector to obtain the projection confidence vector.

[0032] In the method described above, step S3 includes:

[0033] S301. Fit the surface equation of each protrusion according to the protrusion point cloud, and then transform the equation into the camera coordinate system using the same external parameter matrix in S103;

[0034] S302. According to the equation obtained after the transformation in S301, calculate the vertical distance from the surface of the protrusion to the camera, and combine the boundary pixels of each protrusion in the image to inversely deduce the boundary of the surface of the protrusion in space using the internal parameter matrix;

[0035] S303. Calculate the center position and size of the protrusion according to the boundary position of the surface of the protrusion and the elevation equation, complete the boundary fitting based on the image, and form a protrusion queue.

[0036] In the method described above, step S302 includes:

[0037] Taking the surface equation of the protrusion in the camera coordinate system as , the distance from the camera at the origin of the coordinate system to the surface is ; where a, b, and c are the spatial equation parameters in the three-dimensional directions, is the offset parameter of the spatial equation, and x, y, and z are the values in the three-dimensional directions respectively;

[0038] The camera internal parameter matrix is:

[0039]

[0040] where , represent the projection coefficients in the horizontal and vertical directions of the image, , respectively represent the offsets of the camera pixels in the two directions from the center;

[0041] The point cloud is projected onto the image by the camera's internal parameters as follows: , where represents the pixel coordinates, , represents the spatial coordinates of the depth normalization of a certain boundary point, and T represents the matrix transpose; thus, calculate in , , and then according to obtain the position of the boundary in the camera coordinate system, where represents the three-dimensional coordinates of the boundary points of the protrusion; then use the external parameter matrix to transform to the point cloud coordinate system, so as to obtain the boundary position of the protrusion in space.

[0042] For the method described above, step S4 includes:

[0043] S401, traverse the two queues of protrusions obtained by rough recognition of the image and the point cloud according to steps S2 and S3, calculate the Euclidean distance according to the positions of the protrusions in space, and pair the protrusions with distances less than the threshold;

[0044] S402, for each pair of protrusions, take the fitting result of the image boundary to calculate the center of the protrusion, add the projection confidence vector as the new center, and calculate the new boundary of the protrusion; then take the rough estimated boundary of the point cloud, that is, the bounding box obtained by rough estimation, and translate it in the direction of the projection confidence vector for adjustment;

[0045] S403, synthesize the new boundary of the protrusion obtained in S402 and the adjusted rough estimated boundary of the point cloud, take the boundary with a larger range in the two boundaries to form a new queue of protrusions, and complete the accurate detection.

[0046] The technical effect of the present invention is that the present invention can achieve high-precision detection of protrusions in different facade scenarios such as urban elevated sound barriers and tunnel walls under conditions including interference such as water mist and dust, with fast detection speed, low requirements for illumination during the detection process, robustness to texture fluctuations, no need for manual intervention, and is used for environmental perception of facade cleaning equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is a schematic flow chart of the present invention.

[0048] Figure 2 is a view of the facade perception field of view in an embodiment of the present invention.

[0049] Figure 3 is a method for accurate detection of protrusions according to the projection confidence vector in an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0050] The present invention will be further described in detail below with reference to specific embodiments and the accompanying drawings.

[0051] The step - by - step process of this embodiment is as Figure 1 shown. Taking a frame of elevation working - scene data as shown in Figure 2 an example, the intelligent detection method for elevation protrusions based on multi - sensor tight coupling described in this example includes the following steps:

[0052] S1. Combining the image - segmentation results, according to the sensor - field - of - view projection relationship, semantic information is assigned to the image and point - cloud data. Multimodal data including point clouds and images is collected for the elevation to be detected, and then a deep - learning network is used to perform panoramic segmentation on the image to obtain semantic information and transfer the semantic information to the point cloud.

[0053] S101. First, according to the internal - parameter matrix of the camera used to collect images, the collected images are corrected for distortion, and then a pre - trained deep - learning network is used to perform panoramic segmentation on the image, classifying the pixels according to the segmentation results. The deep - learning network can be pre - trained using existing models based on photos of similar working scenes.

[0054] S102. Semantic information is assigned to each class obtained by segmentation according to the characteristics of the elevation working scene, the classes where the elevation and elevation protrusions are located are found, and dilation operations are performed on the pixels of the protrusions; then, the semantic information of the class where each pixel is located is labeled for each pixel within each class, and the positions of the pixels of each class in the image are determined. The steps of assigning semantic information to each class obtained by segmentation according to the characteristics of the elevation working scene include:

[0055] 1) For the pixels of each class, 4 boundaries, namely top, bottom, left, and right, are statistically counted to divide the bounding box, and then the area of the bounding box is calculated.

[0056] 2) Based on historical classification experience, an area threshold is preset, and then the classes with bounding - box areas larger than the area threshold are selected. Then, according to the preset ground threshold, sky threshold, and elevation threshold, the classes with the upper boundary lower than the ground threshold are marked as the ground, the classes with the lower boundary higher than the sky threshold are marked as the sky, and the classes with both upper and lower boundaries within the elevation threshold are marked as the elevation.

[0057] 3) After the marking in step 2) is completed, the classes with areas smaller than the preset protrusion threshold and with bounding boxes inside the elevation class are selected and marked as the protrusion class, thus completing the process of assigning semantic information to each class. The protrusion threshold here is also preset according to historical classification experience.

[0058] S103. According to the pre - calibrated relative - position matrix of the lidar and the camera as the external - parameter matrix, the classes where the protrusions and the elevation are located are projected from the image to the point cloud according to the external - parameter matrix, thereby assigning semantic information to the point cloud.

[0059] S2. Downsample the point cloud, perform density clustering in combination with semantic information, calculate the projection confidence vector, use the random sample consensus algorithm to calculate the plane equation of the facade, calculate the bounding box for the protrusions, and complete the rough estimation of the protrusions based on the point cloud. The specific steps are as follows:

[0060] S201. Perform voxel grid downsampling on the point cloud, and then use half of the voxel grid side length as the clustering radius. In combination with semantic information, perform density clustering on the point clouds corresponding to each protrusion and the facade, so as to obtain point cloud clusters;

[0061] The steps of performing density clustering on the point cloud according to semantic information are as follows:

[0062] 1) Traverse the point cloud, search for neighboring points within the neighborhood of half of the clustering radius for each point. If the semantic information of the neighboring points is the same, mark them as the same class and continue to recursively search with the same-class neighboring points as the center; when traversing the points of the protrusion class, if there are no same-class points in the neighborhood, modify the semantic label of this point to a non-protrusion point;

[0063] 2) After the traversal is completed, perform secondary density clustering with the same clustering radius and the same clustering method to form the point cloud clusters of each class.

[0064] S202. Use the random sample consensus algorithm for the facade to solve the facade equation, and then solve the distance from each point in the protrusion point cloud cluster to the facade. If the distance from the wall is less than a predetermined threshold, then regard this point as an elimination point and eliminate it from the protrusion cluster;

[0065] S203. Next, count the maximum and minimum values of the three-dimensional coordinates in each point cloud cluster, and use the maximum and minimum values to form the outer bounding anchor box of each point cloud cluster. Then use the outer bounding anchor box as the boundary to form the rough estimation result of the point cloud and form a protrusion queue.

[0066] S204. According to the position information of the elimination points in each point cloud cluster obtained in S202, in combination with the information of the protrusion cluster where the elimination points are located, calculate the projection confidence vector of each protrusion as the matching feature for adjusting the image-coupled point cloud. The steps of calculating the projection confidence vector are as follows:

[0067] 1) Calculate the distances from the elimination points in each type of point cloud cluster to the bounding box of the protrusion in the three spatial dimensions, and combine the three distances into an offset vector. If the elimination point is within the bounding box in a certain dimension, the distance in this dimension is zero;

[0068] 2) Count the number of offset vectors, vectorially add each offset vector to obtain the total offset vector, and then divide it by the total number of points in the point cloud cluster to obtain the unit total offset vector of each point;

[0069] 3) Invert the scaled total unit offset vector to obtain the projection confidence vector.

[0070] S3. Use point cloud computing to calculate the surface equation of the protrusion, transform the surface equation of the protrusion into the camera coordinate system, and use the pixel positions of the protrusions in the image and combine with the elevation equation to fit the boundaries of the protrusions in space. The specific steps are as follows:

[0071] S301. Fit the surface equations of each protrusion according to the point cloud of the protrusion, and then use the same external parameter matrix in S103 to transform the equation into the camera coordinate system;

[0072] S302. According to the equation of the protrusion surface in space obtained after the transformation in S301, calculate the vertical distance from the protrusion surface to the camera, combine with the boundary pixels of each protrusion in the image, and use the internal parameter matrix to inversely deduce the boundary of the protrusion in space. The specific steps are as follows:

[0073] 1) The equation of the protrusion surface in the camera coordinate system is , and the distance from the camera at the origin of the coordinate system to the surface is ; where a, b, and c are the spatial equation parameters in the three-dimensional directions respectively, is the offset parameter of the spatial equation, and x, y, and z are the values in the three-dimensional directions respectively;

[0074] 2) The internal parameter matrix of the camera is expressed as:

[0075]

[0076] where , represent the projection coefficients in the horizontal and vertical coordinate directions of the image, , represent the offsets of the camera pixels from the center, and this matrix is determined by the properties of the camera.

[0077] 3) The point cloud is projected onto the image through the camera internal parameters:

[0078] where represents the pixel coordinates, represents the spatial coordinates with the boundary depth normalized, T represents the matrix transpose, and from the above formula, in , can be calculated. According to , the position of the boundary in the camera coordinate system can be obtained, where represents the three-dimensional coordinates of the boundary points of the protrusion. Transform it into the point cloud coordinate system using the external parameter matrix to obtain the boundary position of the protrusion in space;

[0079] S303. Calculate the center position and size of the protrusion based on the boundary position on the surface of the protrusion and the elevation equation, complete the boundary fitting based on the image, and form a protrusion queue. The specific steps are as follows:

[0080] 1) Calculate the distance between the left and right boundaries of the protrusion as the width size of the protrusion, calculate the distance between the upper and lower boundaries of the protrusion as the height size of the protrusion, and take the maximum distance between the protrusion boundary and the elevation as the distance of the protrusion from the wall.

[0081] 2) Calculate the center position of the protrusion according to the above three protrusion sizes.

[0082] S4. Refer to Figure 3 , adjust the rough estimation result of the point cloud and the boundary fitted based on the image according to the projection confidence vector, perform combined filtering on the two types of results for coupling, complete the precise detection of the protrusion, and obtain the protrusion queue. The specific steps include:

[0083] S401. Traverse the two types of protrusion queues roughly recognized from the image and the point cloud according to steps S2 and S3, calculate the Euclidean distance according to the positions of the protrusions in the space in the queue, and pair the protrusions with a distance less than the threshold;

[0084] S402. For each pair of protrusions, calculate the center of the protrusion by taking the image boundary fitting result, add the projection confidence vector as the new center, and calculate the new boundary of the protrusion. Take the rough estimation boundary of the point cloud, that is, the bounding box obtained by rough estimation, and adjust it in the direction of the projection confidence vector. The specific steps are as follows:

[0085] 1) Take the position of the protrusion obtained by boundary fitting, calculate the average values of the upper and lower, left and right, and front and back boundaries of the protrusion as the position of the center in the space;

[0086] 2) Add the projection confidence vector to the protrusion center obtained in the previous step as the new center, and calculate the new boundary according to the sizes in the three directions of up and down, left and right, and front and back;

[0087] 3) Take the boundary of the rough estimation result of the point cloud in the same direction as the projection confidence vector and move it in the direction of the projection confidence vector in the direction parallel to the elevation;

[0088] S403. Calculate the maximum value of the new boundary of the protrusion obtained in S402 and the corresponding boundary of the rough estimation of the point cloud after adjustment to form a new protrusion queue and complete the precise detection.

Claims

1. An intelligent detection method for facade protrusions based on multi-sensor tight coupling, characterized in that: The following steps are involved: S1, collects multimodal data including point cloud and image for the facade to be inspected, then uses deep learning network to perform panoramic segmentation on the image to obtain semantic information, and transfers the semantic information to the point cloud; S2, downsample the point cloud, perform density clustering based on semantic information, calculate the projection confidence vector, use the random sampling consistency algorithm to calculate the plane equation of the facade, calculate the bounding box of the protrusion, and complete the rough estimation of the protrusion based on the point cloud; S3, using point cloud to calculate the surface equation of the protrusion, and transforming the surface equation of the protrusion into the camera coordinate system, using the pixel position of the protrusion in the image and combining the facade equation to fit the boundary of the protrusion in space; S4, adjusting the rough estimation result of the point cloud and the boundary based on image fitting according to the projection confidence vector, combining and filtering the two types of results to couple, completing the accurate detection of the protrusions, and obtaining the protrusion queue; In the step S2, the method for roughly estimating elevation protrusions based on the laser radar point cloud includes: S201, performing voxel grid downsampling on the point cloud, and then using half of the side length of the voxel grid as the clustering radius, combining semantic information, density clustering the point clouds corresponding to each protrusion and facade, so as to obtain a point cloud cluster; S202, using a random sampling consistency algorithm to solve the facade equation, and then solving the distance from the facade to the points in each protrusion point cloud cluster. If the distance from the wall is less than a predetermined threshold, the point is removed from the protrusion cluster as a removal point; S203, next, the maximum and minimum values ​​of the three-dimensional coordinates in each point cloud cluster are counted to form an outer bounding anchor frame of each point cloud cluster, and then the outer bounding anchor frame is used as a boundary to form a point cloud rough estimation result to form a salient object queue; S204, calculating the projection confidence vector of each protrusion according to the position information of the removed points in each point cloud cluster obtained in S202 and the protrusion cluster information where the removed points are located, to serve as a matching feature for adjusting the image coupled point cloud; In step S204, the step of calculating the projection confidence vector includes:

1. Calculate the distances from the rejection points in each point cloud cluster to the protrusion bounding box in three spatial dimensions, and combine the three distances into an offset vector. If the rejection point is in the bounding box in a certain dimension, the distance in this dimension is zero. Second, count the number of offset vectors, add each offset vector to get the total offset vector, and then find the quotient with the total number of points in the point cloud cluster to get the unit total offset vector of each point; 3. Invert the scaled unit total offset vector to obtain the projected confidence vector.

2. The method according to claim 1, characterized in that In step S1, the step of using a deep learning network to perform panoramic segmentation on the image to obtain semantic information includes: S101, firstly, performing distortion correction on the collected image according to the intrinsic parameter matrix of the camera used to collect the image, and then performing panoramic segmentation on the image using a pre-trained deep learning network, and classifying the pixels according to the segmentation results; S102, assigning semantic information to each class obtained by segmentation according to the characteristics of the facade work scene, finding the class of the facade and the facade protrusion, and performing a dilation operation on the protrusion pixels; then annotating the semantic information of the class for each pixel in each class, and determining the position of the pixels of each class in the image.

3. The method according to claim 2, characterized in that In step S102, the various types of semantic information obtained by segmentation according to the characteristics of the facade work scene include: 1) Count the four boundaries of top, bottom, left and right for each class of pixels to divide the bounding box, and then calculate the area of ​​the bounding box; 2) Select the class whose bounding box area is larger than the area threshold, where the area threshold is preset based on historical classification experience. Then, according to the preset ground threshold, sky threshold, and facade threshold, mark the class whose upper boundary is lower than the ground threshold as ground, the class whose lower boundary is higher than the sky threshold as sky, and the class whose upper and lower boundaries are within the facade threshold as facade; 3) After the marking is completed, the class whose area is smaller than the preset protrusion threshold and whose bounding box is inside the facade class is selected and marked as the protrusion class.

4. The method according to claim 2, characterized in that: In the step S1, transferring the semantic information to the point cloud includes: S103, using the pre-calibrated relative position matrix of the laser radar and the camera as the external parameter matrix, projecting the classes of the protrusions and facades from the image into the point cloud to give the point cloud semantic information.

5. The method according to claim 1, characterized in that In step S201, density clustering includes the following steps: ① Traverse the point cloud and search for neighboring points for each point in the neighborhood of half the cluster radius. If the semantic information of the neighboring points is consistent, they are marked as the same type, and recursive search is continued with the neighboring points of the same type as the center. When the traversed point is a convex object, if there is no point of the same type in the neighborhood, the semantic mark of the point is modified to a non-convex object point; ② After the traversal is completed, secondary density clustering is performed with the same clustering radius and the same clustering method to form various point cloud clusters.

6. The method according to claim 4, characterized in that The step S3 comprises: S301, fitting the surface equation of each protrusion according to the protrusion point cloud, and then transforming the equation into the camera coordinate system using the same external parameter matrix in S103; S302, calculating the vertical distance between the protrusion surface and the camera according to the equation obtained after the transformation in S301, combining the boundary pixels of each protrusion in the image, and using the intrinsic parameter matrix to infer the boundary of the protrusion surface in space; S303, according to the boundary position of the protrusion surface combined with the vertical equation, the center position and size of the protrusion are calculated, the image-based boundary fitting is completed, and the protrusion queue is formed.

7. The method according to claim 6, characterized in that The step S302 includes: The surface equation of the protrusion in the camera coordinate system is ax+by+cz+d=0, and the distance between the camera at the origin of the coordinate system and the surface is Where a, b and c are the space equation parameters in the three-dimensional direction, d is the offset parameter of the space equation, and x, y and z are the values ​​in the three-dimensional direction; The camera intrinsic parameter matrix K is: where f x 、f y Represents the projection coefficient of the image in the horizontal and vertical directions, and x0 and y0 represent the offset of the camera pixel to the center in two directions respectively; The point cloud is projected onto the image through the camera intrinsic parameters as follows: (X, Y) = K·P, where (X, Y) represents the pixel coordinates and P = [x, y, 1] T , represents the spatial coordinates of a certain boundary point with normalized depth, and T represents the matrix transpose; thus, x and y in P are calculated, and then according to P real =P·D to get the position of the boundary in the camera coordinate system, where P real Represents the three-dimensional coordinates of the boundary points of the protrusion; then use the external parameter matrix to transform into the point cloud coordinate system to obtain the boundary position of the protrusion in space.

8. The method according to claim 1, characterized in that: Step S4 includes: S401, traverse the two salient object queues obtained according to step S2 and step S3, i.e., the image and point cloud rough recognition, calculate the Euclidean distance according to the positions of the salient objects in the queue in space, and pair the salient objects whose distance is less than a threshold; S402, for each pair of protrusions, the center of the protrusion is calculated by taking the image boundary fitting result, and the projection confidence vector is added as the new center, and the new boundary of the protrusion is calculated; then the rough estimated boundary of the point cloud, that is, the bounding box obtained by rough estimation is taken, and translated in the direction of the projection confidence vector to achieve adjustment; S403, combining the new protrusion boundary obtained in S402 with the roughly estimated boundary of the adjusted point cloud, taking the boundary with a larger range of the two boundaries to form a new protrusion queue, and completing accurate detection.

Images