Facade projection intelligent detection method based on multi-sensor tight coupling

Abstract

The invention discloses a multi-sensor tight coupling-based facade projection intelligent detection method, which comprises the following steps of: acquiring multi-modal data including point clouds and images of a to-be-detected facade, performing panoramic segmentation on the images to obtain semantic information, and transmitting the semantic information to the point clouds; carrying out downsampling on the point cloud, carrying out density clustering in combination with semantic information, calculating a projection confidence vector, calculating an equation of a plane where a vertical face is located by using a random sampling consistency algorithm, calculating a bounding box for a projection, and completing rough estimation of the projection based on the point cloud; calculating a surface equation of the projection by using a point cloud, converting the surface equation of the projection into a camera coordinate system, and fitting the boundary of the projection in the space by using the pixel position of the projection in the image and combining a facade equation; and adjusting a coarse estimation result of the point cloud and a boundary based on image fitting according to the projection confidence vector, and carrying out combined filtering on the two types of results for coupling to finish accurate detection of the projections to obtain a projection queue.

Description

Technical Field

[0001] The present invention relates to the field of intelligent environmental perception, and in particular to an intelligent detection method for facade protrusions based on tight coupling of multiple sensors. Background Art

[0002] At present, tunnel wall cleaning mainly adopts manual cleaning, which requires a lot of time and human resources, and there are great safety hazards. Compared with traditional manual cleaning, wall washing vehicles can complete the work more efficiently and quickly. Manual operation of wall washing vehicles is mainly completed by a team of two people, a driver and a worker. The driver is responsible for driving the wall washing vehicle, and the worker sits in the operating room and controls the washing arm of the wall washing vehicle through the remote control to complete the work of cleaning the stains on the outer surface of the facade. However, due to the complex operating environment of the wall washing vehicle, there are often various protrusions on the facades of tunnels and sound insulation screens, which brings great difficulty and risk to the cleaning operation. If the cleaning worker is not focused or makes an error in operation, the end of the arm may collide with the facade, causing impact damage to the arm system and the facade, and even a car accident in severe cases.

[0003] In the automated cleaning work, the cleaning equipment based on the combination of laser radar and camera imaging is currently considered. It can automatically adjust the cleaning strategy by sensing the distance and shape information of the surrounding environment, avoid hitting the protrusions, and ensure the cleaning effect. This combination can not only improve the cleaning efficiency and quality, but also greatly reduce the cleaning cost, reduce the labor intensity and safety risks of cleaning workers. However, due to the different characteristics of the facades in different scenes, such as different curvatures in the horizontal and vertical directions, and different transparency of the facades; when the protrusions are small in size and the light reflectivity of the color and material is low, the protrusion information obtained by the laser radar is weak and sensitive to environmental changes; for facades with high transparency such as soundproof screens, most of the light beams emitted by the laser radar will penetrate the glass, resulting in the return of objects behind the soundproof screen or no return signal; the water mist sprayed by the vehicle during operation will move in the direction of the vehicle under the drive of the roller brush and enter the field of view of the laser radar. The dense water mist will reflect the radar signal or float around the protrusion. The camera is sensitive to texture information, which makes it difficult to use in scenes such as tunnels, and it is impossible to obtain spatial position information. Summary of the invention

[0004] In view of the above problems, the present invention proposes an intelligent detection method for facade protrusions based on tight coupling of multiple sensors. The method can accurately identify various facade protrusions under different scenes and thin water mist conditions. By applying it to facade cleaning equipment, reliable and stable facade protrusion detection results can be obtained.

[0005] In order to achieve the above technical purpose, the technical solution of the present invention is:

[0006] An intelligent detection method for facade protrusions based on multi-sensor tight coupling includes the following steps:

[0007] S1, collects multimodal data including point cloud and image for the facade to be inspected, then uses deep learning network to perform panoramic segmentation on the image to obtain semantic information, and transfers the semantic information to the point cloud;

[0008] S2, downsample the point cloud, perform density clustering based on semantic information, calculate the projection confidence vector, use the random sampling consistency algorithm to calculate the plane equation of the facade, calculate the bounding box of the protrusion, and complete the rough estimation of the protrusion based on the point cloud;

[0009] S3, using point cloud to calculate the surface equation of the protrusion, and transforming the surface equation of the protrusion into the camera coordinate system, using the pixel position of the protrusion in the image and combining the facade equation to fit the boundary of the protrusion in space;

[0010] S4, adjusts the rough estimation result of the point cloud and the boundary based on image fitting according to the projection confidence vector, performs combined filtering on the two types of results to couple, completes the accurate detection of the protrusions, and obtains the protrusion queue.

[0011] In the method, in step S1, the step of using a deep learning network to perform panoramic segmentation on the image to obtain semantic information includes:

[0012] S101, firstly, performing distortion correction on the collected image according to the intrinsic parameter matrix of the camera used to collect the image, and then performing panoramic segmentation on the image using a pre-trained deep learning network, and classifying the pixels according to the segmentation results;

[0013] S102, assigning semantic information to each class obtained by segmentation according to the characteristics of the facade work scene, finding the class of the facade and the facade protrusion, and performing a dilation operation on the protrusion pixels; then annotating the semantic information of the class for each pixel in each class, and determining the position of the pixels of each class in the image.

[0014] In the method, in step S102, the semantic information obtained by segmentation according to the characteristics of the facade work scene includes:

[0015] 1) Count the four boundaries of top, bottom, left and right for each class of pixels to divide the bounding box, and then calculate the area of ​​the bounding box;

[0016] 2) Select the class whose bounding box area is larger than the area threshold, where the area threshold is preset based on historical classification experience. Then, based on the preset ground threshold, sky threshold, and facade threshold, mark the class whose upper boundary is lower than the ground threshold as ground, the class whose lower boundary is higher than the sky threshold as sky, and the class whose upper and lower boundaries are within the facade threshold as facade;

[0017] 3) After marking is completed, select the class whose area is smaller than the preset protrusion threshold and whose bounding box is inside the facade class and mark it as the protrusion class.

[0018] In the method, in step S1, transferring semantic information to the point cloud includes:

[0019] S103, using the pre-calibrated relative position matrix of the laser radar and the camera as the external parameter matrix, projecting the classes of the protrusions and facades from the image into the point cloud to give the point cloud semantic information.

[0020] The method, in step S2, the method for roughly estimating elevation protrusions based on the laser radar point cloud comprises:

[0021] S201, performing voxel grid downsampling on the point cloud, and then using half of the side length of the voxel grid as the clustering radius, combining semantic information, density clustering the point clouds corresponding to each protrusion and facade, so as to obtain a point cloud cluster;

[0022] S202, using a random sampling consistency algorithm to solve the facade equation, and then solving the distance from the facade to the points in each protrusion point cloud cluster, if the distance from the wall is less than a predetermined threshold, the point is removed from the protrusion cluster as a removal point;

[0023] S203, next, the maximum and minimum values ​​of the three-dimensional coordinates in each point cloud cluster are counted to form an outer bounding anchor frame of each point cloud cluster, and then the outer bounding anchor frame is used as a boundary to form a point cloud rough estimation result to form a salient object queue;

[0024] S204, based on the position information of the removed points in each point cloud cluster obtained in S202, combined with the information of the protrusion cluster where the removed points are located, calculate the projection confidence vector of each protrusion to serve as a matching feature for adjusting the image coupled point cloud.

[0025] The method, in step S201, density clustering includes the following steps:

[0026] ① Traverse the point cloud and search for neighboring points for each point in the neighborhood of half the cluster radius. If the semantic information of the neighboring points is consistent, they are marked as the same type, and recursive search is continued with the neighboring points of the same type as the center. When the traversed point is a convex object, if there is no point of the same type in the neighborhood, the semantic mark of the point is modified to a non-convex object point;

[0027] ② After the traversal is completed, secondary density clustering is performed with the same clustering radius and the same clustering method to form various point cloud clusters.

[0028] In the method, in step S204, the step of calculating the projection confidence vector includes:

[0029] 1. Calculate the distances from the rejection points in each point cloud cluster to the protrusion bounding box in three spatial dimensions, and combine the three distances into an offset vector. If the rejection point is in the bounding box in a certain dimension, the distance in this dimension is zero.

[0030] Second, count the number of offset vectors, add each offset vector to get the total offset vector, and then find the quotient with the total number of points in the point cloud cluster to get the unit total offset vector of each point;

[0031] 3. Invert the scaled unit total offset vector to obtain the projected confidence vector.

[0032] The method described, step S3 comprises:

[0033] S301, fitting the surface equation of each protrusion according to the protrusion point cloud, and then transforming the equation into the camera coordinate system using the same external parameter matrix in S103;

[0034] S302, calculating the vertical distance between the protrusion surface and the camera according to the equation obtained after the transformation in S301, combining the boundary pixels of each protrusion in the image, and using the intrinsic parameter matrix to infer the boundary of the protrusion surface in space;

[0035] S303, according to the boundary position of the protrusion surface combined with the vertical equation, the center position and size of the protrusion are calculated, the image-based boundary fitting is completed, and the protrusion queue is formed.

[0036] The method described, step S302 includes:

[0037] The surface equation of the protrusion in the camera coordinate system is , the distance between the camera and the surface at the origin of the coordinate system ; where a, b and c are the parameters of the spatial equation in the three-dimensional direction, is the offset parameter of the space equation, x, y and z are the values ​​in the three-dimensional directions respectively;

[0038] Camera intrinsic matrix for:

[0039]

[0040] in , Represents the projection coefficient of the image in the horizontal and vertical directions, , Respectively represent the offset of the camera pixel from the center in two directions;

[0041] Then the point cloud is projected onto the image through the camera internal parameters: ,in represents pixel coordinates, , represents the spatial coordinates of a certain boundary point with normalized depth, and T represents the matrix transpose; thus, we can calculate In , , and then according to Get the position of the boundary in the camera coordinate system, where Represents the three-dimensional coordinates of the boundary points of the protrusion; then use the external parameter matrix to transform into the point cloud coordinate system to obtain the boundary position of the protrusion in space.

[0042] The method described, step S4 comprises:

[0043] S401, traverse the two salient object queues obtained according to step S2 and step S3, i.e., the image and point cloud rough recognition, calculate the Euclidean distance according to the positions of the salient objects in the queue in space, and pair the salient objects whose distance is less than a threshold;

[0044] S402, for each pair of protrusions, the center of the protrusion is calculated by taking the image boundary fitting result, and the projection confidence vector is added as the new center, and the new boundary of the protrusion is calculated; then the rough estimated boundary of the point cloud, that is, the bounding box obtained by rough estimation is taken, and translated in the direction of the projection confidence vector to achieve adjustment;

[0045] S403, combining the new protrusion boundary obtained in S402 with the roughly estimated boundary of the adjusted point cloud, taking the boundary with a larger range of the two boundaries to form a new protrusion queue, and completing accurate detection.

[0046] The technical effect of the present invention is that the present invention can achieve high-precision detection of protrusions in different facade scenes such as urban elevated sound insulation screens and tunnel walls under conditions of interference such as water mist and dust. The detection speed is fast, the detection process has low requirements for lighting, is robust to texture fluctuations, and does not require human intervention. It is used for environmental perception of facade cleaning equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a schematic diagram of the process of the present invention.

[0048] Figure 2 This is a diagram of the vertical perception field of view in an embodiment of the present invention.

[0049] Figure 3 This is a method for accurately detecting protrusions based on projection confidence vectors in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The present invention is further described in detail below in conjunction with specific embodiments and drawings.

[0051] The steps of this embodiment are as follows: Figure 1 Here is Figure 2 Taking a frame of facade working scene data as an example, the intelligent detection method for facade protrusions based on multi-sensor tight coupling described in this example includes the following steps:

[0052] S1, combined with the image segmentation results, gives semantic information to the image and point cloud data according to the sensor field of view projection relationship. Multimodal data including point cloud and image are collected for the facade to be inspected, and then the deep learning network is used to perform panoramic segmentation on the image to obtain semantic information, and the semantic information is transferred to the point cloud.

[0053] S101, firstly, according to the intrinsic parameter matrix of the camera used to collect the image, the collected image is distorted, and then a pre-trained deep learning network is used to perform panoramic segmentation on the image, and the pixels are classified according to the segmentation results. The deep learning network can be pre-trained based on photos of similar working scenes using an existing model.

[0054] S102, assigning semantic information to each class obtained by segmentation according to the characteristics of the facade work scene, finding the class of the facade and the facade protrusion, and performing an expansion operation on the protrusion pixels; then annotating the semantic information of the class for each pixel in each class, and determining the position of the pixels of each class in the image. The step of assigning semantic information to each class obtained by segmentation according to the characteristics of the facade work scene includes:

[0055] 1) Count the four boundaries of top, bottom, left and right for each class of pixels to divide the bounding box, and then calculate the area of ​​the bounding box;

[0056] 2) Based on historical classification experience, the area threshold is preset, and then the class whose bounding box area is larger than the area threshold is selected. Then, according to the preset ground threshold, sky threshold, and facade threshold, the class whose upper boundary is lower than the ground threshold is marked as ground, the class whose lower boundary is higher than the sky threshold is marked as sky, and the class whose upper and lower boundaries are within the facade threshold is marked as facade.

[0057] 3) After the marking in step 2) is completed, the class whose area is smaller than the preset protrusion threshold and whose bounding box is inside the facade class is selected and marked as the protrusion class, thus completing the process of assigning semantic information to each class. The protrusion threshold here is also preset based on historical classification experience.

[0058] S103, using the pre-calibrated relative position matrix of the laser radar and the camera as an external parameter matrix, and then projecting the classes of the protrusions and facades from the image to the point cloud according to the external parameter matrix, thereby giving the point cloud semantic information.

[0059] S2, downsample the point cloud, perform density clustering based on semantic information, calculate the projection confidence vector, use the random sampling consistency algorithm to calculate the plane equation of the facade, calculate the bounding box of the protrusion, and complete the rough estimation of the protrusion based on the point cloud. The specific steps include:

[0060] S201, performing voxel grid downsampling on the point cloud, and then using half of the side length of the voxel grid as the clustering radius, combining semantic information, density clustering the point clouds corresponding to each protrusion and facade, so as to obtain a point cloud cluster;

[0061] The steps for density clustering of point clouds based on semantic information are as follows:

[0062] 1) Traverse the point cloud and search for neighboring points for each point in a neighborhood of half the cluster radius. If the semantic information of the neighboring points is consistent, they are marked as the same type, and recursive search is continued with the neighboring points of the same type as the center. When the traversed point is a convex object, if there is no point of the same type in the neighborhood, the semantic mark of the point is modified to a non-convex object point;

[0063] 2) After the traversal is completed, secondary density clustering is performed with the same clustering radius and the same clustering method to form various point cloud clusters.

[0064] S202, using a random sampling consistency algorithm to solve the facade equation, and then solving the distance from the facade to the points in each protrusion point cloud cluster. If the distance from the wall is less than a predetermined threshold, the point is removed from the protrusion cluster as a removal point;

[0065] S203, next, the maximum and minimum values ​​of the three-dimensional coordinates in each point cloud cluster are counted, and the maximum and minimum values ​​are used to form an outer bounding anchor box of each point cloud cluster, and then the outer bounding anchor box is used as a boundary to form a point cloud rough estimation result to form a protrusion queue.

[0066] S204, based on the position information of the removed points in each point cloud cluster obtained in S202, combined with the information of the protrusion cluster where the removed points are located, calculate the projection confidence vector of each protrusion to serve as a matching feature for adjusting the image coupled point cloud. The steps for calculating the projection confidence vector are as follows:

[0067] 1) Calculate the distances from the rejection points in each point cloud cluster to the salient bounding box in three spatial dimensions, and combine the three distances into an offset vector. If the rejection point is within the bounding box in a certain dimension, the distance in this dimension is zero.

[0068] 2) Count the number of offset vectors, add each offset vector to get the total offset vector, and then find the quotient with the total number of points in the point cloud cluster to get the unit total offset vector of each point;

[0069] 3) Invert the scaled unit total offset vector to obtain the projected confidence vector.

[0070] S3, use the point cloud to calculate the surface equation of the protrusion, and transform the surface equation of the protrusion into the camera coordinate system, and use the pixel position of the protrusion in the image and the facade equation to fit the boundary of the protrusion in space. The specific steps include:

[0071] S301, fitting the surface equation of each protrusion according to the protrusion point cloud, and then transforming the equation into the camera coordinate system using the same external parameter matrix in S103;

[0072] S302, according to the equation of the protrusion surface in space in the camera coordinate system obtained after the transformation in S301, the vertical distance of the protrusion surface from the camera is calculated, and the boundary pixels of each protrusion in the image are combined to use the intrinsic parameter matrix to infer the boundary of the protrusion in space. The specific steps are as follows:

[0073] 1) The surface equation of the protrusion in the camera coordinate system is: , the distance between the camera and the surface at the origin of the coordinate system ; where a, b and c are the parameters of the spatial equation in the three-dimensional direction, is the offset parameter of the space equation, x, y and z are the values ​​in the three-dimensional directions respectively;

[0074] 2) Camera intrinsic matrix It is expressed as:

[0075]

[0076] in , Represents the projection coefficient of the image in the horizontal and vertical directions, , Represents the offset of the camera pixel to the center. The matrix is ​​determined by the camera properties.

[0077] 3) The point cloud is projected onto the image using the camera’s intrinsic parameters:

[0078] in represents pixel coordinates, represents the spatial coordinates of the normalized boundary depth, T represents the matrix transpose, and the above formula can be used to calculate middle , ,according to The position of the boundary in the camera coordinate system can be obtained, where Represents the three-dimensional coordinates of the protrusion boundary points. Use the external parameter matrix to transform it into the point cloud coordinate system to obtain the boundary position of the protrusion in space;

[0079] S303, according to the boundary position of the protrusion surface combined with the vertical equation, the center position and size of the protrusion are calculated, the boundary fitting based on the image is completed, and the protrusion queue is formed. The specific steps are as follows:

[0080] 1) Calculate the distance between the left and right boundaries of the protrusion as the width of the protrusion, calculate the distance between the upper and lower boundaries of the protrusion as the height of the protrusion, and take the maximum distance between the protrusion boundary and the facade as the distance of the protrusion from the wall.

[0081] 2) Based on the above three protrusion sizes, calculate the center position of the protrusion.

[0082] S4, see Figure 3 , according to the projection confidence vector, the rough estimation result of the point cloud and the boundary based on image fitting are adjusted, and the two types of results are combined and filtered to couple, so as to complete the accurate detection of protrusions and obtain the protrusion queue. The specific steps include:

[0083] S401, traverse the two salient object queues obtained by rough recognition from the image and point cloud according to step S2 and step S3, calculate the Euclidean distance according to the positions of the salient objects in the queue in space, and pair the salient objects whose distance is less than a threshold;

[0084] S402, for each pair of protrusions, the center of the protrusion is calculated by taking the image boundary fitting result, and the projection confidence vector is added as the new center, and the new boundary of the protrusion is calculated. The rough estimated boundary of the point cloud, that is, the bounding box obtained by the rough estimation, is adjusted in the direction of the projection confidence vector. The specific steps are as follows:

[0085] 1) Take the position of the protrusion obtained by boundary fitting, and calculate the average value of the upper and lower, left and right, and front and back boundaries of the protrusion as the position of the center in space;

[0086] 2) Add the projection confidence vector to the center of the protrusion obtained in the previous step as the new center, and calculate the new boundary based on the dimensions in the up-down, left-right, and front-back directions;

[0087] 3) Take the boundary of the point cloud rough estimation result in the same direction as the projection confidence vector, and move in the direction of the projection confidence vector in a direction parallel to the vertical surface;

[0088] S403, calculating the maximum value of the new boundary of the protrusion obtained in S402 and the roughly estimated corresponding boundary of the adjusted point cloud to form a new protrusion queue and complete accurate detection.

Claims

1. An intelligent detection method for facade protrusions based on multi-sensor tight coupling, characterized in that: The following steps are involved: S1, collects multimodal data including point cloud and image for the facade to be inspected, then uses deep learning network to perform panoramic segmentation on the image to obtain semantic information, and transfers the semantic information to the point cloud; S2, downsample the point cloud, perform density clustering based on semantic information, calculate the projection confidence vector, use the random sampling consistency algorithm to calculate the plane equation of the facade, calculate the bounding box of the protrusion, and complete the rough estimation of the protrusion based on the point cloud; S3, using point cloud to calculate the surface equation of the protrusion, and transforming the surface equation of the protrusion into the camera coordinate system, using the pixel position of the protrusion in the image and combining the facade equation to fit the boundary of the protrusion in space; S4, adjusts the rough estimation result of the point cloud and the boundary based on image fitting according to the projection confidence vector, performs combined filtering on the two types of results to couple, completes the accurate detection of the protrusions, and obtains the protrusion queue.

2. The method according to claim 1, characterized in that In step S1, the step of using a deep learning network to perform panoramic segmentation on the image to obtain semantic information includes: S101, firstly, performing distortion correction on the collected image according to the intrinsic parameter matrix of the camera used to collect the image, and then performing panoramic segmentation on the image using a pre-trained deep learning network, and classifying the pixels according to the segmentation results; S102, assigning semantic information to each class obtained by segmentation according to the characteristics of the facade work scene, finding the class of the facade and the facade protrusion, and performing a dilation operation on the protrusion pixels; then annotating the semantic information of the class for each pixel in each class, and determining the position of the pixels of each class in the image.

3. The method according to claim 2, characterized in that In step S102, the various types of semantic information obtained by segmentation according to the characteristics of the facade work scene include: 1) Count the four boundaries of top, bottom, left and right for each class of pixels to divide the bounding box, and then calculate the area of ​​the bounding box; 2) Select the class whose bounding box area is larger than the area threshold, where the area threshold is preset based on historical classification experience. Then, based on the preset ground threshold, sky threshold, and facade threshold, mark the class whose upper boundary is lower than the ground threshold as ground, the class whose lower boundary is higher than the sky threshold as sky, and the class whose upper and lower boundaries are within the facade threshold as facade; 3) After marking is completed, select the class whose area is smaller than the preset protrusion threshold and whose bounding box is inside the facade class and mark it as the protrusion class.

4. The method according to claim 2, characterized in that: In the step S1, transferring the semantic information to the point cloud includes: S103, using the pre-calibrated relative position matrix of the laser radar and the camera as the external parameter matrix, projecting the classes of the protrusions and facades from the image into the point cloud to give the point cloud semantic information.

5. The method according to claim 1, characterized in that In the step S2, the method for roughly estimating elevation protrusions based on the laser radar point cloud includes: S201, performing voxel grid downsampling on the point cloud, and then using half of the side length of the voxel grid as the clustering radius, combining semantic information, density clustering the point clouds corresponding to each protrusion and facade, so as to obtain a point cloud cluster; S202, using a random sampling consistency algorithm to solve the facade equation, and then solving the distance from the facade to the points in each protrusion point cloud cluster. If the distance from the wall is less than a predetermined threshold, the point is removed from the protrusion cluster as a removal point; S203, next, the maximum and minimum values ​​of the three-dimensional coordinates in each point cloud cluster are counted to form an outer bounding anchor frame of each point cloud cluster, and then the outer bounding anchor frame is used as a boundary to form a point cloud rough estimation result to form a salient object queue; S204, based on the position information of the removed points in each point cloud cluster obtained in S202, combined with the information of the protrusion cluster where the removed points are located, calculate the projection confidence vector of each protrusion to serve as a matching feature for adjusting the image coupled point cloud.

6. The method according to claim 5, characterized in that In step S201, density clustering includes the following steps: ① Traverse the point cloud and search for neighboring points for each point in the neighborhood of half the cluster radius. If the semantic information of the neighboring points is consistent, they are marked as the same type, and recursive search is continued with the neighboring points of the same type as the center. When the traversed point is a convex object, if there is no point of the same type in the neighborhood, the semantic mark of the point is modified to a non-convex object point; ② After the traversal is completed, secondary density clustering is performed with the same clustering radius and the same clustering method to form various point cloud clusters.

7. The method according to claim 5, characterized in that In step S204, the step of calculating the projection confidence vector includes:

1. Calculate the distances from the rejection points in various point cloud clusters to the protrusion bounding box in three spatial dimensions, and combine the three distances into an offset vector. If the rejection point is within the bounding box in a certain dimension, the distance in this dimension is zero. Second, count the number of offset vectors, add each offset vector to get the total offset vector, and then find the quotient with the total number of points in the point cloud cluster to get the unit total offset vector of each point; 3. Invert the scaled unit total offset vector to obtain the projected confidence vector.

8. The method according to claim 4, characterized in that The step S3 comprises: S301, fitting the surface equation of each protrusion according to the protrusion point cloud, and then transforming the equation into the camera coordinate system using the same external parameter matrix in S103; S302, calculating the vertical distance between the protrusion surface and the camera according to the equation obtained after the transformation in S301, combining the boundary pixels of each protrusion in the image, and using the intrinsic parameter matrix to infer the boundary of the protrusion surface in space; S303, according to the boundary position of the protrusion surface combined with the vertical equation, the center position and size of the protrusion are calculated, the image-based boundary fitting is completed, and the protrusion queue is formed.

9. The method according to claim 8, characterized in that The step S302 includes: The surface equation of the protrusion in the camera coordinate system is , the distance between the camera and the surface at the origin of the coordinate system ; where a, b and c are the parameters of the spatial equation in the three-dimensional direction, is the offset parameter of the space equation, x, y and z are the values ​​in the three-dimensional directions respectively; Camera intrinsic matrix for: ； in , Represents the projection coefficient of the image in the horizontal and vertical directions, , Respectively represent the offset of the camera pixel from the center in two directions; Then the point cloud is projected onto the image through the camera internal parameters: ,in represents pixel coordinates, , represents the spatial coordinates of a certain boundary point with normalized depth, and T represents the matrix transpose; thus, we can calculate In , , and then according to Get the position of the boundary in the camera coordinate system, where Represents the three-dimensional coordinates of the boundary points of the protrusion; then use the external parameter matrix to transform into the point cloud coordinate system to obtain the boundary position of the protrusion in space.

10. The method according to claim 1, characterized in that Step S4 includes: S401, traverse the two salient object queues obtained according to step S2 and step S3, i.e., the image and point cloud rough recognition, calculate the Euclidean distance according to the positions of the salient objects in the queue in space, and pair the salient objects whose distance is less than a threshold; S402, for each pair of protrusions, the center of the protrusion is calculated by taking the image boundary fitting result, and the projection confidence vector is added as the new center, and the new boundary of the protrusion is calculated; then the rough estimated boundary of the point cloud, that is, the bounding box obtained by rough estimation is taken, and translated in the direction of the projection confidence vector to achieve adjustment; S403, combining the new protrusion boundary obtained in S402 with the roughly estimated boundary of the adjusted point cloud, taking the boundary with a larger range of the two boundaries to form a new protrusion queue, and completing accurate detection.

Images