A camera-based two-dimensional lidar point cloud semantic assignment method and system

Abstract

This invention belongs to the field of artificial intelligence technology and discloses a method and system for semantic assignment of two-dimensional LiDAR point clouds based on a camera. The method includes: obtaining the projection transformation matrix from the camera image plane to the laser scanning plane; obtaining the pixel set within the target bounding box of the target object image based on the target object image captured by the camera; extracting image features from the target object image to obtain the contour geometric line equation of the target object; inversely projecting the contour geometric line equation of the target object onto the LiDAR coordinate system using the projection transformation matrix to obtain the inverse projection curve; determining the range of laser points hitting the target object based on the inverse projection curve, and assigning semantic information to the laser points within the range. This invention abandons complex clustering algorithms, data fusion algorithms, and a large amount of laser point reprojection calculations. It only requires inverse projection of the contour to filter out the range of laser points hitting the target object, improving the accuracy of point cloud search and the speed of semantic assignment.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, and more specifically, relates to a method and system for semantic assignment of two-dimensional LiDAR point clouds based on a camera. Background Technology

[0002] LiDAR (Light Detection and Ranging) is a radar system that uses laser beams to detect the position, velocity, and other characteristics of targets. Its working principle involves emitting a detection signal (laser beam) towards the target, then comparing the received signal reflected back from the target (target echo) with the emitted signal. After appropriate processing, information about the target can be obtained, such as its distance, azimuth, altitude, velocity, attitude, and even shape. This allows for the detection, tracking, and identification of objects in the surrounding environment. However, the amount of point cloud data in LiDAR is limited by the number of laser beams, and a higher number of beams increases costs. Furthermore, insufficient point cloud data also results in inadequate semantic information. Cameras, on the other hand, inherently possess rich semantic information and have relatively mature target detection algorithms and models. However, calculating the distance and position of objects from 2D images is not accurate enough, largely due to the difficulty in precisely determining the original dimensions of the objects. Therefore, combining the advantages of LiDAR and cameras, and fusing the detection results separately, yields better results.

[0003] Generally, semantic assignment methods for LiDAR point clouds require data fusion between LiDAR point information and camera pixel information. After data fusion, the complexity of depth calculations can be significantly reduced due to the convenient and accurate acquisition range of LiDAR. Currently, a projection matrix is ​​typically used to project the coordinates of the LiDAR points onto the image, and semantic information is assigned to the corresponding LiDAR points by ensuring they fall within the target detection bounding box obtained by a convolutional neural network image recognition algorithm.

[0004] In methods for assigning semantic information to LiDAR point clouds through data fusion with cameras, projecting LiDAR points onto an image target recognition bounding box to complete the semantic assignment requires reprojecting a large number of data points back into the image to find the range of laser points hitting the target. This is computationally intensive and requires complex clustering processes to filter the LiDAR point cloud. Furthermore, existing image recognition algorithms often produce inaccurate target bounding boxes, which are usually larger than the object's outline. These LiDAR point cloud semantic assignment algorithms are limited by deep learning-based target recognition algorithms, which represent the laser points hitting the object as those captured within the bounding box, resulting in inaccurate semantic information assignment to the laser point cloud. How to reduce the computational cost of data fusion and quickly locate the range of the semantically assigned object when faced with a large amount of laser point and pixel data is one of the important technical challenges in this field. Summary of the Invention

[0005] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides a camera-based two-dimensional lidar point cloud semantic assignment method and system, which solves the problems of large computational load, complex algorithm and low accuracy of the existing lidar point cloud semantic assignment method when dealing with a large amount of data.

[0006] To achieve the above objectives, according to one aspect of the present invention, a method for semantic assignment of two-dimensional LiDAR point clouds based on a camera is provided, comprising:

[0007] The pose relationship between the camera and the 2D LiDAR is calibrated to obtain the projection transformation matrix from the camera image plane to the laser scanning plane;

[0008] Based on the target object image captured by the camera, semantic information of the corresponding pixels in the target object image is assigned by a deep learning algorithm to obtain the pixel set within the target bounding box of the target object image.

[0009] Image feature extraction is performed on the target object image to obtain the contour geometric line equation of the target object;

[0010] A point cloud map of the surrounding environment is generated using a two-dimensional lidar.

[0011] The contour geometry equation of the target object is inversely projected onto the lidar coordinate system using the projection transformation matrix to obtain the inverse projection curve.

[0012] The range of laser points hitting the target object is determined based on the inverse projection curve, and semantic information of the laser points within the range is assigned based on the pixel set within the target box of the target object image.

[0013] According to the camera-based two-dimensional lidar point cloud semantic assignment method provided by the present invention, the pose relationship between the camera and the two-dimensional lidar is calibrated, and the projection transformation matrix from the camera image plane to the laser scanning plane is obtained, specifically including:

[0014] The relative positions of the fixed camera and the 2D LiDAR are kept so that the optical center of the camera and the 2D LiDAR are on the same vertical line. A planar object with a straight or circular geometric outline is used as a marker.

[0015] Image features are extracted from the images of the landmark captured by the camera to obtain the geometric line equations of the landmark's outline;

[0016] Extract the coordinates of the intersection point between the laser scanning plane and the outline of the marker;

[0017] Replace or adjust the position of the markers to obtain multiple sets of contour geometric line equations and contour intersection coordinate data corresponding to multiple markers;

[0018] Based on the camera imaging model and the projection relationship from the camera image plane to the laser scanning plane, a mathematical model of the projection transformation matrix is ​​constructed.

[0019] The projection transformation matrix is ​​obtained by solving the mathematical model using multiple sets of contour geometric line equations, contour intersection point coordinate data, and geometric constraints between contour geometric lines and contour intersection points.

[0020] According to the camera-based two-dimensional lidar point cloud semantic assignment method provided by the present invention, the image feature extraction steps specifically include:

[0021] Perform grayscale processing on the image;

[0022] The image after grayscale processing is smoothed using an image-guided filter.

[0023] Extract the contour geometry equations from the smoothed image.

[0024] According to the camera-based two-dimensional lidar point cloud semantic assignment method provided by the present invention, when the contour geometric line features of the captured target are straight lines, the extraction of the contour geometric line equation specifically includes:

[0025] The maximum inter-class variance method is used to extract the foreground image and obtain the target binarized region.

[0026] Based on the extracted target binarized region, edge noise is filtered using opening and closing operations;

[0027] The Sobel vertical operator is used for edge detection to extract contour edge pixels;

[0028] The Hough transform is used to extract the equations of straight lines, which are then used as the equations of the contour geometry lines.

[0029] When the outline geometry of the target is an arc, the extraction of the outline geometry equation specifically includes:

[0030] The ellipse equation is extracted using the arc-supported line segment method and used as the equation for the contour geometry lines.

[0031] According to the camera-based two-dimensional lidar point cloud semantic assignment method provided by the present invention, the coordinates of the intersection point of the laser scanning plane and the contour of the marker are extracted, specifically including:

[0032] Based on the known set of laser points that hit the marker, obtain the first angular interval [θ1, θ2] of the laser point set;

[0033] The laser point set is fitted to a straight line;

[0034] Change the first angle interval to the second angle interval. Where α is the angular resolution of the two-dimensional lidar;

[0035] Using the angles at both ends of the second angle interval as the slope, a straight line passing through the origin of the lidar coordinate system is drawn and intersects the fitted straight line at two points to obtain the coordinates of the two contour intersection points.

[0036] According to the camera-based two-dimensional lidar point cloud semantic assignment method provided by the present invention, the mathematical model of the projection transformation matrix is ​​specifically as follows:

[0037]

[0038] Among them, Z C It is a constant; [u0, v0] are pixel coordinates; [u0, v0] are the pixel coordinates of the camera's optical center; the physical dimensions of the pixel's length and width are dx and dy, respectively, and f is the camera's focal length. It is the rotation transformation matrix from the lidar coordinate system to the camera coordinate system; This is the translation transformation matrix from the lidar coordinate system to the camera coordinate system; the coordinates of point P in the lidar coordinate system are... Δ represents the product of the preceding matrices.

[0039] According to the camera-based two-dimensional lidar point cloud semantic assignment method provided by the present invention, when the contour geometric line feature of the captured target is a straight line, the geometric constraint between the contour geometric line and the contour intersection point is specifically as follows:

[0040]

[0041] Where [a,b,c] are the parameters of the contour geometric line equation extracted when the contour geometric line feature of the photographed target is a straight line; H is the projection transformation matrix; These are coordinates in the lidar coordinate system.

[0042] When the outline geometry of the target being photographed is an arc, the geometric constraints between the outline geometry and the intersection point of the outline are as follows:

[0043]

[0044] Where A represents the parameters of the contour geometry equation extracted when the contour geometry of the target is an arc.

[0045] According to the camera-based two-dimensional lidar point cloud semantic assignment method provided by the present invention, the inverse projection curve is obtained by inversely projecting the contour geometric line equation of the target object onto the lidar coordinate system through the projection transformation matrix. Specifically, this includes:

[0046] When the outline geometry of the target object is a straight line, the inverse projection curve consists of two straight lines.

[0047] When the outline geometry of the target object is a circular arc, the inverse projection curve is an ellipse or a hyperbola.

[0048] According to the camera-based two-dimensional lidar point cloud semantic assignment method provided by the present invention, the range of laser points hitting the target object is determined based on the inverse projection curve, specifically including:

[0049] When the inverse projection curve consists of two straight lines, the range between the two straight lines is taken as the range of the laser point that hits the target object.

[0050] When the inverse projection curve is an ellipse, the range between the two tangents of the ellipse passing through the origin of the lidar coordinate system is taken as the range of the laser point hitting the target object.

[0051] When the inverse projection curve is a hyperbola, the range between the two asymptotes of the hyperbola is taken as the range of the laser point that hits the target object.

[0052] According to another aspect of the present invention, a camera-based two-dimensional lidar point cloud semantic assignment system is provided, comprising:

[0053] The calibration module is used to calibrate the pose relationship between the camera and the 2D LiDAR, and to obtain the projection transformation matrix from the camera image plane to the laser scanning plane.

[0054] The pixel semantic acquisition module is used to assign semantic information to the corresponding pixels of the target object image based on the target object image captured by the camera, and to obtain the pixel set within the target box of the target object image;

[0055] The image feature extraction module is used to extract image features from the target object image and obtain the contour geometric line equation of the target object;

[0056] The point cloud data generation module is used to generate point cloud maps of the surrounding environment using a two-dimensional LiDAR.

[0057] The inverse projection module is used to inversely project the contour geometric line equation of the target object onto the lidar coordinate system through the projection transformation matrix to obtain the inverse projection curve.

[0058] The point cloud semantic assignment module is used to determine the range of laser points hitting the target object based on the inverse projection curve, and to assign semantic information of the laser points within the range based on the pixel set within the target box of the target object image.

[0059] Overall, compared with the prior art, the semantic assignment method and system for camera-based two-dimensional LiDAR point clouds provided by the present invention offer the following advantages:

[0060] 1. Compared with existing methods for semantic assignment of LiDAR point clouds, this method abandons complex clustering algorithms, data fusion algorithms and a large number of laser point reprojection calculations. When faced with a large number of laser points, this method only needs to perform inverse projection on the contour to filter out the range of laser points that hit the target object, which improves the accuracy of searching point clouds and the speed of semantic assignment, thus completing the semantic assignment of LiDAR point clouds more quickly and robustly.

[0061] 2. This method employs inverse projection of the contour geometric line equation of the target object onto the lidar coordinate system, thereby directly obtaining the range of laser points hitting the target object on the laser scanning plane. Compared with existing methods that use laser points captured within the image frame to represent the laser points hitting the object, this method ensures that the determination of the range of laser points hitting the target object is no longer limited by the image frame obtained by the deep learning-based target recognition algorithm, and also improves the accuracy of the semantic information assigned to the laser point cloud. Attached Figure Description

[0062] Figure 1 This is one of the flowcharts illustrating the camera-based two-dimensional lidar point cloud semantic assignment method provided by the present invention;

[0063] Figure 2 The second schematic diagram of the process of the camera-based two-dimensional lidar point cloud semantic assignment method provided by the present invention.

[0064] Figure 3 A flowchart of the image feature extraction steps in the camera-based two-dimensional lidar point cloud semantic assignment method provided by the present invention;

[0065] Figure 4 This is a schematic diagram illustrating the principle of the camera-based two-dimensional lidar point cloud semantic assignment method provided by the present invention when the inverse projection curve is a straight line.

[0066] Figure 5 A schematic diagram illustrating the principle of the inverse projection curve being a hyperbola in the camera-based two-dimensional lidar point cloud semantic assignment method provided by this invention.

[0067] Figure 6 This is a schematic diagram illustrating the principle of the inverse projection curve being an ellipse in the camera-based two-dimensional lidar point cloud semantic assignment method provided by the present invention. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0069] Please see Figure 1 This invention provides a method for semantic assignment of two-dimensional LiDAR point clouds based on a camera, the method comprising:

[0070] The pose relationship between the camera and the 2D LiDAR is calibrated to obtain the projection transformation matrix from the camera image plane to the laser scanning plane;

[0071] Based on the target object image captured by the camera, semantic information of the corresponding pixels in the target object image is assigned by a deep learning algorithm to obtain the pixel set within the target bounding box of the target object image.

[0072] Image feature extraction is performed on the target object image to obtain the contour geometric line equation of the target object;

[0073] A point cloud map of the surrounding environment is generated using a two-dimensional lidar.

[0074] The contour geometry equation of the target object is inversely projected onto the lidar coordinate system using the projection transformation matrix to obtain the inverse projection curve.

[0075] The range of laser points hitting the target object is determined based on the inverse projection curve, and semantic information of the laser points within the range is assigned based on the pixel set within the target box of the target object image.

[0076] The camera-based two-dimensional lidar point cloud semantic assignment method provided by this invention, compared with existing lidar point cloud semantic assignment methods, abandons complex clustering algorithms, data fusion algorithms and a large number of laser point reprojection calculations. When faced with a large number of laser points, this method only needs to perform inverse projection on the contour to filter out the range of laser points hitting the target object, which improves the accuracy of searching the point cloud and the speed of semantic assignment, thus completing the lidar point cloud semantic assignment more quickly and robustly.

[0077] Furthermore, this method employs inverse projection of the contour geometric line equation of the target object onto the lidar coordinate system, thereby directly obtaining the range of laser points hitting the target object on the laser scanning plane. Compared with existing methods that use laser points captured within the image frame to represent the laser points hitting the object, this method ensures that the determination of the range of laser points hitting the target object is no longer limited by the image frame obtained by the deep learning-based target recognition algorithm, and also improves the accuracy of the semantic information assigned to the laser point cloud.

[0078] See Figure 2 Based on the above scheme, to facilitate a better understanding of the camera-based two-dimensional lidar point cloud semantic assignment method provided by this invention, the following is a detailed description:

[0079] The embodiments of the present invention can be implemented through the following steps:

[0080] 1) Calibration of the pose relationship between the camera and the lidar:

[0081] 1-1) Fix the relative positions of the camera and the 2D LiDAR, and keep the optical center of the camera and the emission point of the 2D LiDAR on the same vertical line; select a suitable target that can be observed by the camera and the LiDAR at the same time as the marker for the camera and the LiDAR calibration; any planar object in the environment with a straight line or an arc can be used as the marker; for example, the marker can be a polygonal plate, an elliptical plate or a circular plate.

[0082] 1-2) Capture images of the landmark using a camera, extract image features, and obtain the geometric line equations of the landmark's outline. (Refer to...) Figure 3 :

[0083] 1-2-1) Perform grayscale processing on the image;

[0084] 1-2-2) Image guiding filters are used to smooth the image. Due to significant noise from the external environment and during the shooting process, image guiding filters are used to smooth the image while maintaining edge integrity. When the guided image is identical to the input image, their edges remain identical; using a guiding filter ensures edge integrity.

[0085] 1-2-3) Extraction of contour geometric line equations:

[0086] Alternatively, contour geometry line equation extraction can be performed, with different feature extraction methods depending on whether the contour geometry line features of the target are straight lines or circular arcs.

[0087] 1-2-3-1) Specifically, the method for extracting the geometric features of the target's outline as straight lines includes:

[0088] The maximum inter-class variance method based on the binarization algorithm is used to automatically set the threshold to extract the target binarized region from the foreground image.

[0089] Based on the extracted binarized region, select appropriate structuring elements to perform opening or closing operations to filter edge noise, making the target boundary cleaner and reducing noise interference.

[0090] For markers with straight-line characteristics, since the laser scanning plane of the two-dimensional LiDAR is a horizontal plane, the lines intersecting with the laser scanning plane are generally perpendicular to the horizontal plane or tilted at a certain angle. When the marker is within the range that both the LiDAR and the camera can observe, the outline of the marker in the image satisfies this condition. Therefore, for the outlines that are valuable for image calibration, the Sobel vertical operator is used for edge detection to extract the outline edge pixels and reduce the number of horizontally distributed pixels, thereby reducing the computational load of subsequent line detection and improving the calibration speed.

[0091] The Hough transform is used to extract the contour line equation of the target, resulting in the line equation: ax + by + c = 0, which can be written in matrix form as [a, b, c].

[0092] 1-2-3-2) Methods for extracting the geometric features of the target's outline as arcs, including:

[0093] The equation of the ellipse containing the arc of the target contour is directly extracted using the arc-supported line segment method, resulting in the ellipse equation a1x. 2 +2a2xy+2a3x+a4y 2 +2a5y+a6=0, written in matrix A as:

[0094]

[0095] 1-3) Extraction of the coordinates of the intersection point between the two-dimensional laser scanning plane and the outline of the marker;

[0096] During calibration, the region of the laser points hitting the marker is known. The distance information returned by the lidar is used to determine the two laser beams hitting the edge of the marker, thus obtaining the angle interval [θ1, θ2] of the laser point set hitting the marker. The laser point set hitting the marker is then fitted to a straight line. To more accurately obtain the intersection point of the laser scanning plane and the marker contour, the angle resolution of the lidar is increased by half at each end of the angle interval. Make the angle range of the laser-clicked target become Then, by passing through the origin of the lidar coordinate system, a straight line is drawn with the angles of the two boundaries as the slope, intersecting the fitted straight line at two points. The coordinates of the two intersection points in the lidar coordinate system are obtained, which are the coordinates of the two contour intersection points, that is, the coordinates of the intersection points of the laser scanning plane and the contour of the marker.

[0097] It should be noted that the angle at which a lidar emits laser light is discrete, and the laser cannot precisely land on the edge of a marker. Lidar has a certain angular resolution because the angular error is uniformly distributed. When dealing with large amounts of data, the method described above of increasing the angular range can reduce the expected angular error, allowing for higher accuracy with more data.

[0098] 1-4) Replace the marker with a marker that has the same geometric line features or adjust the position of the marker, and repeat steps 1-1), 1-2), and 1-3) to obtain multiple sets of geometric line equations and intersection point coordinate data.

[0099] 1-5) Based on the camera imaging model and the projection relationship from the image plane to the lidar plane, construct the homography projection transformation matrix H from the camera image plane to the laser scanning plane. The mathematical model is described as follows:

[0100]

[0101] 1-6) Solve the camera and lidar projection transformation matrix H by using multiple sets of contour geometric line equations and contour intersection point coordinate data, as well as the geometric constraints between contour geometric lines and contour intersection points, i.e., point-line geometric constraints;

[0102] 1-6-1) The mathematical description of the geometric constraint between the outline geometric lines and the intersection points of the outline when the outline geometric lines of the target are straight lines is as follows:

[0103]

[0104] When the outline geometry of the target image is an arc, the mathematical description of the geometric constraint between the outline geometry and the intersection point of the outline is as follows:

[0105]

[0106] 1-6-2) The above solution process will adopt different solution methods according to different geometric constraints. The linear equation system constructed by the point-line geometric constraint is solved by SVD (Singular Value Decomposition); the nonlinear equation system constructed by the point-ellipse geometric constraint is solved by LM (Levenberg-Marquarelt) algorithm to perform overall optimization and iteration of the parameter model to obtain the optimal solution.

[0107] The LM optimization algorithm is employed, a classic algorithm that has been repeatedly verified to effectively solve nonlinear least squares functions. This algorithm integrates the advantages of both the Gauss-Newton method and the gradient descent method. By controlling and changing the value of the damping factor variable, i.e., adjusting the step size of each algorithm iteration, the iterative optimization efficiency is effectively improved, achieving least squares optimization of the objective function and finding the optimal rotation and translation matrix.

[0108] 2) Semantic assignment of 2D LiDAR point clouds based on contour inverse projection:

[0109] 2-1) The camera captures target objects that need to be given semantic meaning in the LiDAR point cloud map, and deep learning is used to assign semantic information to the corresponding pixels;

[0110] Any object in the environment whose outline geometry features are straight lines or arcs can be used as the target object that needs to be given semantic meaning in the lidar point cloud map.

[0111] 2-2) List out the pixel set within the deep learning target bounding box separately, and repeat step 1-2) on this pixel set to obtain the contour geometry equation of the target object;

[0112] 2-3) Use a two-dimensional lidar to generate a point cloud map of the surrounding environment;

[0113] 2-4) Using the projection transformation matrix H obtained in steps 1-6), the contour geometry equation of the target object is inversely projected onto the lidar coordinate system to obtain the inverse projection curve;

[0114] It should be noted that, based on projection geometry and camera imaging models, the aforementioned inverse projection curves will vary depending on the geometric features of different target contours; if the target object's contour features are straight lines, then the inverse projection curve will be a straight line, see [link to relevant documentation]. Figure 4 This is a schematic diagram illustrating the principle that the inverse projection curve is a straight line. Figure 4 The image only illustrates the inverse projection curve of one side of the outline; the other side of the target object will also form a straight inverse projection curve. If the outline geometry of the target object is an arc, the inverse projection curve will be a conic section. See [link to documentation]. Figure 5 and Figure 6 This is a schematic diagram illustrating the principle that the inverse projection curve is a conic section;

[0115] 2-4-1) When the above-mentioned inverse projection curve is two straight lines, the range between the two straight lines is taken as the range of the laser point hitting the target object; when the above-mentioned inverse projection curve is a conic section, if the inverse projection curve is an ellipse, then the two tangents passing through the origin of the coordinate system are selected as the inverse projection curve of the target, that is, the range between the two tangents of the ellipse passing through the origin of the lidar coordinate system is taken as the range of the laser point hitting the target object; if the inverse projection curve is a hyperbola, then its asymptotes are selected as the inverse projection curve of the target, that is, the range between the two asymptotes of the hyperbola is taken as the range of the laser point hitting the target object.

[0116] 2-5) Determine the range of laser points hitting the target object based on the slope and angle of the inverse projection curve and the angle and distance information returned by the lidar, and assign the semantic information to the laser points within the range.

[0117] It should be noted that when the inverse projection curve of the geometric lines of the target contour is two straight lines or a hyperbola, the intersection of the two straight lines or asymptotes generally does not coincide with the origin; generally, the horizontal distance between the object and the camera and lidar is much greater than the horizontal distance between the camera and lidar, and the positional deviation of the intersection point does not affect the final experimental results.

[0118] Furthermore, the camera-based two-dimensional LiDAR point cloud semantic assignment method provided by this invention is a two-dimensional LiDAR point cloud semantic assignment algorithm based on contour inverse projection, which can be mainly divided into the following operations:

[0119] Image feature extraction involves extracting the contour geometric line equations of objects identified within the camera's field of view.

[0120] Two-dimensional LiDAR point cloud data extraction operation: Using two-dimensional LiDAR, a point cloud map of the surrounding environment is generated;

[0121] The camera and lidar data calibration steps are used to obtain the projection transformation matrix from the camera image plane to the laser scanning plane;

[0122] The semantic assignment step of 2D LiDAR point cloud based on contour inverse projection uses the projection transformation matrix obtained in the camera and LiDAR data calibration step to inversely project the contour geometric line equation of the target object obtained in the image feature extraction step onto the laser scanning plane to obtain the inverse projection curve. This determines the range of laser points hitting the target object and assigns semantic information to the laser points, that is, to establish a target point cloud group to assign semantic information.

[0123] Furthermore, the image feature extraction operation is specifically divided into: extracting the contour geometric line equations of the markers used in camera and LiDAR data calibration; and extracting the contour geometric line equations of environmental target objects that need to be located in the LiDAR point cloud map and assigned semantic information in the semantic assignment operation of 2D LiDAR point cloud based on contour inverse projection. The image feature extraction steps specifically include: grayscale processing, image edge-preserving filtering, and target contour geometric line equation extraction.

[0124] In this invention, the image feature extraction step includes: a basic image smoothing process, which allows the image to maintain the complete landmark region and its boundary features even under conditions of clutter and noise in the scene. Then, feature extraction methods based on Hough transform and arc support segments are applied to different types of target contours to more accurately obtain the straight line or curve equations of the desired target's edge contour.

[0125] Optionally, for the extraction of the geometric line equation of the target contour, if the intersection of the target contour and the laser scanning plane is a straight line feature, the Otsu's method is used to extract the target foreground region, the Sobel vertical operator is used to extract the edge pixels of the target contour, the opening and closing operation is used to filter edge noise, and the Hough transform is used to extract the straight line equation; if the intersection of the target contour and the laser scanning plane is a circular arc feature, the arc support line segment method is used directly to extract the ellipse equation.

[0126] The two-dimensional lidar point cloud data extraction operation is specifically divided into: a camera and lidar data calibration step, in which the lidar returns the coordinates and angle information of the laser points that hit the markers obtained by its scan, and obtains the coordinates of the intersection point of the laser scanning plane and the outline features of the markers; and a two-dimensional lidar point cloud map of the surrounding environment, in which the lidar scans the environment around, returns the coordinates and angle information of the laser points that hit the markers obtained by its scan, and generates the two-dimensional lidar point cloud map of the surrounding environment.

[0127] In this embodiment of the invention, the camera and lidar data calibration step further includes: using a camera imaging model and projection transformation, converting the data association from the camera image plane to the laser scanning plane into a projection transformation relationship from the front view to the top view of the observed environmental object, constructing a homographic projection transformation matrix from the camera image plane to the laser scanning plane; and constructing point-line geometric constraints on the laser point coordinates of the hit marker obtained in the two-dimensional lidar point cloud data extraction step and the feature equation of the marker contour line obtained in the image feature extraction step, thereby solving the projection transformation matrix from the camera image plane to the laser scanning plane.

[0128] In this invention embodiment, the semantic assignment step for two-dimensional LiDAR point clouds based on contour inverse projection further includes: using a deep learning image recognition algorithm to identify the target object and obtain a target bounding box; applying the image feature extraction step again to the pixel set within the target bounding box to obtain the contour geometric line equation of the environmental target object; and inversely projecting the projection matrix obtained in the camera and LiDAR data calibration step onto the laser scanning plane to obtain an inverse projection curve. Based on the properties of the inverse projection curve and the laser point angle information returned by the LiDAR in the two-dimensional LiDAR point cloud data extraction step, the angle range of the laser point hitting the target object can be quickly locked. Furthermore, by establishing a target point cloud group, the point cloud of the hit target can be extracted more accurately, thus completing the semantic assignment of the two-dimensional LiDAR point cloud.

[0129] In an embodiment of the present invention, optionally, the step of semantic assignment of two-dimensional lidar point cloud based on contour inverse projection, according to the projection geometry and camera imaging model, will obtain different types of inverse projection curves for target objects with different contours. If the geometric lines of the target edge are straight lines, the inverse projection curve is a straight line; if the geometric lines of the target edge are circular arcs, the inverse projection curve is a conic curve.

[0130] In an embodiment of the present invention, optionally, in the step of semantic assignment of two-dimensional lidar point cloud based on contour inverse projection, if the inverse projection curve is an ellipse, then the two tangents passing through the origin of the coordinate system are selected as the inverse projection curve of the target; if the inverse projection curve is a hyperbola, then its asymptote is selected as the inverse projection curve of the target.

[0131] In this embodiment of the invention, the step of semantic assignment of two-dimensional lidar point cloud based on contour inverse projection can determine the range of laser points hitting the target based on the angle between the inverse projection curve and the angle information returned by the lidar.

[0132] Furthermore, the present invention also provides a camera-based two-dimensional lidar point cloud semantic assignment system. This system is used to implement the methods described in any of the above embodiments, and can be understood in correspondence with the methods. The system includes:

[0133] The calibration module is used to calibrate the pose relationship between the camera and the 2D LiDAR, and to obtain the projection transformation matrix from the camera image plane to the laser scanning plane.

[0134] The pixel semantic acquisition module is used to assign semantic information to the corresponding pixels of the target object image based on the target object image captured by the camera, and to obtain the pixel set within the target box of the target object image;

[0135] The image feature extraction module is used to extract image features from the target object image and obtain the contour geometric line equation of the target object;

[0136] The point cloud data generation module is used to generate point cloud maps of the surrounding environment using a two-dimensional LiDAR.

[0137] The inverse projection module is used to inversely project the contour geometric line equation of the target object onto the lidar coordinate system through the projection transformation matrix to obtain the inverse projection curve.

[0138] The point cloud semantic assignment module is used to determine the range of laser points hitting the target object based on the inverse projection curve, and to assign semantic information of the laser points within the range based on the pixel set within the target box of the target object image.

[0139] Furthermore, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the camera-based two-dimensional lidar point cloud semantic assignment method as described in any of the above embodiments.

[0140] Furthermore, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the camera-based two-dimensional lidar point cloud semantic assignment method as described in any of the above embodiments.

[0141] This invention discloses a semantic assignment algorithm for two-dimensional lidar point clouds based on contour inverse projection, belonging to the field of lidar point cloud semantic assignment. The algorithm disclosed in this invention has low computational cost, strong adaptability, and can more quickly and robustly complete the assignment of semantic information to environmental targets.

[0142] It should be noted that, for the sake of simplicity, the above methods or process embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0143] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for semantic assignment of two-dimensional lidar point clouds based on a camera, characterized in that, include: The pose relationship between the camera and the 2D LiDAR is calibrated to obtain the projection transformation matrix from the camera image plane to the laser scanning plane; Based on the target object image captured by the camera, semantic information of the corresponding pixels in the target object image is assigned by a deep learning algorithm to obtain the pixel set within the target bounding box of the target object image. Image feature extraction is performed on the target object image to obtain the contour geometric line equation of the target object; A point cloud map of the surrounding environment is generated using a two-dimensional LiDAR. The projection transformation matrix is ​​used to inversely project the geometric line equation of the target object's contour onto the lidar coordinate system to obtain the inverse projection curve. The range of laser points hitting the target object is determined based on the inverse projection curve, and semantic information of the laser points within the range is assigned based on the pixel set within the target box of the target object image. Specifically, it includes: The relative positions of the fixed camera and the 2D LiDAR are kept so that the optical center of the camera and the 2D LiDAR are on the same vertical line. A planar object with a straight or circular geometric outline is used as a marker. Image features are extracted from the images of the landmark captured by the camera to obtain the geometric line equations of the landmark's outline; Extract the coordinates of the intersection point between the laser scanning plane and the outline of the marker; Replace or adjust the position of the markers to obtain multiple sets of contour geometric line equations and contour intersection coordinate data corresponding to multiple markers; Based on the camera imaging model and the projection relationship from the camera image plane to the laser scanning plane, a mathematical model of the projection transformation matrix is ​​constructed. The projection transformation matrix is ​​obtained by solving the mathematical model using multiple sets of contour geometric line equations, contour intersection point coordinate data, and geometric constraints between contour geometric lines and contour intersection points. Extracting the coordinates of the intersection point between the laser scanning plane and the outline of the marker, specifically including: Based on the known set of laser points that hit the marker, obtain the first angular range of the laser point set. ; The laser point set is fitted to a straight line; Change the first angle interval to the second angle interval. , where α is the angular resolution of the two-dimensional lidar; Using the angles at both ends of the second angle interval as the slope, a straight line passing through the origin of the lidar coordinate system is drawn and intersects the fitted straight line at two points to obtain the coordinates of the two contour intersection points; The inverse projection curve is obtained by inversely projecting the contour geometry equation of the target object onto the lidar coordinate system using the projection transformation matrix. Specifically, this includes: When the outline geometry of the target object is a straight line, the inverse projection curve consists of two straight lines. When the outline geometry of the target object is a circular arc, the inverse projection curve is an ellipse or a hyperbola. The range of the laser point hitting the target object is determined based on the inverse projection curve, specifically including: When the inverse projection curve consists of two straight lines, the range between the two straight lines is taken as the range of the laser point that hits the target object. When the inverse projection curve is an ellipse, the range between the two tangents of the ellipse passing through the origin of the lidar coordinate system is taken as the range of the laser point hitting the target object. When the inverse projection curve is a hyperbola, the range between the two asymptotes of the hyperbola is taken as the range of the laser point that hits the target object.

2. The method for semantic assignment of two-dimensional LiDAR point clouds based on a camera as described in claim 1, characterized in that, The image feature extraction steps specifically include: Perform grayscale processing on the image; The image after grayscale processing is smoothed using an image-guided filter. Extract the contour geometry equations from the smoothed image.

3. The method for semantic assignment of two-dimensional LiDAR point clouds based on cameras as described in claim 2, characterized in that, When the contour geometry of the target is a straight line, the extraction of the contour geometry equation specifically includes: The maximum inter-class variance method is used to extract the foreground image and obtain the target binarized region. Based on the extracted target binarized region, edge noise is filtered using opening and closing operations; The Sobel vertical operator is used for edge detection to extract contour edge pixels; The Hough transform is used to extract the equations of straight lines, which are then used as the equations of the contour geometry lines. When the outline geometry of the target is an arc, the extraction of the outline geometry equation specifically includes: The ellipse equation is extracted using the arc-supported line segment method and used as the equation for the contour geometry lines.

4. The method for semantic assignment of two-dimensional LiDAR point clouds based on cameras as described in claim 1, characterized in that, The mathematical model of the projection transformation matrix is ​​as follows: ； in, It is a constant; These are pixel coordinates; These are the pixel coordinates of the camera's optical center; the physical dimensions of the pixel are length and width, respectively. and , The focal length of the camera; It is the rotation transformation matrix from the lidar coordinate system to the camera coordinate system; It is the translation transformation matrix from the lidar coordinate system to the camera coordinate system; The coordinates of the point in the lidar coordinate system are ; It represents the combination of the preceding matrices.

5. The method for semantic assignment of two-dimensional lidar point clouds based on a camera as described in claim 1, characterized in that, When the outline geometry of the target is a straight line, the geometric constraints between the outline geometry and the intersection point of the outline are as follows: ； in,[ a,b,c ] Parameters of the contour geometry equation extracted when the contour geometry of the target is a straight line; H The projection transformation matrix is; These are coordinates in the lidar coordinate system. When the outline geometry of the target being photographed is an arc, the geometric constraints between the outline geometry and the intersection point of the outline are as follows: ； in, A The parameters of the contour geometry equation extracted when the contour geometry of the target is an arc.

6. A camera-based two-dimensional lidar point cloud semantic assignment system, characterized in that, The method for semantic assignment of camera-based two-dimensional LiDAR point clouds according to any one of claims 1-5 includes: The calibration module is used to calibrate the pose relationship between the camera and the 2D LiDAR, and to obtain the projection transformation matrix from the camera image plane to the laser scanning plane. The pixel semantic acquisition module is used to assign semantic information to the corresponding pixels of the target object image based on the target object image captured by the camera, and to obtain the pixel set within the target box of the target object image; The image feature extraction module is used to extract image features from the target object image and obtain the contour geometric line equation of the target object; The point cloud data generation module is used to generate point cloud maps of the surrounding environment using a two-dimensional LiDAR. The inverse projection module is used to inversely project the contour geometric line equation of the target object onto the lidar coordinate system through the projection transformation matrix to obtain the inverse projection curve. The point cloud semantic assignment module is used to determine the range of laser points hitting the target object based on the inverse projection curve, and to assign semantic information of the laser points within the range based on the pixel set within the target box of the target object image.

Images