An Image Processing Method and Device for the Fusion of Lidar and Camera

Abstract

The present invention discloses an image processing method and device for the fusion of a lidar and a camera. By acquiring the point cloud data of the lidar scanning screen, performing point cloud preprocessing on the point cloud data according to the pose information of the lidar to obtain grid information, performing edge recognition on the grid information to obtain predicted point clouds and real point clouds, solving the spatial planes where the most point clouds fall on the three axes of the grid body respectively based on the predicted point clouds and the real point clouds, performing information fusion using an edge detection graphic recognition algorithm based on the pose information, the image collected by the camera, and the spatial planes to obtain a grid map, and acquiring the virtual pose of the camera in the grid map to match and locate with the actual pose of the camera and perform mapping to complete the image processing of the lidar and camera fusion, improving the accuracy of the target fusion recognition result, and at the same time, the size and spatial position information of the actual screen body can be accurately read, reducing the time of the manual adjustment mapping process and improving the application process efficiency.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and particularly relates to an image processing method and device for the fusion of lidar and camera. Background Art

[0002] In recent years, the XR virtual shooting technology has made great breakthroughs. As the core function of virtual shooting, mapping projection requires the fusion of the actual camera shooting image and the virtual scene. Currently, mapping technologies all use regular graphic models such as cubes and patches as the projection models for mapping, and align the edges of the images captured by the camera according to a fixed rectangular model to achieve a consistent mapping effect. During the mapping process, it is necessary to continuously manually correct the size ratio, angle offset, and coordinate displacement of the projection model according to the fusion effect. The trimming accuracy needs to reach the millimeter level to achieve seamless alignment in the synthesis rendering layer and the shooting layer, and there should be no obvious tearing problems during the tracking process of more than 30 frames. However, in the prior art, since the projected screen model for mapping is a regular rectangular patch or cube, during the actual construction process, the screen to be projected will have construction errors and various gaps caused by the splicing of multiple screens, which is very different from the projected model, resulting in inconsistent mapping and problems such as tearing and crossing during the XR expansion and movement. Summary of the Invention

[0003] In view of this, the present invention provides an image processing method and device for the fusion of lidar and camera, which can accurately read the size and spatial position information of the actual screen body, reduce mapping errors, and improve the efficiency of the application process, so as to solve the above-mentioned existing technical problems, and specifically adopt the following technical solutions to achieve.

[0004] In a first aspect, the present invention provides an image processing method for the fusion of lidar and camera, including the following steps:

[0005] Obtain the point cloud data of the screen scanned by the lidar, and perform point cloud preprocessing on the point cloud data according to the pose information of the lidar to obtain grid information;

[0006] Perform edge recognition on the grid information to obtain predicted point cloud and real point cloud, and solve the spatial planes where the most point clouds fall on the three axes of the grid body according to the predicted point cloud and the real point cloud;

[0007] Based on the pose information, the image collected by the camera, and the spatial plane, use the edge detection graphic recognition algorithm to perform information fusion to obtain a grid map;

[0008] Obtain the virtual pose of the camera in the grid map and match and locate it with the actual pose of the camera for mapping to complete the image processing of lidar-camera fusion. Among them, the actual pose of the camera is calibrated according to the pose information of the lidar.

[0009] As a further optimization of the above technical solution, edge recognition is performed on the grid information to obtain the predicted point cloud and the real point cloud, including:

[0010] Take the lidar point cloud and the RGB image as inputs, convert the three-dimensional point cloud into a two-dimensional depth map, preset the internal parameter P and the external parameter T, and the projection expression is , where x represents the three-dimensional point in the point cloud, and y represents the two-dimensional point in the converted depth map;

[0011] Obtain the uncalibrated depth map by means of adjustment and use it as the input depth map. The added external parameter is , then the input depth map and the random transformation The corresponding expressions are respectively , where represents the rotation vector of the random transformation, represents the translation vector of the random transformation;

[0012] Input the RGB image and the uncalibrated depth map into the feature extraction network and output the rotation vector and translation vector of three degrees of freedom through the feature aggregation network. The calibration network receives the uncalibrated depth map and the corresponding RGB image as inputs and predicts the rotation vector and translation vector;

[0013] Use the predicted rotation vector and the translation vector , convert them into a transformation matrix to further calculate the loss function. The translation vector is directly used as the translation term in the transformation matrix, and the rotation vector is converted into a rotation matrix through the Rodriguez rotation expression , and the rotation expression is , where, I represents the identity matrix ， represents the rotation vector The skew-symmetric matrix of represents the rotation angle, combined with the translation vector to obtain the predicted transformation matrix .

[0014] As a further optimization of the above technical solution, the loss function includes the conversion loss , the depth map loss and the point cloud loss , , and representing their respective loss weights, the total loss function expression is ;

[0015] Transformation loss: The goal is to regress the rotation vector and the translation vector, which are the outputs of the calibration network. Calculate the L-2 norms between the predicted and the true rotation vectors and translation vectors respectively. By adding a scalar to control the scale difference between the L-2 norm of rotation and the L-2 norm of translation, the transformation loss , where represents the predicted rotation vector, represents the true rotation vector, represents the predicted translation vector, represents the true translation vector;

[0016] Depth map loss: Set the predicted transformation matrix , apply the transformation to the input depth map and obtain the predicted depth map, and calculate the deviation between the predicted depth map and the true depth map to get the loss of the depth map The expression of is , where x represents the three-dimensional points in the point cloud, represents the corresponding two-dimensional points in the predicted depth map,

[0017] Point cloud loss: Obtain the predicted point cloud and the true point cloud by back-projecting the predicted depth map and the true depth map, and use the Chamfer distance CD between these two point clouds as the loss function. The expression of the point cloud loss is:

[0018] , where represents the predicted point cloud, represents the true point cloud, N represents the number of points in , M represents the number of points in . As a further optimization of the above technical solution, solve the spatial planes where the most point clouds fall on the three axes of the grid body according to the predicted point cloud and the true point cloud, including:

[0019] Segment and cluster the predicted point cloud and the true point cloud to estimate the expression of the plane model corresponding to the spatial plane as , where d represents the distance from the origin of the lidar coordinates to the projection screen, and a, b, and c respectively represent the Cartesian components of the normal vector of the projection plane. The three-dimensional spatial point cloud of the lidar is projected onto a two-dimensional plane. Two points are randomly selected on the plane to determine a straight line, and the distances from the remaining points to this straight line are calculated. If the distance is less than the threshold, it meets the condition. This loop judgment is performed until the straight line with the most attached points, that is, the spatial plane, is found;

[0020] The DBSCAN algorithm is used to find all dense regions of the sample points and form clustering clusters. The execution process of the DBSCAN algorithm is as follows:

[0021] Step 1: Input the neighborhood, the minimum number of points in a class, and the lidar point cloud PCD data;

[0022] Step 2: Randomly select a point from the point cloud data and count the number of remaining points in the neighborhood of this point;

[0023] Step 3: If the number of points is greater than or equal to minPts, this point is defined as a core point; otherwise, this point is defined as a noise point. Repeat Steps 2 and 3;

[0024] Step 4: Continue to select a point in the neighborhood of this point. Taking the new point as the center of the sphere, count the number of remaining points in the neighborhood of the new point. If the number of points is greater than or equal to minPts, the new point is defined as a core point; otherwise, the new point is defined as a boundary point;

[0025] Step 5: Continue to execute Steps 3 to 5 to find the clustering point cloud of this type of target, and loop from Step 2 to find new targets;

[0026] Among them, the neighborhood represents a parameter greater than 0, the radius with any point p in the lidar point cloud as the origin; the minimum number of points in a class minPts represents the minimum number of sample points within a circle drawn with any point cloud p as the origin and as the radius; the core object means that if the number of sample points within any circle is greater than minPts, then any point p is a core object; the core point represents the set of cluster points closest to any point p, then p is a core point; the boundary point represents the set of points that are not core points but are density-reachable from core points; the noise point represents the points not within any circle, neither core points nor boundary points; density-reachable means for the sample point set D, if there exists any point sequence , if from to is directly density-reachable, then from to is density-reachable; directly density-reachable means that if any point q is within the circle of p, then from p to q is directly density-reachable.

[0027] As a further optimization of the above technical solution, an edge detection graphic recognition algorithm is used to perform information fusion on the pose information, the image collected by the camera, and the spatial plane to obtain a grid map, including:

[0028] The point cloud data H of the preset lidar is input into the spatial transformation network, and the parameters required for spatial transformation are output as , and after the transformation of the parameters is realized through the transformation matrix, according to the parameters and the transformation form, the mapping between Q and the input point cloud is output as , and the expression for completing the corresponding transformation of the point cloud coordinates is:

[0029] , where and respectively represent the input point position coordinates and the position coordinates after spatial transformation, represents the mapping relationship;

[0030] All the coordinates of the preset output V, the coordinates of the input point cloud data H, and the operation based on each coordinate are completed. The integer coordinates of H are obtained through the bilinear interpolation algorithm, and the features of H are collected according to the integer coordinate points. The collected features are incorporated into the output Q, and the corresponding expression is:

[0031] , where a represents the number of point clouds of the input lidar, c represents the number of point clouds to be transformed, and the spatial transformation network is used for the input layer and the convolutional layer of the convolutional neural network;

[0032] The point with the largest distance from any selected point is used as the starting point and iterated continuously until a valuable global point is obtained. To obtain the local shape feature, three-level encoded convolution operations need to be completed in the X, Y, and Z axis directions of the center point to ensure that the center point feature belongs to the tensor , then the expression for the three-level convolution is , where the convolution weights to be optimized in the X, Y, and Z axis directions are , and , the convolutions in the X, Y, and Z axis directions are , and , L represents the activation function, and the d-dimensional vector in the center point neighborhood is described through the convolution operations encoded in the X, Y, and Z axis directions to describe each point cloud.

[0033] As a further optimization of the above technical solution, the lidar point cloud and the camera image information are synchronized in time and space, and the point cloud and the image information are fused at the target level, including:

[0034] First, project the three-dimensional rectangular box of the target point cloud after lidar point cloud clustering onto the image to convert it into a two-dimensional envelope box of the target point cloud, and then perform data association with the two-dimensional envelope box of the target image pixels obtained by the image recognition algorithm. It is preset that the coordinates of the two-dimensional envelope box of the target point cloud and the two-dimensional envelope box of the target image pixels are respectively 、 , then the coordinates of the intersection part of the lidar point cloud two-dimensional envelope box and the camera image pixel two-dimensional envelope box are , and the areas of are respectively obtained 、 the area of and the two intersection parts area The expression is , and the intersection over union of and is obtained ;

[0035] Use the ratio of the area of the intersection part of the lidar point cloud two-dimensional envelope box and the camera image two-dimensional envelope box to the area of the camera image pixel two-dimensional envelope box to represent the matching result P of the two sensors. The expression of P is , where, after the lidar point cloud two-dimensional envelope box and the camera image pixel two-dimensional envelope box are projected, calculate the ratio P of the area of the intersection part of the two to the area of the camera image pixel two-dimensional envelope box. When P exceeds the preset threshold, it means that the lidar and the camera recognize the same target; when multiple Ps meet the preset threshold requirements, select the intersection part with the largest area to obtain the semantic and spatial coordinate information of the target.

[0036] As a further optimization of the above technical solution, the execution process of the edge detection graphic recognition algorithm includes:

[0037] Obtain the static feature points of the current frame of the image, and match the edge points and plane points in the static feature points respectively. Denote the static feature points of the current frame as , and the matching expression is , where 、 respectively represent the set of edge points and the set of plane points in, denote the static feature points of the subgraph corresponding to the current frame as , then there is , where 、 represent the set of edge points and the set of plane points in, represents the edge points converted to the global coordinate system, represents the plane points converted to the global coordinate system;

[0038] Edge points exist in the form of lines in three-dimensional space. Edge point matching is achieved by constructing point-to-line constraints. Assume point i is one of the edge points in . Use the nearest neighbor search algorithm to find the corresponding points u and v of point i in ;

[0039] Assume j is one of the edge points in . Use the nearest neighbor search algorithm to find points u, v, and w of point j in the sub-graph . After constructing the residual models of point-line constraints and point-plane constraints, the established cost function is Construct as the expression of the optimization equation as , where the matrix is the first derivative of with respect to . represents the radius of the trust region, and D represents the coefficient matrix;

[0040] Construct the expression of the Lagrangian function as , where represents the coefficient factor. The pose estimation expression is obtained by iterative solution using the gradient descent method as . When the Youha algorithm converges, the accurate pose estimation value of the current frame is obtained .

[0041] As a further preference of the above technical solution, the point cloud data obtained by lidar scanning is located under the lidar three-dimensional coordinate axes. The pose information of the lidar is obtained by translating and rotating the lidar coordinate system to the camera coordinate system. The camera maps the points in three-dimensional space to pixel points in two-dimensional space, which is determined according to the camera coordinate system, image coordinate system, and pixel coordinate system.

[0042] As a further preference of the above technical solution, the camera image information is cached according to the sampling frequency of the lidar. When the point cloud data at a certain moment is obtained, the current moment is recorded. The similar image information close to this moment is found in the camera image data cache, and the two kinds of information are synchronously fused, and the synchronous timestamp is returned to the lidar and the camera to complete the time synchronization of the two.

[0043] In a second aspect, the present invention also provides an image processing device for lidar and camera fusion, including:

[0044] A data acquisition module, configured to acquire point cloud data of a lidar scanning screen, and perform point cloud preprocessing on the point cloud data according to the pose information of the lidar to obtain grid information;

[0045] A point cloud calculation module, configured to perform edge recognition on the grid information to obtain predicted point cloud and real point cloud, and solve the spatial planes where the most point clouds fall on the three axes of the grid body respectively according to the predicted point cloud and the real point cloud;

[0046] An information fusion module, configured to perform information fusion based on the pose information, the image collected by the camera, and the spatial plane by using an edge detection graphic recognition algorithm to obtain a grid map;

[0047] An image processing module, configured to obtain the virtual pose of the camera in the grid map and match and locate it with the actual pose of the camera for mapping to complete the image processing of the lidar-camera fusion, wherein the actual pose of the camera is calibrated according to the pose information of the lidar.

[0048] The present invention provides an image processing method and device for lidar-camera fusion. By acquiring point cloud data of a lidar scanning screen, performing point cloud preprocessing on the point cloud data according to the pose information of the lidar to obtain grid information, performing edge recognition on the grid information to obtain predicted point cloud and real point cloud, solving the spatial planes where the most point clouds fall on the three axes of the grid body respectively according to the predicted point cloud and the real point cloud, performing information fusion based on the pose information, the image collected by the camera, and the spatial plane by using an edge detection graphic recognition algorithm to obtain a grid map, and obtaining the virtual pose of the camera in the grid map and matching and locating it with the actual pose of the camera for mapping to complete the image processing of the lidar-camera fusion, the accuracy of the target fusion recognition result is improved. At the same time, the size and spatial position information of the actual screen body can be accurately read, the mapping error is reduced, the time for manually adjusting the mapping process is also reduced, and the application process efficiency is improved. Description of the Drawings

[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present invention, and therefore should not be regarded as a limitation of the scope. For those of ordinary skill in the art, other related drawings can be obtained according to these drawings without creative efforts.

[0050] Figure 1 It is a flowchart of the image processing method for lidar-camera fusion of the present invention;

[0051] Figure 2 It is an execution process diagram of the DBSCAN algorithm of the present invention;

[0052] Figure 3 It is a structural block diagram of an image processing device for the fusion of lidar and camera of the present invention. Specific embodiments

[0053] The embodiments of the present invention will be described in detail below. The examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals represent the same or similar elements or elements with the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention and should not be construed as a limitation of the present invention.

[0054] Refer to Figure 1 , the present invention provides an image processing method for the fusion of lidar and camera, including the following steps:

[0055] S10: Obtain the point cloud data of the lidar scanning screen, and perform point cloud preprocessing on the point cloud data according to the pose information of the lidar to obtain grid information;

[0056] S11: Perform edge recognition on the grid information to obtain predicted point cloud and real point cloud, and solve the spatial planes where the most point clouds fall on the three axes of the grid body according to the predicted point cloud and the real point cloud;

[0057] S12: Perform information fusion on the pose information, the image collected by the camera, and the spatial plane using an edge detection graphic recognition algorithm to obtain a grid map;

[0058] S13: Obtain the virtual pose of the camera in the grid map and match and locate it with the actual pose of the camera for mapping to complete the image processing of the fusion of lidar and camera, where the actual pose of the camera is calibrated according to the pose information of the lidar.

[0059] In this embodiment, performing edge recognition on the grid information to obtain predicted point cloud and real point cloud includes: taking the lidar point cloud and the RGB image as inputs, converting the three-dimensional point cloud into a two-dimensional depth map, presetting the internal parameter P and the external parameter T, and the projection expression is , where x represents the three-dimensional point in the point cloud, and y represents the two-dimensional point in the converted depth map; obtaining the uncalibrated depth map by adjusting and using it as the input depth map, and the added external parameter is , then the input depth map and the random transformation The corresponding expressions are respectively , where represents the rotation vector of the random transformation, A translation vector representing a random transformation; the RGB image and the uncalibrated depth map are input into the feature extraction network and the three degrees of freedom rotation vector and translation vector are output through the feature aggregation network. The calibration network receives the uncalibrated depth map and the corresponding RGB image as inputs and predicts the rotation vector and the translation vector; using the predicted rotation vector and the translation vector , they are converted into a transformation matrix to further calculate the loss function. The translation vector is directly used as the translation term in the transformation matrix, and the rotation vector is converted into a rotation matrix through the Rodriguez rotation expression , and the rotation expression is , where I represents the identity matrix ， represents the rotation vector 's skew-symmetric matrix, represents the rotation angle, combined with the translation vector to obtain the predicted transformation matrix .

[0060] It should be noted that the loss function includes the conversion loss , the depth map loss and the point cloud loss , , and represent their respective loss weights, and the total loss function expression is ; Conversion loss: The goal is to regress the rotation vector and the translation vector, which are the outputs of the calibration network. The L-2 norms between the predicted and true rotation vectors and translation vectors are calculated respectively. By adding a scalar to control the scale difference between the L-2 norm of rotation and the L-2 norm of translation, the conversion loss , where represents the predicted rotation vector, represents the true rotation vector, represents the predicted translation vector, represents the true translation vector; Depth map loss: Set the predicted transformation matrix , apply the transformation to the input depth map and obtain the predicted depth map, and calculate the deviation between the predicted depth map and the true depth map to get the loss of the depth map 's expression is , where x represents the three-dimensional point in the point cloud, represents the corresponding two-dimensional point in the predicted depth map, Denote the corresponding two-dimensional points in the ground truth depth map, and N denote the number of pixels in the depth map; Point cloud loss: The predicted point cloud and the ground truth point cloud are obtained by back-projecting the predicted depth map and the ground truth depth map, and the Chamfer Distance CD between these two point clouds is used as the loss function, the point cloud loss is expressed as: , where denotes the predicted point cloud, denotes the ground truth point cloud, N denotes the number of points in , and M denotes the number of points in .

[0061] In addition, in image preprocessing, the magnitudes of the frequency domain components in the lidar image are distinguished. The image information contained in the low-frequency component part of the image is relatively blurred, mainly including relevant image information such as the background and contours. The component information with low frequency in the frequency domain affects the contrast of the image and interferes with information recognition. The linear transformation algorithm is used to process the contrast of the lidar image so that the contrast of the image reaches an equilibrium state. After decomposing the lidar image into high and low frequency components, the original detail information and some noise in the image are retained in the high-frequency component part. This noise seriously affects the details and clarity, and the noise needs to be separated to effectively and completely retain the detail information of the image. In order to effectively prevent the uneven pixels existing in the lidar image from affecting the integrity of the edge information and the effect of edge detection, the multi-scale uneven filtering algorithm can be used to adjust the pixel feature point information of the image, and by analyzing the structural relationship between the gray value of the pixel point and the weighted gray density of the pixel, the feature filtering of the pixel point is realized.

[0062] It should be understood that the point cloud preprocessing can be region of interest division, point cloud filtering, and point cloud segmentation. The amount of original point cloud data is large, and there is a lot of non-essential point cloud data, which affects the target recognition effect. In order to facilitate target recognition, it is necessary to filter out non-essential point clouds and reduce the number of point clouds. When the lidar scans the LED screen, there is a certain relationship between the laser beam and the target surface. Let a, b, c, and d be the four point clouds generated by the lidar beam scanning the target. The lidar beam scans the target through the rotation angle , and since the angle is small, it is considered that the distances from points a, b, c, and d to the lidar are equal. The calculation expression for the arc length between a and b is , where is the vertical resolution, is the horizontal resolution. The four points formed by the adjacent two laser beams of the lidar scanning the target surface are approximately connected to form a rectangle. For the point cloud clustering radius on one surface of the same target, it can be set as the diagonal length of this rectangle After setting the minimum neighborhood value and the minimum number of clusters, the size of rectangle abcd is related to the distance from the lidar to the target. When the target is close to the lidar, the rectangle is smaller and the diagonal is shorter; when the target is far from the lidar, the rectangle is larger and the diagonal is longer. Therefore, the minimum clustering radius can be adaptively adjusted according to the distance from the lidar to the target, thus solving the problem of poor clustering effect of target point clouds caused by distance. It improves the accuracy of the target fusion recognition result, and at the same time can accurately read the size and spatial position information of the actual screen body, reduce the mapping error, and also reduce the time of manual adjustment of the mapping process, improving the efficiency of the application process.

[0063] Refer to Figure 2 , optionally, solve the spatial planes where the most point clouds fall on the three axes of the grid body according to the predicted point cloud and the real point cloud, including:

[0064] The expression of the plane model corresponding to the spatial plane estimated by performing point cloud segmentation and point cloud clustering on the predicted point cloud and the real point cloud is , where d represents the distance from the origin of the lidar coordinate to the projection screen, and a, b, and c respectively represent the Cartesian components of the normal vector of the projection plane. The three-dimensional spatial point cloud of the lidar is projected onto a two-dimensional plane. Two points are randomly selected on the plane to determine a straight line, and the distances from the remaining points to this straight line are calculated. If the distance is less than the threshold, it meets the condition. This loop judgment is performed until the straight line with the most attached points, that is, the spatial plane, is found;

[0065] Use the DBSCAN algorithm to find all the dense regions of the sample points and form clustering clusters. The execution process of the DBSCAN algorithm is as follows:

[0066] Step 1: Input neighborhood, minimum number of points in a class, and lidar point cloud PCD data;

[0067] Step 2: Randomly select a point from the point cloud data and count the number of the remaining points in its neighborhood;

[0068] Step 3: If the number of points is greater than or equal to minPts, this point is defined as a core point; otherwise, this point is defined as a noise point. Repeat steps 2 and 3;

[0069] Step 4: Continue to select a point in its neighborhood. Taking the new point as the center of the sphere, count the number of the remaining points in the neighborhood of the new point. If the number of points is greater than or equal to minPts, the new point is defined as a core point; otherwise, the new point is defined as a boundary point;

[0070] Step 5: Continue to execute Steps 3 to 5 to find the target clustering point cloud of this type, and loop from Step 2 to find new targets;

[0071] Among them, Neighborhood representation parameter is greater than 0, which is the radius with any point p of the lidar point cloud as the origin; the minimum number of points in a class minPts represents the minimum number of samples within the circle drawn with any point cloud p as the origin and as the radius; the core object means that if the number of samples within any circle is greater than minPts, then any point p is a core object; the core point represents the set of cluster points closest to any point p, then p is a core point; the boundary point represents the set of points that are not core points but are density-reachable from core points; the noise point represents the points not within any circle, which are neither core points nor boundary points; density reachability represents the sample point set D. If there exists any point sequence , if from to is directly density-reachable, then from to is density-reachable; direct density-reachability means that if any point q is within the circle of p, then from p to q is directly density-reachable.

[0072] In this embodiment, based on the pose information, the images collected by the camera, and the spatial plane, an edge detection graphic recognition algorithm is used for information fusion to obtain a grid map, including:

[0073] The point cloud data H of the preset lidar is input into the spatial transformation network, and the output parameters required for spatial transformation are , and after the transformation of the parameters is realized through the transformation matrix, according to the parameters and the transformation form, the mapping between Q and the input point cloud is output as , and the expression for completing the corresponding transformation of the point cloud coordinates is:

[0074] , where and represent the input point position coordinates and the position coordinates after spatial transformation respectively, represents the mapping relationship;

[0075] Preset all the coordinates of the output V, and complete the operation based on the coordinates of the input point cloud data H and each coordinate. Take the integer coordinates of H through the bilinear interpolation algorithm and collect the features of H according to the integer coordinate points, and integrate the collected features into the output Q. The corresponding expression is:

[0076] , where a represents the number of point clouds of the input lidar, c represents the number of point clouds to be transformed, and the spatial transformation network is used for the input layer and convolutional layer of the convolutional neural network;

[0077] Arbitrarily select the point with the largest distance from a certain point as the starting point and continuously iterate until a valuable global point is obtained. To obtain local shape features, three-level encoded convolutional operations need to be completed in the X, Y, and Z-axis directions of the center point to ensure that the center point features belong to the tensor , then the expression for the three-level convolution is , where the convolutional weights to be optimized in the X, Y, and Z-axis directions are respectively , and , the convolutions in the X, Y, and Z-axis directions are respectively , and , L represents the activation function, and the d-dimensional vector in the center point neighborhood is described through the convolutional operations in the X, Y, and Z-axis directions to describe each point cloud

[0078] It should be noted that to solve the problem of large fluctuations in target recognition results caused by lidar image point clouds, a spatial network needs to be integrated into the deep convolutional neural network for image point cloud target recognition. At the same time, the stability of the point cloud data arrangement makes it impossible for the point cloud data to be effectively input into the deep convolutional neural network. Through the max-pooling differential symmetric function processing, a deep convolutional neural network point cloud feature extraction network for image point cloud target recognition is constructed. After convolution, a large number of MLPs are generated in the network. Take the convolutional kernel as the first layer structure of the network. This layer is used to process the input of the three-dimensional coordinates of the lidar image. The convolutional kernel size of the subsequent layers of this layer . The deep convolutional neural network uses two spatial transformation networks and two MLPs to map the input lidar three-dimensional features to a high-dimensional space and uses the max-pooling layer to take the global features. To obtain the final image point cloud target recognition result, k scores need to be obtained through the fully connected layer and the softmax output result is connected

[0079] As a further optimization of the above technical solution, the lidar point cloud and camera image information are synchronized in time and space, and the point cloud and image information are fused at the target level, including:

[0080] First, project the three-dimensional rectangular frame of the target point cloud after clustering the lidar point cloud onto the image to convert it into a two-dimensional envelope frame of the target point cloud, and then perform data association with the two-dimensional envelope frame of the target image pixels obtained by the image recognition algorithm. It is preset that the coordinates of the two-dimensional envelope frame of the target point cloud and the two-dimensional envelope frame of the target image pixels are respectively

[0081] , , then the coordinate of the intersection part of the two-dimensional envelope frame of the lidar point cloud and the two-dimensional envelope frame of the camera image pixels is , respectively obtain area of , area of and the areas of two intersection parts area The expression is , correspondingly obtain and intersection over union of ;

[0082] The matching result P of the two sensors is represented by the ratio of the area of the intersection part of the two-dimensional bounding box of the lidar point cloud and the two-dimensional bounding box of the camera image to the area of the two-dimensional bounding box of the camera image pixels. The expression of P is , where, after the two-dimensional bounding box of the lidar point cloud and the two-dimensional bounding box of the camera image pixels are projected, calculate the ratio P of the area of the intersection part of the two to the area of the two-dimensional bounding box of the camera image pixels. When P exceeds the preset threshold, it means that the lidar and the camera recognize the same target; when multiple Ps meet the preset threshold requirements, select the intersection part with the largest area to obtain the semantic and spatial coordinate information of the target.

[0083] Optionally, the execution process of the edge detection graphic recognition algorithm includes:

[0084] Obtain the static feature points of the current frame of the image, and match the edge points and plane points in the static feature points respectively. Denote the static feature points of the current frame as , and the matching expression is , where , respectively represent the set of edge points and the set of plane points in , then there is , where , represent the set of edge points and the set of plane points in represents the edge points transformed into the global coordinate system, represents the plane points transformed into the global coordinate system;

[0085] The edge points exist in the form of lines in three-dimensional space. The matching of the edge points is achieved by constructing the constraint from points to lines. Assume that the preset point i is an edge point in , use the nearest neighbor search algorithm to find the corresponding points u and v of point i in ;

[0086] Assume that j is For an edge point in, use the nearest neighbor search algorithm to find points u, v, and w of point j in the subgraph . The points u, v, and w form a plane, and the residual expression from point to plane is constructed as . After completing the construction of the residual model for point-line constraint and point-plane constraint, the established cost function is ;

[0087] Taking as the expression of the optimization equation is , where the matrix is the first derivative of with respect to, represents the radius of the trust region, and D represents the coefficient matrix;

[0088] The expression for constructing the Lagrangian function is , where represents the coefficient factor, and the expression for pose estimation obtained by iterative solution through the gradient descent method is . When the Youha algorithm converges, the accurate pose estimation value of the current frame is obtained .

[0089] In this embodiment, the point cloud data obtained by lidar scanning is located under the three-dimensional coordinate axes of the lidar. The pose information of the lidar is obtained by translating and rotating the lidar coordinate system to the camera coordinate system. The camera maps the points in the three-dimensional space to the pixel points in the two-dimensional space, which is determined according to the camera coordinate system, image coordinate system, and pixel coordinate system. The camera image information is cached according to the sampling frequency of the lidar. When the point cloud data at a certain moment is obtained, the current moment is recorded. The similar image information close to this moment is found in the camera image data cache, and the two kinds of information are synchronously fused, and the synchronous timestamp is returned to the lidar and the camera to complete the time synchronization of the two.

[0090] It should be noted that in order to achieve the consistency of point cloud data and image information, the lidar coordinate system is converted to the camera coordinate system through translation and rotation coordinates, and then the coordinate conversion from the camera coordinate system to the pixel coordinate system is completed. Finally, the complete conversion matrix from the lidar coordinate system to the pixel coordinate system is obtained. In practical applications, the sampling periods of each sensor are different. Therefore, the collected data will be asynchronous in time. There are significant differences in the data collected by the lidar and the camera at a certain moment. Data fusion can only be performed by collecting data at the same moment. For example, the sensor refresh frequency can be increased to reduce the time deviation, and the PPS time source can be used to keep synchronized with the host. The host sends the timestamp synchronization request to each sensor to make each sensor on the same time axis.

[0091] Refer to Figure 3 , the present invention also provides an image processing device for the fusion of lidar and camera, including:

[0092] A data acquisition module, configured to acquire the point cloud data of the lidar scanning the screen, and perform point cloud preprocessing on the point cloud data according to the pose information of the lidar to obtain grid information;

[0093] A point cloud calculation module, configured to perform edge recognition on the grid information to obtain predicted point cloud and real point cloud, and solve the spatial planes where the most point clouds fall on the three axes of the grid body respectively according to the predicted point cloud and the real point cloud;

[0094] An information fusion module, configured to perform information fusion based on the pose information, the image collected by the camera, and the spatial plane by using an edge detection graphic recognition algorithm to obtain a grid map;

[0095] An image processing module, configured to obtain the virtual pose of the camera in the grid map and match and locate it with the actual pose of the camera to complete the image processing of the lidar and camera fusion, wherein the actual pose of the camera is calibrated according to the pose information of the lidar.

[0096] In this embodiment, after the point cloud data scanned by the lidar is topologized into a 3D model and then mapped, the process of automatically aligning the three-dimensional grid data is as follows: install the lidar and record the height and horizontal of the lidar; scan the LED screen through the lidar to obtain the screen point cloud data; clean and denoise the obtained point cloud data and then perform topology to form a grid body; perform edge recognition on the formed grid body and solve the spatial planes where the most point clouds fall on the x, y, and z axes respectively; fix the camera to the position of the lidar, and use the edge detection graphic recognition algorithm to automatically and seamlessly fuse the image collected by the camera and the generated spatial plane to achieve the alignment effect; obtain the virtual camera lens file and send it to the real camera for automatic mapping. It improves the accuracy of the target fusion recognition result, and at the same time can accurately read the size and spatial position information of the actual screen body, reduce the mapping error, and also reduce the time for manually adjusting the mapping process, improving the efficiency of the application process.

[0097] In all the examples shown and described here, any specific value should be construed as merely exemplary, rather than as a limitation. Therefore, other examples of the exemplary embodiments may have different values.

[0098] It should be noted that: similar reference numerals and letters denote similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0099] The above-described embodiments merely represent several implementation manners of the present invention. The description thereof is relatively specific and detailed, but it should not be construed as a limitation to the scope of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can still be made, and these all fall within the protection scope of the present invention.

Claims

1. An image processing method for the fusion of lidar and camera, characterized in that, It includes the following steps: Obtain the point cloud data of the lidar scanning screen, and perform point cloud preprocessing on the point cloud data according to the pose information of the lidar to obtain grid information; Perform edge recognition on the grid information to obtain predicted point cloud and real point cloud, and solve the spatial planes where the most point clouds fall on the three axes of the grid body respectively according to the predicted point cloud and the real point cloud; Based on the pose information, the images collected by the camera, and the spatial plane, use the edge detection graphic recognition algorithm to perform information fusion to obtain a grid map; Obtain the virtual pose of the camera in the grid map and match and locate it with the actual pose of the camera to complete the image processing of lidar-camera fusion. Among them, the actual pose of the camera is calibrated according to the pose information of the lidar; Perform edge recognition on the grid information to obtain predicted point cloud and real point cloud, including: Taking lidar point cloud and RGB image as inputs, converting the three-dimensional point cloud into a two-dimensional depth map, presetting the intrinsic parameter P and the extrinsic parameter T, the expression for projection is , where x represents the three-dimensional point in the point cloud and y represents the two-dimensional point in the converted depth map; An uncalibrated depth map is obtained by adjustment and used as the input depth map. The added external parameters are , then the input depth map and the random transformation The corresponding expressions are respectively , where represents the rotation vector of the random transformation,[[]] represents the translation vector of the random transformation; The RGB image and the uncalibrated depth map are input into the feature extraction network and the three degrees of freedom rotation vector and translation vector are output through the feature aggregation network. The calibration network receives the uncalibrated depth map and the corresponding RGB image as inputs and predicts the rotation vector and translation vector; Using the predicted rotation vector and the translation vector , they are converted into a transformation matrix to further calculate the loss function. The translation vector is directly used as the translation term in the transformation matrix, and the rotation vector is converted into a rotation matrix through the Rodriguez rotation expression . The rotation expression is , where I represents the identity matrix ， represents the rotation vector 's skew-symmetric matrix, represents the rotation angle, combined with the translation vector to obtain the predicted transformation matrix ; The loss function includes the conversion loss , the depth map loss and the point cloud loss , , and represent the respective loss weights, and the total loss function expression is ; Conversion loss: The goal is to regress the rotation vector and the translation vector, which are the outputs of the calibration network. Calculate the L-2 norms between the predicted and the ground truth rotation vectors and translation vectors respectively. By adding a scalar to control the scale difference between the L-2 norm of rotation and the L-2 norm of translation, the conversion loss is obtained, where represents the predicted rotation vector, represents the ground truth rotation vector, represents the predicted translation vector, represents the ground truth translation vector; Depth map loss: Set the predicted transformation matrix . Apply the transformation to the input depth map to obtain the predicted depth map, and calculate the deviation between the predicted depth map and the ground truth depth map to get the loss of the depth map . The expression of is , where x represents the 3D point in the point cloud, represents the corresponding 2D point in the predicted depth map, represents the corresponding 2D point in the ground truth depth map, and N represents the number of pixels in the depth map; Point cloud loss: The predicted point cloud and the ground truth point cloud are obtained by back-projecting the predicted depth map and the ground truth depth map, and the Chamfer Distance CD between these two point clouds is used as the loss function, the point cloud loss The expression of which is: , where represents the predicted point cloud, represents the ground truth point cloud, and N represents the number of points in and M represents the number of points in 2. The image processing method for the fusion of lidar and camera according to claim 1, characterized in that, Solve the spatial planes where the most point clouds fall on the three axes of the grid body respectively according to the predicted point cloud and the real point cloud, including: The expression for estimating the plane model corresponding to the spatial plane by performing point cloud segmentation and point cloud clustering on the predicted point cloud and the true point cloud is , where d represents the distance from the origin of the lidar coordinates to the projection screen, and a, b, and c respectively represent the Cartesian components of the normal vector of the projection plane. The three-dimensional spatial point cloud of the lidar is projected onto a two-dimensional plane. Two points are randomly selected on the plane to determine a straight line, and the distances from the remaining points to the straight line are calculated. If the distance is less than the threshold, it meets the conditions. This loop judgment is performed until the straight line with the most attached points, that is, the spatial plane, is found. Use the DBSCAN algorithm to find all dense regions of the sample points and form clustering clusters. The execution process of the DBSCAN algorithm is: Step 1: Input neighborhood, minimum number of points in a class, and laser point cloud PCD data; Step 2: Randomly select a point from the point cloud data and count the number of the remaining points in the neighborhood of this point in its neighborhood; Step 3: If the number of points is greater than or equal to minPts, this point is defined as a core point; otherwise, this point is defined as a noise point. Repeat steps 2 and 3; Step 4: Continue to select this point A point within the neighborhood. Using the new point as the center of the sphere, count the number of the remaining points within the neighborhood of the new point If the number of points is greater than or equal to minPts, the new point is defined as a core point; otherwise, the new point is defined as a boundary point Step 5: Continue to execute steps 3 to 5 to find the clustering point cloud of this type of target, and loop from step 2 to find new targets; Among them, Neighborhood representation parameter Greater than 0, the radius with any point p of the lidar point cloud as the origin; the minimum number of points in a class minPts represents any point cloud p as the origin, Draw a circle with a radius. The minimum number of samples inside the circle; the core object means that if the number of samples inside any circle is greater than minPts, then any point p is a core object; the core point means the set of cluster points closest to any point p, then p is a core point; the boundary point means the set of points that are not core points but are density-reachable from core points; the noise point means the points not inside any circle, neither core points nor boundary points; density reachability means the sample point set D. If there exists any point sequence , if from to is directly density-reachable, then from to is density-reachable; direct density-reachability means that if any point q is inside the circle p, then from p to q is directly density-reachable.

3. The image processing method for the fusion of lidar and camera according to claim 1, characterized in that, Based on the pose information, the images collected by the camera, and the spatial plane, use the edge detection graphic recognition algorithm to perform information fusion to obtain a grid map, including: The point cloud data H of the preset lidar is input into the spatial transformation network, and the parameters required for spatial transformation are output as , and after the transformation of the parameters is realized through the transformation matrix, according to the parameters and the transformation form, the mapping between Q and the input point cloud is output as , and the expression for completing the corresponding transformation of the point cloud coordinates is: , where and represent the input point position coordinates and the position coordinates after spatial transformation respectively, represents the mapping relationship; All coordinates of the preset output V, the coordinates of the input point cloud data H, and based on After the operations on each coordinate are completed, the integer coordinates of H are obtained through the bilinear interpolation algorithm, and the features of H are collected according to the integer coordinate points. The collected features are incorporated into the output Q. The corresponding expression is: , where a represents the number of point clouds of the input lidar, c represents the number of point clouds to be transformed, and the spatial transformation network is used for the input layer and convolutional layer of the convolutional neural network; Arbitrarily select the point with the largest distance from a certain point as the starting point and continuously iterate until a valuable global point is obtained. To obtain local shape features, three-level encoded convolution operations need to be completed in the X, Y, and Z axis directions of the center point to ensure that the center point features belong to the tensor. Then, the expression for the three-level convolution is , where the convolution weights to be optimized in the X, Y, and Z axis directions are , and , the convolutions in the X, Y, and Z axis directions are , and , L represents the activation function, and the d-dimensional vector of the center point neighborhood is described through the X, Y, and Z axis direction encoded convolution operations to describe each point cloud.

4. The image processing method for the fusion of lidar and camera according to claim 3, characterized in that, The lidar point cloud and the camera image information complete time and space synchronization, and perform object-level fusion on the point cloud and image information, including: First, project the three-dimensional rectangular box of the target point cloud after clustering the lidar point cloud onto the image to convert it into a two-dimensional envelope box of the target point cloud, and then perform data association with the two-dimensional envelope box of the target image pixels obtained by the image recognition algorithm. It is preset that the coordinates of the two-dimensional envelope box of the target point cloud and the two-dimensional envelope box of the target image pixels are respectively , , then the coordinates of the intersection part of the lidar point cloud two-dimensional envelope box and the camera image pixel two-dimensional envelope box are . Respectively obtain area , area and the two intersection parts area expression is . Correspondingly obtain and intersection over union ; The matching result P of the two sensors is represented by the ratio of the area of the intersection of the two-dimensional bounding box of the lidar point cloud and the two-dimensional bounding box of the camera image to the area of the two-dimensional bounding box of the camera image pixels. The expression of P is , where after the two-dimensional bounding box of the lidar point cloud and the two-dimensional bounding box of the camera image pixels are projected, the ratio P of the area of their intersection to the area of the two-dimensional bounding box of the camera image pixels is calculated. When P exceeds the preset threshold, it means that the lidar and the camera recognize the same target; when multiple Ps meet the preset threshold requirements, the intersection part with the largest area is selected to obtain the semantic and spatial coordinate information of the target.

5. The image processing method for the fusion of lidar and camera according to claim 1, characterized in that, The execution process of the edge detection graphic recognition algorithm includes: Obtain the static feature points of the current frame of the image, and perform matching on the edge points and plane points in the static feature points respectively. Denote the static feature points of the current frame as , and the matching expression is , where and represent the edge point set and the plane point set in respectively. Denote the static feature points of the sub-graph corresponding to the current frame as , then there is , where and represent the edge point set and the plane point set in , represents the edge points transformed to the global coordinate system, represents the plane points transformed to the global coordinate system; Edge points exist in the form of lines in three-dimensional space. Edge point matching is achieved by constructing point-to-line constraints. Assume that point i is one of the edge points in. Use the nearest neighbor search algorithm to find the corresponding points u and v of point i in . The line is formed by points u and v, and the residual expression from point to line is constructed as ; Preset j as one of the edge points in, and use the nearest neighbor search algorithm to find the points u, v, and w of point j in the subgraph . The points u, v, and w form a plane, and the residual expression from the point to the plane is constructed as . After completing the construction of the residual models for the point-line constraint and the point-plane constraint, the established cost function is ； Build The expression constructed as the optimization equation is , where the matrix is the first derivative of denotes the radius of the trust region, and D denotes the coefficient matrix; The expression for constructing the Lagrangian function is , where represents the coefficient factor, and the expression for pose estimation is solved iteratively by the gradient descent method as . When the Yuha algorithm converges, the accurate pose estimation value of the current frame is obtained .

6. The image processing method for the fusion of lidar and camera according to claim 1, characterized in that, The point cloud data obtained by lidar scanning is located under the three-dimensional coordinate axes of the lidar. The lidar coordinate system is translated and rotated to the camera coordinate system to obtain the pose information of the lidar. The camera maps the points in the three-dimensional space to the pixel points in the two-dimensional space, which is determined according to the camera coordinate system, the image coordinate system, and the pixel coordinate system.

7. The image processing method for the fusion of lidar and camera according to claim 6, characterized in that, Cache the camera image information according to the sampling frequency of the lidar. When the point cloud data at a certain moment is obtained, record the current moment, find the similar image information in the camera image data cache that is close to this moment, and synchronously fuse the two pieces of information, and return the synchronization timestamp to the lidar and the camera to complete the time synchronization of the two; 8. An image processing device for the fusion of lidar and camera of an image processing method for the fusion of lidar and camera according to any one of claims 1-7, comprising: Data acquisition module, used to obtain the point cloud data of the lidar scanning screen, and perform point cloud preprocessing on the point cloud data according to the pose information of the lidar to obtain grid information; Point cloud calculation module, used to perform edge recognition on the grid information to obtain predicted point cloud and real point cloud, and solve the spatial planes where the most point clouds fall on the three axes of the grid body respectively according to the predicted point cloud and the real point cloud; Information fusion module, used to perform information fusion based on the pose information, the images collected by the camera, and the spatial plane using the edge detection graphic recognition algorithm to obtain a grid map; An image processing module, which is used to obtain the virtual pose of the camera in the grid map and match and locate it with the actual pose of the camera to complete the image processing of lidar-camera fusion. Among them, the actual pose of the camera is calibrated according to the pose information of the lidar.

Images