A semantic-integrated lidar-vision fusion target recognition and 3D positioning method

Abstract

The application relates to the technical field of robot vision 3D positioning, in particular to a laser radar-vision fusion target recognition and 3D positioning method combined with semantics, which comprises the following steps: acquiring RGB information and laser radar point cloud data of an object; inputting the RGB information into a semantic two-dimensional positioning module; the two-dimensional positioning module obtains target plane coordinates through the method of establishing a semantic mask matrix and an EDT algorithm; inputting the laser radar point cloud data into a semantic mask compensation module; the semantic mask compensation module compensates the semantic mask matrix through the method of depth map filling and morphological edge and depth consistency constraint; inputting the laser radar point cloud data and the target plane coordinates into a spatial positioning module to obtain target space coordinates; inputting the compensated semantic mask matrix, the target plane coordinates and the target space coordinates into a visualization module to show the target recognition and 3D positioning results. The application effectively improves the two-dimensional positioning precision, the semantic mask quality and the 3D positioning robustness.

Description

Technical Field

[0001] This invention relates to the field of robot vision 3D positioning technology, specifically to a semantic-based LiDAR-visual fusion target recognition and 3D positioning method. Background Technology

[0002] With the rapid development of mobile robots, autonomous driving, and intelligent security, the demand for environmental perception and target interaction is increasing. Target recognition and spatial positioning technologies have become a key foundation for supporting autonomous decision-making and safe operation of equipment. In complex scenarios, single sensors, due to their inherent physical limitations, cannot simultaneously meet the requirements of semantic recognition and high-precision spatial positioning. Visual sensors can capture rich color and texture information, providing important basis for target category recognition, but they cannot directly obtain the depth information of the target, leading to errors in two-dimensional positioning and difficulty in accurately mapping to three-dimensional space. Although LiDAR can generate high-precision three-dimensional point cloud data, providing spatial distance and geometric structure information of the target, its point cloud resolution is low and lacks semantic annotation, making it unable to distinguish target categories when used alone.

[0003] To overcome the limitations of single sensors, the industry has gradually developed target recognition and localization technology that combines LiDAR and vision, aiming to combine the advantages of both sensors to achieve joint perception of semantic categories and spatial coordinates. Current mainstream methods typically achieve localization by simply coupling visual semantic masks with LiDAR point cloud data, but significant technical bottlenecks remain in practical applications. Existing methods generally rely on the target's centroid or geometric center as a two-dimensional localization reference. However, this strategy does not consider the diversity of target shape and mass distribution. For targets with irregular shapes or uneven mass distribution, the centroid may deviate from the target entity area or even be located outside the target, leading to systematic deviations during subsequent three-dimensional coordinate transformations. In severe cases, this can cause misjudgments of the target's position by the device. For example, in autonomous driving scenarios, misestimation of pedestrian positions may lead to obstacle avoidance decisions errors; in industrial robot scenarios, deviations in workpiece positions may cause grasping failures. Furthermore, insufficient edge accuracy of semantic segmentation also affects the reliability of fusion. To balance computational efficiency and real-time performance, lightweight semantic segmentation models are often used in practical applications to process visual images. However, these models have weak feature extraction capabilities for small targets or edge regions, making them prone to missegmentation. This results in the semantic mask failing to accurately reflect the true contour of the target, thus affecting the spatial alignment accuracy between point cloud and visual data. Meanwhile, LiDAR point cloud data often suffers from data omissions due to reduced resolution and visual blind spots. Existing solutions lack effective data compensation mechanisms and struggle to correct edge errors in semantic segmentation through the depth continuity of the point cloud, further reducing the robustness of target recognition and localization. Summary of the Invention

[0004] To address the aforementioned technical problems, such as reliance on target centroid or geometric center points leading to positioning deviations, insufficient semantic segmentation edge accuracy affecting target positioning accuracy, and the lack of compensation mechanisms for missing LiDAR point cloud data, this invention provides a semantic-based LiDAR-visual fusion method for target recognition and 3D positioning. This invention abandons the traditional centroid positioning method, fully utilizing semantic mask information to calculate the 2D coordinates of different target categories within the image, ensuring that these coordinates are always within the target mask, significantly improving the accuracy of 2D spatial positioning. Then, depth information from the point cloud is used to optimize the semantic segmentation results, reducing missegmentation and thus improving the edge accuracy of the semantic mask and its adaptability to different scenes. Finally, by fusing visual semantic information with LiDAR point clouds, different target categories are identified and their 3D spatial positioning in the world coordinate system is achieved, effectively improving the safety performance of the device during operation.

[0005] The technical means employed in this invention are as follows: A semantic-visual fusion method for target recognition and 3D localization using lidar includes the following steps: Acquire the RGB information and LiDAR point cloud data of the object; The RGB information is input into the semantic two-dimensional positioning module, which obtains the target plane coordinates by establishing a semantic mask matrix and using the EDT algorithm. The lidar point cloud data is input into the semantic mask compensation module. The semantic mask compensation module compensates the depth map and uses the compensated depth map and a method based on morphological edge and depth consistency constraints to compensate the semantic mask matrix. The laser radar point cloud data and the target planar coordinates are input into the spatial positioning module to obtain the target spatial coordinates. The compensated semantic mask matrix, target plane coordinates, and target spatial coordinates are input into the visualization module to display the results of target recognition and 3D localization.

[0006] Furthermore, the calculation process of the target plane coordinates includes: The RGB information is processed through a semantic segmentation model to obtain a pseudo-color image and a semantic mask matrix; Remove mask regions of non-target categories from the semantic mask matrix and retain mask regions of target categories to obtain the filtered semantic mask matrix; The boundary tracing algorithm is used to remove artifacts from the filtered semantic mask matrix to obtain the semantic mask matrix after artifact removal. The semantic mask matrix after artifact removal is calculated using the Euclidean distance transformation method to obtain the target plane coordinates.

[0007] Furthermore, the method of using Euclidean distance transformation to calculate the semantic mask matrix after artifact removal to obtain the target plane coordinates includes: The set of all pixels belonging to non-target categories in the semantic mask matrix after artifact removal is taken as the obstacle point set; Iterate through each column of the semantic mask matrix after artifact removal, find the pixels belonging to the obstacle point set, calculate the minimum squared distance from the pixel to the nearest obstacle point, and use the minimum squared distance from the pixel to the nearest obstacle point as the value of the consumption function to obtain the updated consumption function. The formula for calculating the minimum squared distance is as follows:

[0008] in, It is the squared minimum distance from the pixel to the nearest obstacle. q For pixels, For the obstacle point set, f ( q ) is the consumption function of obstacles. y The ordinate of the pixel is y q The ordinate of the nearest obstacle point; Traverse each row of the semantic mask matrix after artifact removal, find the pixels belonging to the obstacle point set, and calculate the minimum squared distance from the pixel to all obstacle points based on the updated consumption function. The formula for calculating the minimum squared distance from the pixel to all obstacle points is as follows:

[0009] in, It is the squared minimum distance from the pixel to all obstacle points. x The x-coordinate of the pixel is x q The x-coordinate of the nearest obstacle point. Updated consumption function; The minimum squared distance from a pixel to all obstacle points is taken as the squared Euclidean distance from the pixel to the nearest obstacle. The maximum value of the squared Euclidean distance is found, and the pixel coordinates corresponding to the maximum value are taken as the target plane coordinates.

[0010] Furthermore, the formula for calculating the filtered semantic mask matrix includes:

[0011] in, This is the filtered semantic mask matrix. i is the x-coordinate of the pixel position. j y is the ordinate of the pixel position. To represent pixel position Category tags, The set of target categories that need to be retained.

[0012] Furthermore, the process of eliminating artifacts in the filtered semantic mask matrix using a boundary tracing algorithm includes: The first non-zero pixel is found from the filtered semantic mask matrix. The formula for calculating the non-zero pixel is as follows:

[0013] in, For the first non-zero pixel, The x-coordinate of a non-zero pixel. The ordinate of non-zero pixels ，y For any value on the ordinate; Define a counterclockwise direction vector, which includes an 8-neighborhood offset. The formula for calculating the counterclockwise direction vector is:

[0014] in, D It is a counterclockwise vector. D 0~ D 7 represents the offset from the first neighborhood to the eighth neighborhood; In the filtered semantic mask matrix, starting from the point to the left of the first non-zero pixel, search clockwise for the first non-zero point A in the 8-neighborhood of the point to the left of the first non-zero pixel. n Starting from the point to the left of the first non-zero pixel, according to the counterclockwise direction vector, find the first non-zero point A1 in the 8-neighborhood of the point to the left of the non-zero pixel. With A1 as the center and the first non-zero pixel as the starting point, according to the counterclockwise direction vector, find the first non-zero point A2 in the 8-neighborhood of the first non-zero pixel. Repeat this counterclockwise traversal of all pixels in the filtered semantic mask matrix until A is found. n End, output all non-zero points, construct a contour point set based on all non-zero points, the contour point set includes several contours; The area of ​​the contour region for each contour is calculated based on the set of contour points. The formula for calculating the area of ​​the contour region is as follows:

[0015] in, For the first i’ Area of ​​each outline region For the first i’ The x-coordinate of the j'-th point in the contour For the first i’The ordinate of the j'-th point in the contour. For the first i’ The x-coordinate of the (j'+1)th point in the contour. For the first i’ The ordinate of the (j'+1)th point in the contour. n For the first i’ Number of midpoints in the contour ，j’ As the first index, i’ For the second index; Based on the area of ​​the contour region, the contour with the largest area is selected as the target contour. The contours other than the target contour are searched in the semantic mask matrix, and the coordinate pixel values ​​of the contours other than the target contour are set to 0 to obtain the semantic mask matrix after artifact removal.

[0016] Furthermore, the semantic mask compensation module compensates the depth map, including: Using intrinsic and extrinsic parameters, the lidar point cloud data is converted from the lidar coordinate system to the camera coordinate system to obtain a point cloud depth map. A weighted average fill algorithm is used to generate fill values ​​for data gaps in the point cloud depth map. These fill values ​​are then used to compensate for the data gaps in the point cloud depth map. The formula for calculating the fill values ​​is as follows:

[0017] in, For fill values, Here, x is the x-axis value of the depth map, and y is the y-axis value of the depth map. and These are the relative coordinate offsets along the x-axis and y-axis, respectively. The coordinates in the depth map are The depth value, the weight function is calculated using the following formula: ; Analyze each missing part in the point cloud depth map, and use a weighting function and average filling algorithm to fill the missing parts of the point cloud depth map with filling values ​​to obtain a compensated point cloud depth map.

[0018] Furthermore, the method of compensating the semantic mask matrix using the compensated depth map and a method based on morphological edge and depth consistency constraints includes: For each semantic category of the semantic mask, a binary mask is generated, and the formula for calculating the binary mask is as follows:

[0019] in, The coordinates in the semantic mask are located at The semantic segmentation category results It is a binary mask. For semantic categories; Edge maps are detected using an edge detection kernel and a binary mask, and morphological gradient operations are employed. The formula for calculating the edge map is as follows:

[0020] in, For category Edge map, For edge detection kernel, For image dilation operation, For image erosion operations, Indicates category The binary mask is used for dilation operation. This indicates that an etching operation is being performed on the binary mask; Assuming that the depth value corresponding to each edge pixel in the edge map is d, and that for each edge pixel... The depth continuity of the depth value is verified within the neighborhood using the following formula:

[0021] in, It is a Boolean variable. The depth values ​​are assumed for the edge pixels. This represents the neighborhood depth value of the edge pixel. As the depth threshold, when When the semantic category in the semantic mask matrix corresponding to the pixel is compensated, the compensated semantic mask matrix is ​​obtained. The formula for calculating the compensated semantic mask matrix is ​​as follows:

[0022] in, The compensated semantic mask matrix, Candidate semantic categories, semantic mask median coordinate The semantic segmentation category label value at the location, For coordinates The neighborhood set centered at a size of 3×3. The coordinates of each pixel in the neighborhood. Used to select in The most frequently occurring semantic category, This is an indicator function that takes the value 1 when the condition is true and 0 otherwise.

[0023] Further, the step of inputting the lidar point cloud data and the target planar coordinates into the spatial positioning module to obtain the target spatial coordinates includes: Construct transformation formulas for radar coordinate system, camera coordinate system and image coordinate system; Based on the transformation formulas of radar coordinate system, camera coordinate system and image coordinate system, the lidar point cloud data and semantic mask matrix are spatially aligned to obtain the three-dimensional point cloud coordinates of each effective pixel in the semantic mask matrix. Based on the target plane coordinates, a traversal search is performed in the three-dimensional point cloud coordinates to extract the two-dimensional point cloud coordinate set corresponding to the target plane coordinates and its 8 neighboring pixel coordinates; Find the valid 3D point cloud coordinates of each pixel in the 2D point cloud coordinate set, retain the valid 3D point cloud coordinates, and construct a valid point set based on the valid 3D point cloud coordinates; The spatial centroid of the effective point set is calculated to obtain the target spatial coordinates.

[0024] Furthermore, the transformation formulas for constructing the radar coordinate system, camera coordinate system, and image coordinate system include: Establish the transformation formula between the camera coordinate system and the radar coordinate system:

[0025] in, , and These represent the x, y, and z coordinates of the target point in the radar coordinate system, respectively. , and These represent the x, y, and z coordinates of the same target point in the camera coordinate system. This is the rotation matrix from the camera coordinate system to the radar coordinate system. This is the translation vector from the origin of the camera coordinate system to the origin of the radar coordinate system; Establish the transformation matrix between the camera coordinate system and the image coordinate system:

[0026] in, This is the transformation matrix between the camera coordinate system and the image coordinate system. and These are the equivalent focal lengths of the image in the x and y directions, respectively. The pixel coordinates of the principal point in the horizontal direction of the image. The pixel coordinates of the principal point in the vertical direction of the image; Based on the transformation matrix between the camera coordinate system and the image coordinate system, the transformation relationship between the camera coordinate system and the image coordinate system is obtained:

[0027] in, and These are the x and y coordinates of the same target point in the image coordinate system.

[0028] Compared with the prior art, the present invention has the following advantages: This invention uses visual and lidar sensors as inputs. It performs semantic segmentation on visual RGB images to obtain pseudo-color images and semantic masks. A semantic mask-guided 2D localization algorithm is then employed. A binary target mask matrix is ​​generated through category filtering. A boundary tracking algorithm is used to extract and filter target contours. Euclidean distance transformation is then used to determine 2D localization points located within the target, effectively preventing coordinate overflow. Next, a point cloud depth map is generated through coordinate transformation. A weighted average filling algorithm is used to compensate for missing areas in the depth map. Depth difference information is combined to correct the semantic mask, improving edge segmentation accuracy. Finally, rigid transformation is used to achieve coordinate system registration, aligning the point cloud data with the semantic mask space. The 3D spatial coordinates of the 2D localization points and their neighbors are traversed and searched. The 3D localization coordinates of the target in the world coordinate system are obtained through spatial centroid calculation.

[0029] This invention effectively improves the accuracy of 2D positioning, the quality of semantic masks, and the robustness of 3D positioning, and can provide reliable target recognition and 3D spatial positioning support for the autonomous decision-making and safe operation of equipment in fields such as mobile robots, autonomous driving, and intelligent security.

[0030] Based on the above reasons, this invention can be widely applied in fields such as robot vision 3D positioning. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 This is a flowchart illustrating a semantic-based lidar-visual fusion target recognition and 3D localization method according to the present invention.

[0033] Figure 2 This is a visual sensor data diagram from an embodiment of the present invention.

[0034] Figure 3 This is a schematic diagram of two-dimensional positioning according to an embodiment of the present invention.

[0035] Figure 4 This is a point cloud depth map according to an embodiment of the present invention.

[0036] Figure 5This is the point cloud depth map after being filled by the algorithm in this embodiment of the invention.

[0037] Figure 6 This is a schematic diagram of the three-dimensional spatial positioning results in an embodiment of the present invention. Detailed Implementation

[0038] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0039] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0040] like Figure 1 As shown, this invention provides a semantic-based lidar-visual fusion target recognition and 3D localization method, the specific steps of which are as follows: S1. Obtain the RGB information of the object and the point cloud data of the LiDAR.

[0041] Specifically, data from visual sensors and laser sensors are read, and images from different sources and LiDAR point cloud data are synchronized in time to obtain data to be fused under the same time conditions. Then, the RGB image input from the visual sensor is segmented based on a semantic segmentation model to obtain a pseudo-color image and a category index image (semantic mask).

[0042] This method mainly includes a vision module, a lidar module, a semantic 2D localization module, a semantic mask compensation module, a spatial localization module that fuses semantic and lidar data, and a visualization module, such as... Figure 1As shown. A Livox-Horizon solid-state LiDAR is used, which can accumulate scan frames over time and output 3D point cloud data including distance, reflection intensity, and timestamps, with a sampling frequency of 10 FPS. The vision sensor uses an OAK-D camera, which can acquire RGB information of objects within a field of view parallel to the LiDAR, providing input information for semantic segmentation visual processing, such as... Figure 2 As shown, by reading data from visual sensors and laser sensors, image and lidar point cloud data from different sources are synchronized in time to obtain data to be fused under unified time conditions.

[0043] S2. Input the RGB information into the semantic 2D localization module. The 2D localization module obtains the target plane coordinates by establishing a semantic mask matrix and using the EDT algorithm.

[0044] Current 2D object localization methods generally rely on the object's center of gravity as a position estimate. However, since the object's center of gravity depends on its mass distribution and geometry, for objects with uniform mass distribution and regular shapes, the center of gravity is located at its geometric center. For irregular objects, the center of gravity may deviate from its geometric center, or even be located outside the object. Given these limitations, regardless of whether the center point or the center of gravity is used as the 2D localization strategy, there is a possibility that the localization point will be located outside the target object, leading to positional deviations in subsequent 3D spatial localization. Therefore, to overcome this problem, we use a semantic mask-guided 2D localization algorithm. By performing a category-based information filtering operation on the semantic mask, all masks of non-target category instances are removed, generating a binarized target mask matrix. Then, a boundary-based tracking algorithm is used to extract all outer contour point sets in the binarized mask, filtering out all contours that do not conform to the target category. Finally, the Euclidean distance transform algorithm is used to solve for the maximum inscribed circle center of the target contour region, and this center is used as the 2D image localization point of the target. This method ensures that the positioning point is always located inside the object, thereby improving the robustness and accuracy of 2D image positioning and 3D spatial positioning.

[0045] Specifically, S2 includes: S21. Obtain a pseudo-color image and a semantic mask matrix by using the RGB information through a semantic segmentation model.

[0046] The semantic segmentation model PP_LiteSeg is used to segment RGB images input from a visual sensor to obtain pseudo-color images and category index images (semantic masks).

[0047] S22. Remove the mask regions of non-target categories from the semantic mask matrix, retain the mask regions of the target categories, and obtain the filtered semantic mask matrix.

[0048] Let the mask prediction result output by the model inference be... (A two-dimensional matrix) is subjected to a category-based information filtering operation, removing masks of all non-target category instances and retaining only the mask regions corresponding to target category instances. The values ​​of the target mask regions are set to 255, and the values ​​of non-target regions are set to 0, ultimately generating a binarized target mask matrix. Let... The input semantic mask matrix (height) ,width ), each element Indicates pixel position Category tags, This is the set of target categories that need to be retained. Therefore, the filtered semantic mask matrix... The calculation formulas include:

[0049] in, This is the filtered semantic mask matrix. i is the x-coordinate of the pixel position. j y is the ordinate of the pixel position. To represent pixel position Category tags, The set of target categories that need to be retained.

[0050] S23. Use the boundary tracing algorithm to remove artifacts from the filtered semantic mask matrix to obtain the semantic mask matrix after artifact removal.

[0051] S231. Considering that instances of the same category may be over-segmented in the semantic segmentation results (i.e., generating multiple discrete interference regions), these erroneous mask artifacts need to be filtered out. First, a boundary-based tracking algorithm is used to extract the set of all outer contour points in the binary mask. This algorithm scans the given binary mask matrix... Find the first non-zero pixel from the filtered semantic mask matrix. The formula for calculating a non-zero pixel is:

[0052] in, For the first non-zero pixel, The x-coordinate of a non-zero pixel. The ordinate of non-zero pixels ，y For any value on the vertical axis.

[0053] Given any y-value on the vertical axis that is less than y0, we can obtain the first non-zero coordinate where there are no non-zero values ​​above it (on the vertical axis). A0 is the first non-zero coordinate, and so on, A1 is the second non-zero coordinate, and A2 is the third non-zero coordinate.

[0054] S232. Define a counterclockwise direction vector. The counterclockwise direction vector includes the offset of the 8-neighborhood. The formula for calculating the counterclockwise direction vector is:

[0055] in, D It is a counterclockwise vector. D 0~ D 7 represents the offset from the first neighborhood to the eighth neighborhood.

[0056] S233. In the filtered semantic mask matrix, starting from the first non-zero pixel... Starting from the point on the left, search clockwise for the first non-zero point A in the 8-neighborhood of the point to the left of the first non-zero pixel. n From non-zero pixels Starting from the point on the left, according to the counterclockwise direction vector, find the first non-zero point A1 in the 8-neighborhood of the point to the left of the non-zero pixel. Using A1 as the center, and starting from the first non-zero pixel... Starting from a point, and based on the counter-clockwise direction vector, find the first non-zero point A2 in the 8-neighborhood of the first non-zero pixel. This search process assumes that the first non-zero point is a point... With point as the center, Find the first non-zero point in the 8-neighborhood of the starting point in a counter-clockwise direction:

[0057] in, Given the pixel position as the center of the circle in the current iteration, search for the next non-zero point within its 8-neighborhood. For the next non-zero point found, for The x-coordinate of the point for The ordinate of the point, This represents the neighborhood offset of the horizontal axis. This represents the neighborhood offset of the ordinate. This represents the direction vector of the 8-neighborhood.

[0058] This process is repeated counterclockwise, iterating and tracking the pixel coordinates that satisfy the boundary point conditions, traversing all pixels in the filtered semantic mask matrix, until the desired boundary point is found. The process ends with a point, outputting all non-zero points, and then constructing a discrete set of contour points based on these non-zero points. The contour point set includes several contours. Each contour contains all the ordered pixel coordinates that make up that contour:

[0059] in, For the first to the Kth contours, Let n be a non-zero point in the i-th contour, and let n be the n-th non-zero point in the i-th contour.

[0060] S234. To eliminate artifact interference caused by oversegmentation, the area of ​​the contour region for each contour is calculated based on the contour point set. The formula for calculating the area of ​​the contour region is:

[0061] in, For the first i’ Area of ​​each outline region For the first i’ The x-coordinate of the j'-th point in the contour For the first i’ The ordinate of the j'-th point in the contour. For the first i’ The x-coordinate of the (j'+1)th point in the contour. For the first i’ The ordinate of the (j'+1)th point in the contour. n For the first i’ Number of midpoints in the contour ，j’ As the first index, i’ This is the second index.

[0062] S235. Based on the area of ​​the contour region, select the contour with the largest area as the target contour:

[0063] in, The target contour is defined. Based on the target contour, the coordinate pixel values ​​of other contours in the semantic mask matrix are set to 0 to obtain the semantic mask matrix after artifact removal.

[0064] S24. The semantic mask matrix after artifact removal is calculated using the Euclidean distance transformation method to obtain the target plane coordinates.

[0065] S241. Treat the set of all pixels belonging to the non-target category in the semantic mask matrix after artifact removal as the obstacle point set. ,for Each point in q Its consumption function is defined as f ( q When calculating the target category pixels in the semantic mask matrix... p To the obstacle point set When calculating the distance, if a one-dimensional distance transformation is used, then... f ( q If the distance is 0, then if a two-dimensional distance transformation is used, then f (q ) is calculated by a one-dimensional transformation. q The initial distance value from the point to the nearest obstacle in its column.

[0066] S242. For each column of the semantic mask matrix, find the set of points belonging to obstacles in that column. The pixels are considered obstacles, for each column of pixels. Calculate the nearest obstacle point in that column. The formula for calculating the squared distance and the minimum squared distance is:

[0067] in, It is the squared minimum distance from the pixel to the nearest obstacle. q For pixels, For the obstacle point set, f ( q ) is the consumption function of obstacles. y The ordinate of the pixel is y q Let be the ordinate of the nearest obstacle point. Since it's a one-dimensional distance transformation, the consumption function... f ( q The value is 0.

[0068] S243. After the above one-dimensional transformation, for the obstacle point set Every point in q Calculate it in the above formula The result is used as the updated consumption function. f ( q The value of )

[0069] in, This is the updated consumption function.

[0070] S244. For each row of pixels in the semantic mask matrix Calculate the distance from the point to all obstacle points on that row. The squared distance, combined with the new consumption function f’ ( q The calculation formula is:

[0071] in, For pixels p To the obstacle point set The minimum total cost, i.e. p arrive The square of the shortest path distance, xThe x-coordinate of the pixel is x q The x-coordinate of the nearest obstacle point.

[0072] After the above transformation, for each target category pixel in the semantic mask matrix p The obtained This is the square of the Euclidean distance from the target pixel to the nearest obstacle (i.e., a non-target pixel that forms the boundary), and iterating through all target pixel locations. Value, maximum The corresponding pixel coordinates are the coordinates of the center of the inscribed circle, and are used as the planar coordinate positioning point of the target. For example... Figure 3 As shown, by performing category filtering, binarization, and boundary tracking on the semantic mask results obtained from semantic segmentation, the image localization coordinates of all targets in the two-dimensional image are obtained, and different categories of targets are covered with different colors.

[0073] S3. Input the LiDAR point cloud data into the semantic mask compensation module. The semantic mask compensation module compensates the depth map and uses the compensated depth map and the method based on morphological edge and depth consistency constraints to compensate the semantic mask matrix.

[0074] To address the insufficient segmentation accuracy of lightweight semantic segmentation models at small object edges, we employ a semantic mask compensation method that integrates radar point cloud depth information. First, the point cloud is transformed from the radar coordinate system to the camera coordinate system, generating a corresponding point cloud depth map. Due to the blind spots in radar point clouds, there are data gaps in the upper and lower regions of the depth map in the camera coordinate system. Therefore, a weighted average filling algorithm is used to compensate for these missing areas. To ensure the smoothness of depth filling and suppress errors caused by numerical instability, an inverse weighting strategy is adopted, where the weight of the depth value is inversely proportional to its magnitude. Since larger depth values ​​typically correspond to more distant object surfaces with relatively higher measurement uncertainty, while smaller depth values ​​correspond to closer objects with higher measurement confidence, a weighting function is used to reduce the impact of distant depth values ​​on the filling result, effectively suppressing depth jump phenomena and maintaining the geometric consistency of the scene structure. Then, based on the filled depth map, the semantic mask categories corresponding to pixel regions with large depth differences are compensated and corrected using its depth difference information. This repairs mis-segmented and missed-segmented pixels at the segmentation edges of each category, improving the edge segmentation accuracy of semantic segmentation.

[0075] Specifically, S3 includes: S31. Use intrinsic and extrinsic parameters to convert the lidar point cloud data from the lidar coordinate system to the camera coordinate system to obtain a point cloud depth map.

[0076] like Figure 4 The image shows a point cloud depth map with the same size as an RGB image, generated by using a Livox-Horizon solid-state lidar and coordinate transformation to convert the 3D point cloud data into a point cloud depth map.

[0077] S32. Due to the blind spots in radar point clouds, there are data gaps in the upper and lower parts of the depth map in the camera coordinate system. Therefore, a weighted average filling algorithm is used to generate filling values ​​for the data gaps in the point cloud depth map. These filling values ​​are then used to compensate for the data gaps in the point cloud depth map. For each hole pixel in the depth map, it is set to 0. The formula for calculating the fill value is:

[0078] in, For fill values, Here, x is the x-axis value of the depth map, and y is the y-axis value of the depth map. and These are the relative coordinate offsets along the x-axis and y-axis, respectively. The coordinates in the depth map are The depth value, the weight function is calculated using the following formula: .

[0079] S33. Analyze each missing part in the point cloud depth map, and fill the missing parts of the point cloud depth map with filling values ​​using a weighting function and an average filling algorithm to obtain the compensated point cloud depth map. To ensure smooth depth filling and suppress errors caused by numerical instability, an inverse weighting strategy is adopted, meaning the weight of each depth value is inversely proportional to its magnitude. Larger depth values ​​typically correspond to more distant object surfaces, resulting in higher measurement uncertainty; conversely, smaller depth values ​​correspond to closer objects, offering higher measurement confidence. The weighting function reduces the impact of distant depth values ​​on the filling result, effectively suppressing depth jumps and maintaining the geometric consistency of the scene structure. The filling algorithm iterates until no new holes are filled. Figure 5 The image shows the point cloud depth map after the algorithm fills in the data. The missing areas of the depth map are filled and compensated by using a weighted average filling algorithm, so that each pixel in the depth map contains depth information. An inverse weighting strategy is also used to suppress depth jump phenomenon.

[0080] S34.. For each semantic category, generate a binary mask. The formula for calculating the binary mask is:

[0081] in, The coordinates in the semantic mask are located at The semantic segmentation category results It is a binary mask. For semantic categories, .

[0082] S35. Using an edge detection kernel and a binary mask, morphological gradient operations are used to detect the edge map. The formula for calculating the edge map is:

[0083] in, For category Edge map, for Edge detection kernel, For image dilation operation, i.e., if the edge detection kernel If any foreground pixel exists within the coverage area, the center pixel is set as the foreground pixel. For image erosion operations, i.e., only when the edge detection kernel... The center pixel is set as foreground only when all pixels within the coverage area are foreground pixels. Indicates category Binary mask Perform an expansion operation. Indicates to The image obtained by performing an erosion operation and comparing the two results is the image obtained from the difference between them. , used to characterize the boundary contour of this category.

[0084] S36. Assume that the depth value corresponding to each edge pixel in the edge map is d, and that for each edge pixel... The depth continuity is verified within the neighborhood using the following formula:

[0085] in, This is a Boolean variable used to determine whether the continuity of the current edge pixel in depth space meets the threshold. Requirements The depth values ​​are assumed for the edge pixels. This represents the neighborhood depth value of the edge pixel. As the depth threshold, when When this occurs, it indicates that the edge pixel has a depth discontinuity, and the semantic category in the semantic mask matrix corresponding to that pixel needs to be compensated to obtain the compensated semantic mask matrix. The formula for calculating the compensated semantic mask matrix is ​​as follows:

[0086] in, The compensated semantic mask matrix, Candidate semantic categories, semantic mask median coordinate The semantic segmentation category label value at the location, For coordinates The neighborhood set centered at the point and of size 3×3, containing the point and its surrounding elements (up, down, left, right, and diagonal), totaling 9 pixels. The coordinates of each pixel in the neighborhood. Used to select in The most frequently occurring semantic category, This is an indicator function that takes the value 1 when the condition is true and 0 otherwise. The algorithm improves the accuracy of the original semantic segmentation at object boundaries by combining depth continuity detection and semantic neighborhood information.

[0087] S4. Input the lidar point cloud data and target plane coordinates into the spatial positioning module to obtain the target spatial coordinates.

[0088] Because LiDAR and visual cameras are installed in different spatial positions and each has its own independent coordinate system, the spatial coordinates of the same target in different sensor coordinate systems will differ. Therefore, to achieve the unified position representation required for multi-sensor fusion perception, rigid transformation is used for coordinate system registration. That is, a specific rotation matrix and translation vector are applied to the coordinate points in the camera coordinate system to transform them to the coordinates in the LiDAR coordinate system, and vice versa. Utilizing the transformation relationship between the LiDAR and camera coordinate systems, the point cloud data acquired by the LiDAR is spatially aligned with the semantic mask generated by image semantic segmentation. Each effective pixel in the mask is assigned a corresponding 3D point cloud coordinate. Then, based on the 2D positioning coordinates of each target, a traversal search is performed on the 3D point cloud coordinate set corresponding to the semantic mask. To suppress the influence of noise and obtain more stable spatial position coordinates, the 3D coordinate point set associated with the 2D positioning coordinate point and its 8 neighboring pixels is first extracted. For each pixel position in the set, if it has a valid 3D coordinate, its corresponding 3D coordinate point is collected. Finally, the spatial centroid is calculated on the 3D coordinates of the valid point set to calculate the target's coordinate position in 3D space.

[0089] Specifically, S4 includes: S41. Construct transformation formulas for radar coordinate system, camera coordinate system and image coordinate system.

[0090] S411. Establish the transformation formula between the camera coordinate system and the radar coordinate system:

[0091] in, , and These represent the x, y, and z coordinates of the target point in the radar coordinate system, respectively. , and These represent the x, y, and z coordinates of the same target point in the camera coordinate system. This is the rotation matrix from the camera coordinate system to the radar coordinate system. Let be the translation vector from the origin of the camera coordinate system to the origin of the radar coordinate system. S412. Then, transform the camera coordinate system and the image coordinate system. Let K be the transformation matrix (intrinsic parameter matrix) from the camera coordinate system to the image coordinate system. The transformation matrix K between the camera coordinate system and the image coordinate system is as follows:

[0092] in, This is the transformation matrix between the camera coordinate system and the image coordinate system. and Let be the equivalent focal lengths of the image in the x and y directions, respectively. and Principal point coordinates, that is, the position of the intersection of the optical axis and the imaging plane in the pixel coordinate system. The pixel coordinates of the principal point in the horizontal direction of the image. The pixel coordinates of the principal point in the vertical direction of the image.

[0093] S413. Based on the transformation matrix between the camera coordinate system and the image coordinate system, obtain the transformation relationship between the camera coordinate system and the image coordinate system:

[0094] in, and These are the x and y coordinates of the same target point in the image coordinate system.

[0095] S42. Based on the transformation formulas of the radar coordinate system, camera coordinate system, and image coordinate system, the lidar point cloud data and semantic mask matrix are spatially aligned to obtain the three-dimensional point cloud coordinates of each effective pixel in the semantic mask matrix:

[0096] in, For 3D point cloud coordinates, For searching point cloud coordinate functions, For lidar point cloud data, and represents the x and y coordinates of the target point in the semantic mask matrix.

[0097] S43. Based on the target plane coordinates, a traversal search is performed in the 3D point cloud coordinates. To suppress the influence of noise and obtain a more stable spatial position coordinate, the coordinate point set of the 2D positioning coordinate point and its 8 neighboring pixels is first extracted:

[0098] in, For a two-dimensional coordinate point set, and Let x and y be the x and y coordinates of the two-dimensional image of the target point. and This represents the offset of the horizontal and vertical axes. i Let x be the x-coordinate of a pixel in the neighborhood. j It represents the ordinate of a pixel in the neighborhood.

[0099] S44. Find a set of two-dimensional coordinate points The effective 3D point cloud coordinates corresponding to the pixel positions are used to construct an effective point set:

[0100] in, For the effective point set, For effective point cloud 3D coordinates, This provides the three-dimensional spatial location information of the point. and These are coordinate points within a two-dimensional coordinate point set.

[0101] Specifically, it checks whether each point cloud coordinate in the three-dimensional point cloud coordinate set contains spatial coordinate information. Points that do not contain spatial coordinate information are removed from the point set, and only valid point cloud coordinates containing xyz information are retained. A valid point set is constructed based on the valid three-dimensional coordinates.

[0102] The 3D coordinate point set contains 9 coordinates, each of which should contain x, y, and z information. However, since radar point cloud data does not have a value for every pixel like camera data, some coordinates in the point cloud are missing (i.e., there is no x, y, or z information at that location). Therefore, not every one of the 9 coordinates in the obtained 3D coordinate point set may have x, y, and z values. Thus, it is necessary to determine the validity of each coordinate and discard invalid coordinates. The determination method is: if it is not a null value, then it is definitely a valid value.

[0103] S45. Calculate the spatial centroid of the effective point set to obtain the target spatial coordinates:

[0104] in, The x-coordinate of the target's final spatial coordinates. The y-coordinate of the target's final spatial coordinates. The z-coordinate is the final spatial coordinate of the target.

[0105] The result obtained is the coordinate position of the target in three-dimensional space under the radar coordinate system. Simultaneously, this coordinate position can be converted to coordinates in the camera coordinate system through coordinate transformation, thus completing the design of the 3D spatial positioning function. For example... Figure 6 The diagram shows the results of three-dimensional spatial positioning. By inputting point cloud data from the LiDAR and two-dimensional positioning coordinate data, coordinate transformation and spatial alignment are used to obtain the three-dimensional spatial information corresponding to the two-dimensional positioning coordinates. At the same time, centroid calculation is used to reduce the error of the three-dimensional coordinates. Finally, the 3D spatial positioning coordinates of each target are output.

[0106] S5. Input the compensated semantic mask matrix, target plane coordinates, and target spatial coordinates into the visualization module to display the results of target recognition and 3D localization.

[0107] After the obtained data is timestamped and fused, it is transmitted to the visualization interface to complete the visualization process.

[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A semantic-visual fusion method for target recognition and 3D localization using lidar, characterized in that, Includes the following steps: Acquire the RGB information and LiDAR point cloud data of the object; The RGB information is input into a semantic 2D localization module. This module obtains the target plane coordinates by establishing a semantic mask matrix and using an Euclidean distance transformation algorithm. The calculation process for the target plane coordinates includes: using the RGB information through a semantic segmentation model to obtain a pseudo-color image and a semantic mask matrix; removing mask regions of non-target categories from the semantic mask matrix and retaining mask regions of the target category to obtain a filtered semantic mask matrix; using a boundary tracing algorithm to eliminate artifacts from the filtered semantic mask matrix to obtain an artifact-free semantic mask matrix; and using an Euclidean distance transformation method to calculate the artifact-free semantic mask matrix to obtain the target plane coordinates. The lidar point cloud data is input into the semantic mask compensation module. The semantic mask compensation module compensates the depth map and uses the compensated depth map and a method based on morphological edge and depth consistency constraints to compensate the semantic mask matrix. The laser radar point cloud data and the target planar coordinates are input into the spatial positioning module to obtain the target spatial coordinates. The compensated semantic mask matrix, target plane coordinates, and target spatial coordinates are input into the visualization module to display the results of target recognition and 3D localization.

2. The semantic-based lidar-vision fusion target recognition and 3D localization method according to claim 1, characterized in that, The method of using Euclidean distance transformation to calculate the semantic mask matrix after artifact removal to obtain the target plane coordinates includes: The set of all pixels belonging to non-target categories in the semantic mask matrix after artifact removal is taken as the obstacle point set; Iterate through each column of the semantic mask matrix after artifact removal, find the pixels belonging to the obstacle point set, calculate the minimum squared distance from the pixel to the nearest obstacle point, and use the minimum squared distance from the pixel to the nearest obstacle point as the value of the consumption function to obtain the updated consumption function. The formula for calculating the minimum squared distance is as follows: in, It is the squared minimum distance from the pixel to the nearest obstacle. q For pixels, For the obstacle point set, f ( q ) is the consumption function of obstacles. y The ordinate of the pixel is y q The ordinate of the nearest obstacle point; Traverse each row of the semantic mask matrix after artifact removal, find the pixels belonging to the obstacle point set, and calculate the minimum squared distance from the pixel to all obstacle points based on the updated consumption function. The formula for calculating the minimum squared distance from the pixel to all obstacle points is as follows: in, It is the squared minimum distance from the pixel to all obstacle points. x The x-coordinate of the pixel is x q The x-coordinate of the nearest obstacle point. Updated consumption function; The minimum squared distance from a pixel to all obstacle points is taken as the squared Euclidean distance from the pixel to the nearest obstacle. The maximum value of the squared Euclidean distance is found, and the pixel coordinates corresponding to the maximum value are taken as the target plane coordinates.

3. The semantic-based lidar-vision fusion target recognition and 3D localization method according to claim 1, characterized in that, The formula for calculating the filtered semantic mask matrix includes: in, This is the filtered semantic mask matrix. i is the x-coordinate of the pixel position. j y is the ordinate of the pixel position. To represent pixel position Category tags, The set of target categories that need to be retained.

4. The semantic-based lidar-vision fusion target recognition and 3D localization method according to claim 1, characterized in that, The process of eliminating artifacts in the filtered semantic mask matrix using a boundary tracing algorithm includes: The first non-zero pixel is found from the filtered semantic mask matrix. The formula for calculating the non-zero pixel is as follows: in, For the first non-zero pixel, The x-coordinate of a non-zero pixel. The ordinate of non-zero pixels ，y For any value on the ordinate; Define a counterclockwise direction vector, which includes an 8-neighborhood offset. The formula for calculating the counterclockwise direction vector is: in, D It is a counterclockwise vector. D 0~ D 7 represents the offset from the first neighborhood to the eighth neighborhood; In the filtered semantic mask matrix, starting from the point to the left of the first non-zero pixel, search clockwise for the first non-zero point A in the 8-neighborhood of the point to the left of the first non-zero pixel. n Starting from the point to the left of the first non-zero pixel, according to the counterclockwise direction vector, find the first non-zero point A1 in the 8-neighborhood of the point to the left of the non-zero pixel. With A1 as the center and the first non-zero pixel as the starting point, according to the counterclockwise direction vector, find the first non-zero point A2 in the 8-neighborhood of the first non-zero pixel. Repeat this counterclockwise traversal of all pixels in the filtered semantic mask matrix until A is found. n End, output all non-zero points, construct a contour point set based on all non-zero points, the contour point set includes several contours; The area of ​​the contour region for each contour is calculated based on the set of contour points. The formula for calculating the area of ​​the contour region is as follows: in, For the first i’ Area of ​​each outline region For the first i’ The x-coordinate of the j'-th point in the contour For the first i’ The ordinate of the j'-th point in the contour. For the first i’ The x-coordinate of the (j'+1)th point in the contour. For the first i’ The ordinate of the (j'+1)th point in the contour. n For the first i’ Number of midpoints in the contour ，j’ As the first index, i’ For the second index; Based on the area of ​​the contour region, the contour with the largest area is selected as the target contour. The contours other than the target contour are searched in the semantic mask matrix, and the coordinate pixel values ​​of the contours other than the target contour are set to 0 to obtain the semantic mask matrix after artifact removal.

5. The semantic-based lidar-vision fusion target recognition and 3D localization method according to claim 1, characterized in that, The semantic mask compensation module compensates the depth map, including: Using intrinsic and extrinsic parameters, the lidar point cloud data is converted from the lidar coordinate system to the camera coordinate system to obtain a point cloud depth map. A weighted average fill algorithm is used to generate fill values ​​for data gaps in the point cloud depth map. These fill values ​​are then used to compensate for the data gaps in the point cloud depth map. The formula for calculating the fill values ​​is as follows: in, For fill values, Here, x is the x-axis value of the depth map, and y is the y-axis value of the depth map. and These are the relative coordinate offsets along the x-axis and y-axis, respectively. The coordinates in the depth map are The depth value, the weight function is calculated using the following formula: ； Analyze each missing part in the point cloud depth map, and use a weighting function and average filling algorithm to fill the missing parts of the point cloud depth map with filling values ​​to obtain a compensated point cloud depth map.

6. The semantic-based lidar-vision fusion target recognition and 3D localization method according to claim 1, characterized in that, The method of compensating the semantic mask matrix using the compensated depth map and a method based on morphological edge and depth consistency constraints includes: For each semantic category of the semantic mask, a binary mask is generated, and the formula for calculating the binary mask is as follows: in, The coordinates in the semantic mask are located at The semantic segmentation category results It is a binary mask. For semantic categories; Edge maps are detected using an edge detection kernel and a binary mask, and morphological gradient operations are employed. The formula for calculating the edge map is as follows: in, For category Edge map, For edge detection kernel, For image dilation operation, For image erosion operations, Indicates category The binary mask is used for dilation operation. This indicates that an etching operation is being performed on the binary mask; Assuming that the depth value corresponding to each edge pixel in the edge map is d, and that for each edge pixel... The depth continuity of the depth value is verified within the neighborhood using the following formula: in, It is a Boolean variable. The depth values ​​are assumed for the edge pixels. This represents the neighborhood depth value of the edge pixel. As the depth threshold, when When the semantic category in the semantic mask matrix corresponding to the pixel is compensated, the compensated semantic mask matrix is ​​obtained. The formula for calculating the compensated semantic mask matrix is ​​as follows: in, The compensated semantic mask matrix, Candidate semantic categories, semantic mask median coordinate The semantic segmentation category label value at the location, For coordinates The neighborhood set centered at a size of 3×3. The coordinates of each pixel in the neighborhood. Used to select in The most frequently occurring semantic category, This is an indicator function that takes the value 1 when the condition is true and 0 otherwise.

7. The semantic-based lidar-vision fusion target recognition and 3D localization method according to claim 1, characterized in that, The step of inputting the lidar point cloud data and the target plane coordinates into the spatial positioning module to obtain the target spatial coordinates includes: Construct transformation formulas for radar coordinate system, camera coordinate system and image coordinate system; Based on the transformation formulas of radar coordinate system, camera coordinate system and image coordinate system, the lidar point cloud data and semantic mask matrix are spatially aligned to obtain the three-dimensional point cloud coordinates of each effective pixel in the semantic mask matrix. Based on the target plane coordinates, a traversal search is performed in the three-dimensional point cloud coordinates to extract the two-dimensional point cloud coordinate set corresponding to the target plane coordinates and its 8 neighboring pixel coordinates; Find the valid 3D point cloud coordinates of each pixel in the 2D point cloud coordinate set, retain the valid 3D point cloud coordinates, and construct a valid point set based on the valid 3D point cloud coordinates; The spatial centroid of the effective point set is calculated to obtain the target spatial coordinates.

8. The semantic-based lidar-vision fusion target recognition and 3D localization method according to claim 7, characterized in that, The transformation formulas for constructing the radar coordinate system, camera coordinate system, and image coordinate system include: Establish the transformation formula between the camera coordinate system and the radar coordinate system: in, , and These represent the x, y, and z coordinates of the target point in the radar coordinate system, respectively. , and These represent the x, y, and z coordinates of the same target point in the camera coordinate system. This is the rotation matrix from the camera coordinate system to the radar coordinate system. This is the translation vector from the origin of the camera coordinate system to the origin of the radar coordinate system; Establish the transformation matrix between the camera coordinate system and the image coordinate system: in, This is the transformation matrix between the camera coordinate system and the image coordinate system. and These are the equivalent focal lengths of the image in the x and y directions, respectively. The pixel coordinates of the principal point in the horizontal direction of the image. The pixel coordinates of the principal point in the vertical direction of the image; Based on the transformation matrix between the camera coordinate system and the image coordinate system, the transformation relationship between the camera coordinate system and the image coordinate system is obtained: in, and These are the x and y coordinates of the same target point in the image coordinate system.

Images