Radar and vision-based 6d point cloud data fusion method, medium and device

Abstract

This application discloses a method, medium, and device for fusion of 6D point cloud data based on radar and vision. The method includes the following steps: distortion correction of the vision camera and initial alignment of the radar coordinate system and the vision coordinate system; pitch angle back projection of the original 3D point cloud of the radar based on the pitch angle of the vision camera; matching and fusing the 2D projected points with the pixels in the vision image to construct 6D point cloud data; constructing a global multimodal feature map based on the obtained 6D point cloud data; and inputting the global multimodal feature map into multiple parallel detection heads to perform parallel perception and construct a global environmental information matrix. The medium and device are used to execute the above method. The beneficial effects of this application are: by constructing 6D point cloud data, deep fusion of radar geometry and visual RGB features is achieved, compensating for the defects of single-modal perception; and the accuracy of radar-vision feature binding is improved based on pitch angle projection and proximity matching.

Description

Technical Field

[0001] This application relates to the field of robot perception technology, and in particular to a 6D point cloud data fusion method, medium and device based on radar and vision. Background Technology

[0002] Current robot perception systems mostly rely on single radar or vision sensors. Radar can only acquire three-dimensional geometric coordinates (XYZ) and lacks texture and semantic information; vision can only acquire RGB color features and lacks precise depth information. Using either alone results in low perception accuracy, weak scene understanding, and poor robustness in extreme environments. Traditional radar-vision fusion methods often remain at the level of simple feature stitching, failing to achieve deep coupling between geometry and texture. Furthermore, large models have limited understanding of single-modal data, making it difficult to simultaneously meet the perception requirements of multiple tasks such as object detection, semantic segmentation, and instance segmentation. Summary of the Invention

[0003] One objective of this application is to provide a radar and vision-based 6D point cloud data fusion method that can solve at least one of the deficiencies in the aforementioned background art.

[0004] Another object of this application is to provide a computer-readable storage medium capable of implementing a radar and vision-based 6D point cloud data fusion method that addresses at least one of the deficiencies in the aforementioned background art.

[0005] Another object of this application is to provide an electronic device capable of implementing a radar and vision-based 6D point cloud data fusion method that addresses at least one of the deficiencies in the aforementioned background art.

[0006] To achieve at least one of the above objectives, one aspect of this application provides a 6D point cloud data fusion method based on radar and vision, comprising the following steps:

[0007] S100: Perform distortion correction on the images acquired by the vision camera and establish the external parameter calibration relationship between the radar and the vision camera, so that the radar coordinate system and the vision coordinate system are initially aligned.

[0008] S200: Based on the pitch angle of the visual camera, the original 3D point cloud acquired by the radar is rotated in the opposite direction of the pitch angle and projected onto a vertical 2D image plane to generate 2D projection points, while maintaining the mapping relationship of the point cloud before and after projection.

[0009] S300: In the two-dimensional image plane, based on the principle of proximity, the two-dimensional projection points are matched with the pixels in the visual image, and the RGB color features of the pixels are extracted and fused with the original three-dimensional coordinates of the radar to construct 6D point cloud data.

[0010] S400: The obtained 6D point cloud data is input into the UNet network architecture that integrates the Transformer model. Global context features and local detail features are extracted through the encoder-decoder structure to construct a global multimodal feature map.

[0011] S500: The global multimodal feature map is input into multiple parallel detection heads to perform target detection, semantic segmentation, instance segmentation and key point detection tasks respectively. The perception information output by each detection head is normalized and fused to construct a global environment information matrix for the robot.

[0012] Preferably, in step S200, the coordinates (X, Y, X) of the original three-dimensional point cloud of the radar are... r Y r Z r ) and the coordinates (x) of the two-dimensional projection point p y p The relation is:

[0013] ;

[0014] In the formula, θ represents the pitch angle of the visual camera, and Z... proj This represents the depth information after projection.

[0015] Preferably, in step S300, the proximity principle adopts the minimum Euclidean distance criterion, so the two-dimensional projection point passes through coordinates (x... p y p The expression for matching the coordinates (u, v) of a pixel is:

[0016] ;

[0017] In the formula, (u * v * ) represents the coordinates of the pixel that meets the matching condition.

[0018] Preferably, in step S500, the execution process of the object detection task is as follows: object classification and bounding box regression are performed using the object detection head, outputting the object category, confidence score, and spatial location, and determining the relative spatial relationship of objects; the specific expression is:

[0019] ;

[0020] In the formula, C cls σ represents the class confidence, W represents the sigmoid activation function. cls and W reg b represents the classification weight and regression weight, respectively. cls and b reg Both represent the corresponding bias terms, B boxrepresents the bounding box coordinates, and F represents the global multimodal feature map.

[0021] Preferably, in step S500, the semantic segmentation task is executed as follows: pixel-level semantic annotation is performed using the semantic segmentation detection head, and a probability map of each pixel belonging to different semantic categories is output; the specific expression is as follows:

[0022] ;

[0023] In the formula, S seg Represents a pixel-level semantic probability map, where Softmax(·) represents the normalized exponential function, and W seg The segmentation weights are represented by `upSample(·)`, which represents the upsampling operator used to restore the global multimodal feature map F to the original image resolution. seg This indicates the bias term.

[0024] Preferably, in step S500, the execution process of the embodiment segmentation task is as follows: the embodiment segmentation detection head filters out graspable targets from the region of interest and outputs an instance mask to distinguish independent object instances; the specific expression is:

[0025] ;

[0026] In the formula, I ins Let W represent the instance binary mask, σ represent the Sigmoid activation function, and W represent the sigmoid activation function. ins F represents the segmentation weight of the embodiment. roi b represents the region of interest features extracted from the global multimodal feature map F. ins This indicates the bias term.

[0027] Preferably, in step S500, the execution process of the key point detection task is as follows: The key point detection head locates the object and captures the two-dimensional coordinate set of key points; the specific expression is:

[0028] ;

[0029] In the formula, K pt W represents the set of key point coordinates. kpt Let b represent the keypoint detection weights, F represent the global multimodal feature map, and b represent the keypoint detection weights. kpt This indicates the bias term.

[0030] Preferably, in step S100, the vision camera performs radial and tangential distortion correction on the image, and the specific distortion correction formula is as follows:

[0031] ;

[0032] In the formula, (u corr vcorr (u) represents the corrected pixel coordinates. raw v raw ) represents the original distorted pixel coordinates, k1 and k2 both represent radial distortion coefficients, p1 and p2 both represent tangential distortion coefficients, and r represents the Euclidean distance from the pixel to the image center.

[0033] Another aspect of this application provides a computer-readable storage medium storing a computer program; when the computer program is executed by a processor, it implements the above-described 6D point cloud data fusion method based on radar and vision.

[0034] Another aspect of this application provides an electronic device including a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program to implement the above-described radar and vision-based 6D point cloud data fusion method.

[0035] Compared with the prior art, the beneficial effects of this application are as follows:

[0036] (1) Construct 6D point cloud data to achieve deep fusion of radar geometry and visual RGB features, and make up for the defects of single-modal perception.

[0037] (2) Based on pitch angle projection and proximity matching, improve the accuracy of radar feature binding and reduce mismatches.

[0038] (3) The UNet-Transformer architecture is used to extract global features, which is adapted to the large robot model input and significantly improves the scene understanding ability.

[0039] (4) Multiple detection heads process in parallel, simultaneously meeting the needs of multiple tasks such as target detection, segmentation, grasping and positioning, resulting in more refined and comprehensive environmental perception. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the overall working steps of this application. Detailed Implementation

[0041] The present application will now be further described in conjunction with specific embodiments. It should be noted that, in the description of this specification, the use of terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicates that the specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms should not be construed as necessarily referring to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0042] In the description of this application, it should be noted that the terms "center", "lateral", "longitudinal", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", etc., which indicate the orientation and positional relationship based on the orientation or positional relationship shown in the accompanying drawings, are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and should not be construed as limiting the specific protection scope of this application.

[0043] It should be noted that the terms "first," "second," etc., in the specification and claims of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0044] In this application, unless otherwise expressly specified and limited, the terms "installation," "connection," "joining," and "fixing," etc., should be interpreted broadly. For example, they can refer to a connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0045] In this application, unless otherwise expressly specified and limited, "above" or "below" the second feature can include direct contact between the first and second features, or contact between the first and second features through another feature between them. Furthermore, "above," "over," and "on top" of the second feature includes the first feature being directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature includes the first feature being directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0046] The terms “comprising” and “having”, and any variations thereof, in the specification and claims of this application are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or device.

[0047] One aspect of this application provides a 6D point cloud data fusion method based on radar and vision, such as... Figure 1 As shown, one preferred embodiment includes the following steps:

[0048] S100: Perform distortion correction on the images acquired by the vision camera and establish the extrinsic parameter calibration relationship between the radar and the vision camera, so that the radar coordinate system and the vision coordinate system are initially aligned.

[0049] Understandably, during robot startup or system initialization, the raw images acquired by the vision camera lens often exhibit radial distortion (caused by the lens shape) and tangential distortion (caused by the lens not being parallel to the imaging plane) due to manufacturing and installation errors. These two types of distortion cause pixel positions to shift on the actual imaging plane, reducing the accuracy of subsequent feature matching; therefore, distortion correction of the vision camera is necessary. After distortion correction, to ensure successful fusion of data acquired by the vision camera and radar, it is necessary to establish extrinsic parameter calibration relationships between the vision camera and radar sensors, calibrating the overlapping areas in the visual image corresponding to the radar point cloud, and achieving precise initial alignment of the radar-visual coordinate system.

[0050] S200: Based on the pitch angle of the visual camera, the original 3D point cloud acquired by the radar is rotated in the opposite direction of the pitch angle and projected onto a vertical 2D image plane to generate 2D projection points, while maintaining the mapping relationship of the point cloud before and after projection.

[0051] Understandably, since radar point clouds are sparse in three dimensions while visual images are dense in two dimensions, direct matching would involve enormous computational costs and be prone to mismatches. Furthermore, visual cameras suffer from perspective projection effects, resulting in different image sizes for the same object at different distances. Additionally, inherent parallax exists between radar point clouds and visual images, leading to systematic biases when directly matching raw 3D coordinates with pixel coordinates. Therefore, this step projects the 3D radar point cloud back onto a virtual 2D plane based on the pitch angle of the visual camera, generating 2D projection points and establishing a one-to-one mapping between the points before and after projection.

[0052] S300: In the two-dimensional image plane, based on the principle of proximity, the two-dimensional projection points are matched with the pixels in the visual image, and the RGB color features of the pixels are extracted and fused with the original three-dimensional coordinates of the radar to construct 6D point cloud data.

[0053] Understandably, after obtaining the two-dimensional projection point set, this step performs proximity matching within the two-dimensional image plane, binding the visual RGB color features to the corresponding original three-dimensional radar coordinates, thereby constructing 6D point cloud data containing geometric and texture information. By constructing 6D point cloud data, deep fusion of radar geometry and visual RGB features is achieved, compensating for the deficiencies of single-modal perception; simultaneously, based on pitch angle projection and proximity matching, the accuracy of radar visual feature binding is improved, and mismatches are reduced.

[0054] S400: The obtained 6D point cloud data is input into the UNet network architecture that integrates the Transformer model. Global context features and local detail features are extracted through the encoder-decoder structure to construct a global multimodal feature map.

[0055] Understandably, given the massive amount of 6D point cloud data generated in step S300 and its complex spatial structure and semantic information, direct use for downstream tasks would be inefficient. This step employs an improved UNet network architecture (UNet-Transformer) that integrates the Transformer model to extract high-dimensional, globally perceptive multimodal feature maps. Using the UNet-Transformer architecture to extract global features adapts to large robot model inputs, significantly improving scene understanding capabilities.

[0056] S500: The global multimodal feature map is input into multiple parallel detection heads to perform target detection, semantic segmentation, instance segmentation and key point detection tasks respectively. The perception information output by each detection head is normalized and fused to construct a global environment information matrix for the robot.

[0057] Understandably, to efficiently meet the robot's diverse environmental perception needs, this step deploys four independent detection heads in parallel on the extracted global multimodal feature map F, each performing object detection, semantic segmentation, instance segmentation, and keypoint detection tasks respectively. All detection heads share the same feature map F, resulting in extremely high computational efficiency; furthermore, the parallel processing of multiple detection heads simultaneously satisfies the needs of multiple tasks such as object detection, segmentation, and grasping / localization, leading to more refined and comprehensive environmental perception.

[0058] Based on the above technical solutions, this application aims to address the problems of low perception accuracy, lack of texture or depth information, and weak scene understanding capabilities caused by insufficient multimodal data fusion in existing single sensors (such as pure radar or pure vision). The core idea is to deeply couple the precise three-dimensional geometric coordinates (X, Y, Z) of radar with the rich color and texture information (R, G, B) of the visual sensor to construct point cloud data containing six-dimensional information (X, Y, Z, R, G, B). Combined with an advanced neural network architecture and a multi-task detection head, this provides robots (especially mobile robots or service robots) with accurate, comprehensive, and robust environmental perception capabilities, thereby meeting the needs of parallel processing of multiple tasks such as target detection, semantic segmentation, instance segmentation, and keypoint localization. For ease of understanding, the specific working process of each step in this application will be described in detail below.

[0059] In this embodiment, during step S100, the visual camera performs radial and tangential distortion correction on the image. The specific distortion correction formula is as follows:

[0060] .

[0061] In the formula, (u corr v corr (u) represents the corrected pixel coordinates. raw v raw ) represents the original distorted pixel coordinates; k1 and k2 both represent radial distortion coefficients, and p1 and p2 both represent tangential distortion coefficients. The specific values ​​of the distortion coefficients can be obtained offline in advance using standard camera calibration tools (such as the Zhang Zhengyou calibration method) and stored in the robot system; r represents the Euclidean distance from the pixel to the image center, calculated using the following formula: .

[0062] After completing distortion correction for the visual camera, joint extrinsic parameter calibration of the radar and visual camera is performed. The goal of the calibration is to solve for a rotation and translation matrix R. ext This matrix can represent the coordinates (X, Y, Z) of the original 3D point cloud in the radar coordinate system. r Y r Z rThe image is transformed to the camera coordinate system and then projected onto the corrected image plane through the camera intrinsic parameter matrix K. The specific calculation expression is as follows:

[0063] .

[0064] In the formula, K is a 3×3 camera intrinsic parameter matrix (containing parameters such as focal length and principal point coordinates), and R ext The external parameter matrix is ​​3×4 (including rotation and translation components). By collecting multiple sets of radar point cloud data and the corresponding corner points in the visual images, the optimal rotation and translation matrix R can be solved. ext After completing this step, the radar point cloud and the visual image achieve preliminary global alignment in space, meaning that a radar point cloud can theoretically be projected onto a roughly correct area of ​​the image.

[0065] In this embodiment, when executing step S200, the pitch angle θ of the current vision camera can first be obtained from the robot's posture sensor or the vision camera's installation parameters. It should be noted that in the coordinate system defined in this application, the pitch angle is the rotation angle around the Y-axis. When the vision camera tilts up, the pitch angle θ is positive; when it tilts down, the pitch angle θ is negative.

[0066] Then, each point P in the original 3D point cloud of the radar... raw =(X r Y r Z r Rotating the original 3D point cloud by an angle θ in the opposite direction of the pitch (i.e., around the Y-axis) transforms it into an intermediate coordinate system. That is, if the visual camera's pitch angle is downward, the radar's original 3D point cloud will rotate upward, and vice versa. From the visual camera's perspective, this is equivalent to rotating the optical axis to a horizontal plane, resulting in a virtual camera. The geometric meaning of this transformation is to eliminate the effect of the visual camera's pitch on imaging, making the virtual camera's optical axis parallel to the ground (or perpendicular to the radar coordinate system's Z-axis), thus obtaining a "flattened" projection plane.

[0067] Then, the original 3D point cloud of the radar can be projected onto the 2D image plane corresponding to the intermediate coordinate system, and transformed to generate the corresponding 2D projection point P. proj =( x p y p The coordinates (X, y) of the original 3D point cloud of the radar are then obtained. r Y r Z r ) and the coordinates (x) of the two-dimensional projection point p y p The transformation relationship of ) is:

[0068] .

[0069] In the formula, θ represents the pitch angle of the visual camera, and Z... proj This represents the depth information after projection.

[0070] The above projection transformation relationship can be simplified as follows:

[0071] .

[0072] In the formula, The projection operator representing the pitch angle.

[0073] Finally, during the projection process, the system maintains a mapping table Γ, recording the original 3D points P of each radar. raw Its corresponding projection point P proj Index relationships between them: This mapping relationship is stored in the form of a hash table or array, ensuring that subsequent steps can quickly backtrack to the original 3D coordinates based on the projection point coordinates, thereby accurately filling the matched RGB color features back into the radar point cloud.

[0074] To make it easier to understand, a specific example will be used to illustrate this in detail below.

[0075] Suppose that at a certain moment, the camera on the robot has a pitch angle of θ = 10°, and the radar detects a point P located 5 meters in front, 1 meter to the left, and 0.5 meters above it. raw =(5, 1, 0.5). The specific projection transformation is calculated as follows: cos10°≈0.9848, sin10°≈0.1736; substituting these values ​​into the transformation expression, we can calculate x. p ≈4.8372, y p =1, Z proj ≈1.3604. Therefore, the projection point P... proj The coordinates are (4.8372, 1), and its horizontal distance from the origin is approximately 4.84 meters. The next step will be to find the pixel in the image that is closest to (4.837, 1.0), extract its RGB color, and backfill it to the original three-dimensional point coordinates (5, 1, 0.5) of the radar.

[0076] Compared to traditional forward projection, this application uses pitch angle backward projection to reduce the dimensionality of the 3D point cloud to a 2D plane. This simplifies the subsequent radar-vision matching problem from a 3D spatial search to a 2D plane search, significantly reducing computational complexity and improving real-time processing capabilities. Furthermore, the backward rotation eliminates system biases caused by perspective projection effects in the visual camera, making the geometric correspondence between projected points and image pixels more accurate and reducing mismatches caused by viewing angle. Simultaneously, the depth information Z of the point cloud is preserved after the transformation. projIt can be used for subsequent occlusion judgment or depth consistency verification (such as rejecting matching pairs with excessively large depth differences).

[0077] In this embodiment, when executing step S300, the nearest location principle used in the matching process is preferably the minimum Euclidean distance criterion. Therefore, each two-dimensional projection point is determined by its coordinates (x...). p y p The expression for matching the coordinates (u, v) of a pixel is:

[0078] ;

[0079] In the formula, (u * v * ) represents the coordinates of the pixel that meets the matching condition.

[0080] Understandably, pixel-level matching searches can be efficiently implemented by constructing a KD-tree (K-Dimensional Tree) or by leveraging GPU parallel acceleration. The search range can be limited to a local window (e.g., 20×20 pixels) around the projected point to reduce computational complexity.

[0081] The best matching pixel (u) of the two-dimensional projection point is found through the above matching search. * v * After that, the values ​​of the red, green, and blue channels (R, G, B) of the pixel can be directly extracted from the image data. Then, through the mapping relationship Γ established in step S200, the original radar three-dimensional coordinates (X, G, B) corresponding to the projection point can be found. r Y r Z r Finally, the two are merged to generate a 6D point cloud data P. 6D =( X r Y r Z r (R, G, B); After traversing all valid original 3D point clouds of radar, a 6D point cloud set {P} for the entire scene is obtained. 6D This data format deeply couples the geometric precision of radar with the semantic richness of vision at the raw data level, avoiding information loss during later feature stitching. For example, a radar point that originally only provided information that "there is an obstacle at coordinates (1, 2, 3)" now also has the semantic information that "the obstacle is a red, textured sphere," greatly facilitating subsequent model understanding.

[0082] In this embodiment, when performing step S400, the 6D point cloud data P is first... 6DVoxelization or projection into a pseudo-image is performed to adapt to the input format of a 2D convolutional neural network (CNN). For example, point clouds can be projected onto a 2D grid as a bird's-eye view (BEV) or front view (FV). The point cloud features (X, Y, Z, R, G, B) within each grid cell are aggregated into a feature vector through pooling operations (such as max pooling or average pooling), thus forming a C×H×W feature map. Finally, this feature map is input into the UNet network architecture that integrates the Transformer model, i.e., the UNet-Transformer network.

[0083] Specifically, the architecture of the UNet-Transformer network includes:

[0084] The encoder consists of multiple downsampling blocks, each containing convolutional and pooling layers, used to progressively extract local detail features and spatial hierarchy features while reducing the spatial size of the feature map.

[0085] The Transformer module is embedded in the bottleneck layer or skip connection of the encoder. Utilizing the Transformer's self-attention mechanism, the model can capture long-distance dependencies between any two locations in the feature map, thereby obtaining global contextual features. This is crucial for understanding large objects, occluded scenes, or tasks requiring global reasoning (such as road segmentation). For example, the Transformer module can help the model understand the spatial and semantic relationships between a distant traffic light and a nearby stop line.

[0086] The decoder consists of multiple upsampled blocks that progressively restore the spatial resolution of the feature map. During decoding, skip connections are used to concatenate the detailed features of the corresponding layer of the encoder with the features of the current layer of the decoder, thereby preserving both local details and global semantic information.

[0087] It is understandable that the entire feature extraction process can be formally represented as: F=T UNet (P 6D ), where T UNet (·) denotes the UNet-Transformer feature extraction operator, and F is the final output global multimodal feature map. This feature map F simultaneously encodes geometric structure, color texture, local details, and global contextual information, providing a rich and compact feature representation for subsequent multi-task perception.

[0088] In this embodiment, when executing step S500, each detection head can be defined as a target detection head, a semantic segmentation detection head, an embodiment segmentation detection head, and a key point detection head, depending on the task being performed. For ease of understanding, the specific working process of each detection head will be described in detail below.

[0089] For the object detection head, the process of performing object detection tasks is as follows: the object detection head performs object classification and bounding box regression, outputs object category, confidence score and spatial location, and determines the relative spatial relationship of objects.

[0090] Specifically, the object detection head employs a lightweight classification-regression parallel structure; where the classification branch outputs the class confidence C of each preset box belonging to a specific category (such as pedestrian, vehicle, chair) through a fully connected layer and a sigmoid activation function σ. cls The regression branch directly predicts the bounding box coordinates B of the object through the fully connected layer. box (Usually the center point coordinates, width, and height); the specific expression is as follows:

[0091] .

[0092] In the formula, W cls and W reg These represent classification weights and regression weights, respectively. The specific values ​​can be set according to the actual needs of those skilled in the art. cls and b reg Both represent the corresponding bias terms; among them, bias term b cls Its function is to adjust the classification threshold; in the absence of bias term b cls At that time, it is assumed by default that positive and negative samples have equal probabilities; while the bias term b cls This baseline can be raised or lowered overall; for example, when there are far more negative samples than positive samples in the dataset, a negative bias term b can be learned. cls This can lower the initial predicted probability, thus helping the model converge to the appropriate decision boundary more quickly; bias term b reg Predicted values ​​that directly affect the bounding box coordinates; for example, even if the input global multimodal feature map F is zero, the bias term b... reg It can also output a reasonable default box size or position, so that the model has reasonable initial predictions in the early stage of training and accelerates convergence.

[0093] For the semantic segmentation detection head, the process of performing semantic segmentation tasks is as follows: the semantic segmentation detection head performs pixel-level semantic annotation and outputs a probability map of each pixel belonging to different semantic categories.

[0094] Specifically, the semantic segmentation detection head first uses an upsampling operator UpSample(·) (such as bilinear interpolation or transposed convolution) to enlarge the global multimodal feature map F, restoring F to its original image resolution; then, after passing through a convolutional layer and a Softmax(·) normalization function, it outputs a pixel-level semantic probability map S. seg The value at each position represents the probability that the pixel belongs to each category. Finally, the category with the highest probability for each pixel is taken as its semantic label, thus completing the fine-grained division of active areas, obstacles, etc. The specific expression is as follows:

[0095] .

[0096] In the formula, W seg This represents the segmentation weight; the specific value can be set by those skilled in the art according to their actual needs. seg This represents the bias term, which applies to the score of each class for each pixel. Because the frequency of each class is often unbalanced in segmentation tasks (e.g., road pixels far outnumber pedestrian pixels), the bias term b... seg It can learn a positive bias for a minority of classes, thereby improving their prediction probability and alleviating the class imbalance problem; at the same time, it can also compensate for the systematic errors that may be introduced during the upsampling process.

[0097] The process of the segmentation detection head in the embodiment performing the segmentation task is as follows: based on the target detection, further distinguish different individuals under the same category (for example, distinguish "person A" and "person B" in the image); filter out the grabbable targets from the region of interest through the segmentation detection head in the embodiment and output the instance mask to distinguish independent object instances.

[0098] Specifically, this detection head typically first uses the results from the target detection head to extract features F for each region of interest in the global multimodal feature map F. roi Then, a small network (such as a fully convolutional network FCN) is used to predict an instance binary mask I for each pixel in the region of interest. ins This indicates whether a pixel belongs to the instance; this mask can be used to filter out graspable target objects (such as cups or tools), providing precise contour information for robotic arm operations. The specific expression is:

[0099] .

[0100] In the formula, W ins This represents the segmentation weight of the embodiment; the specific value can be set by those skilled in the art according to their actual needs. ins This represents the bias term, which directly determines the initial probability that a pixel within each region of interest belongs to a foreground instance. By adjusting the bias term b...ins The model can control the "activation threshold" of the instance binary mask. For example, in the early stage of training, the mask can be biased towards the entire background and gradually adjusted to the foreground output as training progresses, thus avoiding getting trapped in local optima.

[0101] For the key point detection head, the process of performing key point detection tasks is as follows: the key point detection head locates the object and captures the two-dimensional coordinate set of key points.

[0102] Specifically, this detection head is used to locate two-dimensional key points on an object that have specific functional significance (such as the center of a door handle or the gripping point of a cup); the detection head directly regresses the global multimodal feature map F and outputs a set of key point coordinates K through a fully connected layer. pt The specific expression is:

[0103] .

[0104] In the formula, W kpt This represents the keypoint detection weight; the specific value can be set according to the actual needs of those skilled in the art; b kpt This represents the bias term, which is directly added to the regression values ​​of the keypoint coordinates. Since keypoint coordinates are typically within a finite image range, the bias term b... kpt A global prior location can be learned (for example, for key points like "door handles," which are usually located in the lower part of the image), so that the model does not have to learn the absolute location from scratch, which significantly speeds up the convergence and improves the localization accuracy.

[0105] In this embodiment, after all the detection heads have completed their tasks in step S500, in order to provide a unified and concise interface to the robot's decision-making system, the system normalizes and fuses the output information of all the detection heads to construct a global environmental information matrix M. env This matrix is ​​used to provide unified environmental perception data for robot decision-making. It can be structured data, containing bounding box coordinates B... box Semantic probability graph S seg Instance binary mask I ins and the set of key point coordinates K pt Organized together. The specific expression is: M env =[ B box S seg I ins K pt ] T .

[0106] Another aspect of this application provides a computer-readable storage medium, in a preferred embodiment of which a computer program is stored on the storage medium; when the computer program is executed by a processor, it implements the above-described radar and vision-based 6D point cloud data fusion method.

[0107] Another aspect of this application provides an electronic device, in one preferred embodiment of which includes a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program to implement the above-described radar and vision-based 6D point cloud data fusion method.

[0108] The basic principles, main features, and advantages of this application have been described above. Those skilled in the art should understand that this application is not limited to the above embodiments. The embodiments and descriptions in the specification are merely the principles of this application. Various changes and modifications can be made to this application without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claims. The scope of protection claimed by this application is defined by the appended claims and their equivalents.

Claims

1. A 6D point cloud data fusion method based on radar and vision, characterized in that, Includes the following steps: S100: Perform distortion correction on the images acquired by the vision camera and establish the external parameter calibration relationship between the radar and the vision camera, so that the radar coordinate system and the vision coordinate system are initially aligned. S200: Based on the pitch angle of the visual camera, the original 3D point cloud acquired by the radar is rotated in the opposite direction of the pitch angle and projected onto a vertical 2D image plane to generate 2D projection points, while maintaining the mapping relationship of the point cloud before and after projection. S300: In the two-dimensional image plane, based on the principle of proximity, the two-dimensional projection points are matched with the pixels in the visual image, and the RGB color features of the pixels are extracted and fused with the original three-dimensional coordinates of the radar to construct 6D point cloud data. S400: The obtained 6D point cloud data is input into the UNet network architecture that integrates the Transformer model. Global context features and local detail features are extracted through the encoder-decoder structure to construct a global multimodal feature map. S500: The global multimodal feature map is input into multiple parallel detection heads to perform target detection, semantic segmentation, instance segmentation and key point detection tasks respectively. The perception information output by each detection head is normalized and fused to construct a global environment information matrix for the robot.

2. The 6D point cloud data fusion method based on radar and vision as described in claim 1, characterized in that, In step S200, the coordinates (X, X) of the original three-dimensional point cloud of the radar are... r Y r Z r ) and the coordinates (x) of the two-dimensional projection point p y p The relation is: ； In the formula, θ represents the pitch angle of the visual camera, and Z... proj This represents the depth information after projection.

3. The 6D point cloud data fusion method based on radar and vision as described in claim 1, characterized in that, In step S300, the principle of proximity is based on the minimum Euclidean distance criterion, so the two-dimensional projection point passes through coordinates (x... p y p The expression for matching the coordinates (u, v) of a pixel is: ； In the formula, (u * v * ) represents the coordinates of the pixel that meets the matching condition.

4. The 6D point cloud data fusion method based on radar and vision as described in claim 1, characterized in that, In step S500, the execution process of the object detection task is as follows: The object detection head performs object classification and bounding box regression, outputting the object category, confidence score, and spatial location, and determining the relative spatial relationship of objects; the specific expression is: ； In the formula, C cls σ represents the class confidence, W represents the sigmoid activation function. cls and W reg b represents the classification weight and regression weight, respectively. cls and b reg Both represent the corresponding bias terms, B box represents the bounding box coordinates, and F represents the global multimodal feature map.

5. The 6D point cloud data fusion method based on radar and vision as described in claim 1, characterized in that, In step S500, the semantic segmentation task is executed as follows: pixel-level semantic annotation is performed using the semantic segmentation detection head, and a probability map of each pixel belonging to different semantic categories is output; the specific expression is as follows: ； In the formula, S seg Represents a pixel-level semantic probability map, where Softmax(·) represents the normalized exponential function, and W seg The segmentation weights are represented by `upSample(·)`, which represents the upsampling operator used to restore the global multimodal feature map F to the original image resolution. seg This indicates the bias term.

6. The 6D point cloud data fusion method based on radar and vision as described in claim 1, characterized in that, In step S500, the execution process of the instance segmentation task is as follows: The instance segmentation detection head filters out grabbable targets from the region of interest and outputs an instance mask to distinguish independent object instances; the specific expression is: ； In the formula, I ins Let W represent the instance binary mask, σ represent the Sigmoid activation function, and W represent the sigmoid activation function. ins F represents the instance splitting weight. roi b represents the region of interest features extracted from the global multimodal feature map F. ins This indicates the bias term.

7. The 6D point cloud data fusion method based on radar and vision as described in claim 1, characterized in that, In step S500, the execution process of the key point detection task is as follows: the key point detection head locates the object and captures the two-dimensional coordinate set of key points; the specific expression is: ； In the formula, K pt W represents the set of key point coordinates. kpt Let b represent the keypoint detection weights, F represent the global multimodal feature map, and b represent the keypoint detection weights. kpt This indicates the bias term.

8. The 6D point cloud data fusion method based on radar and vision as described in claim 1, characterized in that, In step S100, the vision camera performs radial and tangential distortion correction on the image. The specific distortion correction formula is as follows: ； In the formula, (u corr v corr (u) represents the corrected pixel coordinates. raw v raw ) represents the original distorted pixel coordinates, k1 and k2 both represent radial distortion coefficients, p1 and p2 both represent tangential distortion coefficients, and r represents the Euclidean distance from the pixel to the image center.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program; when the computer program is executed by a processor, it implements the 6D point cloud data fusion method based on radar and vision as described in any one of claims 1-8.

10. An electronic device, characterized in that, It includes a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program to implement the 6D point cloud data fusion method based on radar and vision as described in any one of claims 1-8.

Images