Multi-modal data fusion method, system, equipment and medium

Abstract

The invention relates to a multi-modal data fusion method, system and device and a medium. The method comprises the following steps: acquiring synchronous RGB (Red, Green and Blue) images and laser radar point cloud data; geometric alignment is carried out on the point cloud data, and a mapping relation between the point cloud data and image pixels is established; respectively extracting two-dimensional visual features of the image and three-dimensional geometric features of the point cloud based on the mapping relation, and re-projecting the three-dimensional features to a two-dimensional space aligned with the visual features; and finally, dynamic weighted fusion is carried out on the two types of features through an adaptive attention mechanism, and a multi-modal fusion feature map is generated. By adopting the method, the fusion weight can be automatically adjusted according to the environment change, and the robustness and accuracy of a sensing system in a complex scene are effectively improved.

Description

Technical Field

[0001] This invention belongs to the field of data fusion technology, and in particular relates to a multimodal data fusion method, system, device and medium. Background Technology

[0002] With the rapid development of autonomous driving, mobile robots, and drone inspection, environmental perception technology, which relies on the collaborative work of multiple sensors (such as cameras and LiDAR), has become crucial for achieving highly reliable intelligent systems. Among these technologies, integrating the rich texture and color semantic information provided by visual images with the precise three-dimensional geometric information provided by LiDAR constitutes the core technical path for improving the robustness of perception systems in complex and dynamic scenarios.

[0003] In existing multimodal data fusion methods, a common approach is to first establish a rough spatial correspondence through sensor calibration, then use models such as convolutional neural networks to extract features from images and point clouds respectively, and finally fuse them through feature concatenation, addition, or weighted averaging based on fixed rules. These methods focus on designing general feature extraction networks and fixed fusion architectures to achieve complementarity of multi-source information.

[0004] However, those skilled in the art have found that the quality and reliability of both image and LiDAR data exhibit dramatic and unbalanced dynamic fluctuations due to real-time effects from factors such as changes in ambient lighting (e.g., backlighting, nighttime), weather conditions (e.g., rain and fog), and occlusion. For example, overexposure of camera images under strong light leads to loss of texture information, making point cloud data more reliable; while in foggy weather, the signal-to-noise ratio of LiDAR drops significantly, making visual information more critical. Existing fusion methods, which rely on fixed rules or simple weighting strategies, lack the ability to dynamically adjust fusion weights based on real-time data quality, making it difficult to "choose the best" when modal reliability changes abruptly. This results in unstable perception accuracy and robustness of existing fusion methods in complex real-world scenarios, limiting the safety and reliability of systems such as autonomous driving in edge scenarios. Summary of the Invention

[0005] Therefore, it is necessary to provide a multimodal data fusion method, system, device, and medium that can adapt to environmental changes and dynamically adjust fusion strategies to address the above-mentioned technical problems.

[0006] Firstly, this application provides a multimodal data fusion method, including:

[0007] S1. Acquire synchronized RGB image data and LiDAR point cloud data;

[0008] S2. Perform geometric alignment on the lidar point cloud data to establish the geometric mapping relationship between the three-dimensional points in the lidar point cloud data and the two-dimensional pixels in the RGB image data.

[0009] S3. Based on geometric mapping relationships, visual features are extracted from RGB image data to generate a two-dimensional visual semantic feature map;

[0010] S4. Based on geometric mapping relationships, extract three-dimensional features from lidar point cloud data to generate three-dimensional point cloud features;

[0011] S5. Based on geometric mapping relationships, perform two-dimensional reprojection on the three-dimensional point cloud features to generate a two-dimensional point cloud feature map aligned with the two-dimensional visual semantic feature map space.

[0012] S6. Adaptive attention fusion is performed on the two-dimensional visual semantic feature map and the two-dimensional point cloud feature map to generate a multimodal fusion feature map.

[0013] In one embodiment, S2 includes:

[0014] S21. Based on the extrinsic parameter matrix, the coordinate system transformation of the lidar point cloud data is performed to obtain the three-dimensional point coordinates in the camera coordinate system;

[0015] S22. Based on the pinhole imaging model, perform perspective projection on the three-dimensional point coordinates in the camera coordinate system to obtain the normalized imaging plane coordinates.

[0016] S23. Based on the camera intrinsic parameter matrix, perform pixel coordinate transformation on the normalized imaging plane coordinates to obtain two-dimensional pixel coordinates on the RGB image data;

[0017] S24. Based on depth constraints and image boundary constraints, the two-dimensional pixel coordinates are filtered to obtain the two-dimensional pixel coordinates that meet the validity conditions, and the geometric mapping relationship is generated by combining the corresponding three-dimensional point coordinates.

[0018] In one embodiment, S21 includes:

[0019] S211. Rotate the lidar point cloud data based on the rotation matrix to obtain the rotated three-dimensional point coordinates;

[0020] S212. Based on the translation vector, perform a translation transformation on the coordinates of the rotated 3D point to obtain the coordinates of the translated 3D point.

[0021] S213. Store the translated 3D point coordinates as 3D point coordinates in the camera coordinate system; coordinate system transformation is calculated using the following formula:

[0022]

[0023] in, The point coordinate vector in the camera coordinate system. The coordinate vector of a point in the lidar coordinate system. For rotation matrix, It is a translation vector.

[0024] In one embodiment, S23 includes:

[0025] S231. Scale the normalized imaging plane coordinates based on the focal length parameter to obtain the scaled image plane coordinates;

[0026] S232. Based on the principal point coordinate parameters, offset the scaled image plane coordinates to obtain the offset image plane coordinates;

[0027] S233. Store the offset image planar coordinates as two-dimensional pixel coordinates; the two-dimensional pixel coordinate transformation is calculated using the following formula:

[0028]

[0029] in,( , ) represents two-dimensional pixel coordinates, ( , () represents the normalized imaging plane coordinates. For focal length parameters, ( , () are the coordinate parameters of the main point.

[0030] In one embodiment, S6 includes:

[0031] S61. Perform channel concatenation on the two-dimensional visual semantic feature map and the two-dimensional point cloud feature map to obtain a concatenated feature map;

[0032] S62. Calculate the channel attention weights of the stitched feature map to obtain the channel attention weights of the image modality and the point cloud modality;

[0033] S63. Calculate the spatial attention weights of the stitched feature map to obtain the spatial attention weights of the image modality and the point cloud modality;

[0034] S64. Based on channel attention weight and spatial attention weight, the two-dimensional visual semantic feature map and the two-dimensional point cloud feature map are weighted and fused to generate a multimodal fusion feature map.

[0035] In one embodiment, S62 includes:

[0036] S621. Perform global average pooling on the spliced ​​feature map to obtain the global context feature vector;

[0037] S622. Process the global context feature vector with a fully connected layer to obtain the original channel scores;

[0038] S623. Normalize the original channel scores to obtain the channel attention weights; the normalization is calculated using the following formula:

[0039]

[0040] in, For the channel attention weights of the image modality, For the channel attention weights of the point cloud modality, and These are the original channel scores for the image modality and the point cloud modality, respectively.

[0041] In one embodiment, S64 includes:

[0042] S641. Multiply the spatial attention weights element-wise with the two-dimensional visual semantic feature map to obtain the spatially modulated visual features.

[0043] S642. Multiply the spatial attention weights element-wise with the two-dimensional point cloud feature map to obtain the spatially modulated point cloud features.

[0044] S643. Multiply the channel attention weights of the image modality by the spatially modulated visual features by a scalar to obtain the channel- and spatially weighted visual features.

[0045] S644. Multiply the channel attention weights of the point cloud mode by the spatially modulated point cloud features to obtain the channel- and spatially weighted point cloud features.

[0046] S645. Add the channel- and spatial dual-weighted visual features and the channel- and spatial dual-weighted point cloud features element by element to generate a multimodal fusion feature map.

[0047] Secondly, this application also provides a multimodal data fusion system, including:

[0048] The data acquisition module is used to acquire synchronized RGB image data and LiDAR point cloud data;

[0049] The geometric alignment module is used to perform geometric alignment on LiDAR point cloud data and establish a geometric mapping relationship between three-dimensional points in LiDAR point cloud data and two-dimensional pixels in RGB image data.

[0050] The two-dimensional visual feature extraction module is used to extract visual features from RGB image data based on geometric mapping relationships and generate two-dimensional visual semantic feature maps.

[0051] The 3D point cloud feature extraction module is used to extract 3D features from LiDAR point cloud data based on geometric mapping relationships, and generate 3D point cloud features.

[0052] The 2D reprojection module is used to perform 2D reprojection on 3D point cloud features based on geometric mapping relationships, generating a 2D point cloud feature map aligned with the 2D visual semantic feature map space.

[0053] The multimodal fusion module is used to perform adaptive attention fusion of two-dimensional visual semantic feature maps and two-dimensional point cloud feature maps to generate multimodal fused feature maps.

[0054] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement a multimodal data fusion method as described in the first aspect.

[0055] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a multimodal data fusion method as described in the first aspect.

[0056] The aforementioned multimodal data fusion method, system, device, and medium establish pixel-level mapping relationships through strict geometric alignment, ensuring spatial consistency between visual and geometric features. Furthermore, the introduced adaptive attention mechanism dynamically assesses the reliability of each modality based on environmental changes, enabling real-time adjustment of fusion weights. This allows the system to adaptively enhance the contribution of reliable modalities and suppress interference from failed modalities in complex scenarios such as sudden changes in lighting and severe weather, significantly improving the robustness and perception accuracy of multimodal fusion. This effectively solves the problem that traditional fixed fusion strategies cannot cope with dynamic environmental changes. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 A flowchart illustrating a multimodal data fusion method provided by the present invention;

[0059] Figure 2 A flowchart illustrating a method for generating multimodal fusion feature maps in an optional embodiment of the present invention;

[0060] Figure 3This is a schematic diagram of the structure of a multimodal data fusion system provided by the present invention. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0062] In one embodiment, such as Figure 1 As shown, a multimodal data fusion method is provided. This embodiment illustrates the method's application to a terminal. It is understood that this method can also be applied to a server, and further to a system including both a terminal and a server, and is implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0063] S1. Acquire synchronized RGB image data and LiDAR point cloud data.

[0064] Specifically, a timestamp matching mechanism is used to synchronously acquire RGB images and LiDAR point clouds. During operation, a high-precision timestamp is added to each frame of the RGB image and each frame of the LiDAR point cloud. Then, a timestamp alignment algorithm is used to filter data groups whose timestamp differences are within a preset threshold, ensuring that both correspond to the same spatiotemporal scene. The acquired RGB image data is converted into a three-dimensional tensor for storage. This tensor contains color channels, height, and width dimensions, representing the red, green, and blue color information and pixel distribution of the image, respectively. The LiDAR point cloud data is stored as a set of points, with each point containing three-dimensional spatial coordinates (x, y, z) and reflection intensity, where reflection intensity characterizes the object's reflection of laser light.

[0065] S2. Perform geometric alignment on the lidar point cloud data to establish the geometric mapping relationship between the three-dimensional points in the lidar point cloud data and the two-dimensional pixels in the RGB image data.

[0066] Optionally, the geometric alignment of the lidar point cloud is achieved using the extrinsic parameters obtained from sensor calibration, including the rotation matrix R and the translation vector t. This is first done using the coordinate system transformation formula: Perform coordinate transformation, where, This is the transformed coordinate vector in the camera coordinate system. Let be the coordinate vector of the point in the lidar coordinate system. Then, based on the pinhole imaging model, combined with camera intrinsic parameters (including focal length)... , and principal point coordinates , Perform perspective projection; the projection formula is: , ,in( , ) represents pixel coordinates, ( , , () represents the coordinates in the camera coordinate system. Through the above two transformations, a one-to-one geometric mapping relationship between three-dimensional points and two-dimensional pixels is established.

[0067] S3. Based on geometric mapping relationships, visual features are extracted from RGB image data to generate a two-dimensional visual semantic feature map.

[0068] Optionally, a pre-trained Convolutional Neural Network (CNN) such as the ResNet series can be used for visual feature extraction. The process involves first inputting an RGB image into the model. The convolutional layers in the model perform sliding window operations on image pixels using convolutional kernels of different sizes to extract low-level features such as edges and textures. Then, pooling layers downsample the feature maps output by the convolutional layers, reducing computation while preserving key features. Finally, fully connected layers or residual blocks containing ReLU activation functions are used to progressively aggregate low-level features to generate high-level two-dimensional visual semantic feature maps. Each feature unit in this feature map corresponds to a specific pixel region in the original image through a previously established geometric mapping, ensuring the correlation between semantic features and scene regions.

[0069] S4. Based on geometric mapping relationships, three-dimensional features are extracted from the lidar point cloud data to generate three-dimensional point cloud features.

[0070] Optionally, the PointNet++ model is used to extract 3D features from the LiDAR point cloud. First, the LiDAR point cloud data is preprocessed to remove outliers and reduce noise interference. Then, the model's hierarchical sampling module uses the Farthest Point Sampling (FPS) algorithm to select key sampling points in the point cloud. Next, the BallQuery algorithm is used to obtain a set of neighboring points within a certain range around each sampling point, and the coordinates of the neighboring points are centered relative to the sampling point to eliminate the influence of position offset. Then, the processed neighboring point features are input into a Multi-Layer Perceptron (MLP) for feature mapping, and the neighborhood features are aggregated by max pooling to generate local features for each sampling point. Finally, through this multi-layer feature aggregation process, 3D point cloud features containing global geometric information of the point cloud are generated, and each 3D point cloud feature is associated with a specific region of the RGB image through geometric mapping.

[0071] S5. Based on geometric mapping relationships, perform two-dimensional reprojection on the three-dimensional point cloud features to generate a two-dimensional point cloud feature map aligned with the two-dimensional visual semantic feature map space.

[0072] Optionally, the RGB image pixel coordinates corresponding to each 3D point in the LiDAR point cloud are determined based on the geometric mapping relationship. Then, according to the downsampling factor of the 2D visual semantic feature map, the feature unit coordinates corresponding to the pixel coordinates on the 2D visual semantic feature map are calculated. Specifically, the horizontal and vertical coordinates of the pixel coordinates are divided by the downsampling factor of the feature map relative to the original image to obtain the integer coordinates of the feature unit. Subsequently, the 3D point cloud features corresponding to each 3D point are assigned to the calculated feature unit positions. For feature units not covered by the point cloud features, zero-value padding is used. Finally, a 2D point cloud feature map is generated. The size of this feature map is completely consistent with the 2D visual semantic feature map, and the two are precisely aligned in space through geometric mapping.

[0073] S6. Adaptive attention fusion is performed on the two-dimensional visual semantic feature map and the two-dimensional point cloud feature map to generate a multimodal fusion feature map.

[0074] Optionally, the 2D visual semantic feature map and the 2D point cloud feature map are concatenated along the channel dimension to obtain a concatenated feature map. For the concatenated feature map, a channel attention mechanism is used to calculate the global weights of the two modal features: first, a global average pooling operation is performed on the concatenated feature map to compress the feature map of each channel into a single scalar, resulting in a global channel feature vector; then, this vector is input into a network consisting of two fully connected layers, and after nonlinear transformation by the ReLU activation function, two weight values ​​corresponding to the visual features and point cloud features are output, respectively. After normalization by the Softmax function, the channel attention weights are obtained. At the same time, a spatial attention mechanism is used to calculate the local weights: a convolution operation is performed on the concatenated feature map to extract local spatial features, and the feature values ​​are mapped to the (0,1) interval by the Sigmoid activation function to generate a spatial attention weight map. Finally, the 2D visual semantic feature map and the 2D point cloud feature map are multiplied element-wise with the corresponding channel weights and spatial weights, respectively, and the results are added together to obtain a multimodal fusion feature map.

[0075] In the aforementioned multimodal data fusion method, timestamp matching ensures the spatiotemporal consistency between RGB images and LiDAR point clouds. Sensor calibration parameters are then used to achieve geometric alignment between the point cloud and the image, establishing a mapping relationship. Subsequently, CNN and PointNet++ models are used to extract features from both modalities. Reprojection aligns the point cloud features with the visual feature space. Finally, an adaptive attention mechanism combining channel and spatial features completes feature fusion. This method addresses the problem of existing fixed-rule fusion methods being unable to adapt to changes in ambient lighting and fluctuating weather conditions, leading to uneven modal reliability. It dynamically adjusts the contribution of the two modalities based on real-time data quality, achieving "selective adoption," significantly improving the robustness of multimodal data fusion in complex and dynamic scenarios. This provides more accurate and reliable feature support for environmental perception in fields such as autonomous driving, mobile robots, and drone inspection, thereby ensuring the security and perception accuracy of intelligent systems in edge scenarios.

[0076] In one embodiment, S2 includes:

[0077] S21. Based on the extrinsic parameter matrix, the coordinate system of the lidar point cloud data is transformed to obtain the three-dimensional point coordinates in the camera coordinate system.

[0078] Optionally, the extrinsic parameter matrix consists of key parameters pre-obtained through sensor calibration. It comprises a 3×3 rotation matrix R and a 3×1 translation vector t. The former describes the attitude angle relationship between the LiDAR coordinate system and the camera coordinate system, while the latter characterizes the position offset of the origins of the two coordinate systems. During coordinate system transformation, the coordinates of each 3D point in the LiDAR point cloud data are used. As input, the matrix operation formula is used: The coordinates of the three-dimensional point in the camera coordinate system are calculated. The matrix multiplication operation ensures accurate conversion of rotation angles, while vector addition achieves translation correction of the origin position. This entire process can be applied in batches to all LiDAR points, unifying the overall coordinate system of the point cloud. This operation solves the coordinate system incompatibility problem caused by differences in the installation positions of LiDAR and cameras.

[0079] S22. Based on the pinhole imaging model, perform perspective projection on the three-dimensional point coordinates in the camera coordinate system to obtain the normalized imaging plane coordinates.

[0080] Optionally, the pinhole imaging model is a classic model simulating the optical imaging principle of a camera. Its core is to project a three-dimensional point in the camera coordinate system along the optical axis onto a normalized imaging plane perpendicular to the optical axis. This plane is 1 unit away from the camera's optical center and parallel to the image plane. When performing perspective projection, the coordinates of the three-dimensional point in the camera coordinate system are used... For input, via formula , The normalized imaging plane coordinates were calculated. , ).in, This represents the distance along the optical axis from a 3D point to the camera's optical center. By using division, the influence of distance on the projected coordinates is eliminated, allowing points at different depths to form 2D coordinates on the normalized plane that are consistent with the scale of the actual scene.

[0081] S23. Based on the camera intrinsic parameter matrix, perform pixel coordinate transformation on the normalized imaging plane coordinates to obtain two-dimensional pixel coordinates on the RGB image data.

[0082] Optionally, the camera intrinsic parameter matrix is ​​also obtained through sensor calibration; it is a 3×3 matrix in the form of... ,in , These are the effective focal lengths of the camera in the x-axis and y-axis directions, respectively, used to scale the normalized planar coordinates to the pixel scale; , These are the principal point coordinates, corresponding to the center pixel position of the image plane, used to correct for projection coordinate offsets. During pixel coordinate transformation, normalized imaging plane coordinates are used (…). , () is the input, expressed by the formula: , The two-dimensional pixel coordinates on the RGB image are calculated. , The multiplication operation performs coordinate scaling, the addition operation performs center offset correction, and the final output pixel coordinates directly correspond to the pixel positions in the RGB image.

[0083] S24. Based on depth constraints and image boundary constraints, the two-dimensional pixel coordinates are filtered to obtain the two-dimensional pixel coordinates that meet the validity conditions, and the geometric mapping relationship is generated by combining the corresponding three-dimensional point coordinates.

[0084] Optionally, the depth constraint is determined based on the three-dimensional points in the camera coordinate system. Values ​​need to be filtered out since the camera can only capture the scene in front. For 3D points with values ​​greater than 0, invalid points located behind the camera are removed; image boundary constraints are based on the inherent resolution of the RGB image, requiring the determination of pixel coordinates ( , The process involves checking whether the pixel coordinates of each 3D point simultaneously satisfy both constraints (0 ≤ u < image width and 0 ≤ v < image height) and excluding coordinates outside the image range. During filtering, each pixel coordinate that satisfies both conditions is checked, and its associated 3D point coordinates are retained. A one-to-one correspondence between 3D point coordinates and 2D pixel coordinates (geometric mapping) is then established using a list or dictionary. This effectively eliminates invalid data caused by environmental interference (such as false points due to rain or fog, or invalid points in overexposed areas caused by backlighting) or sensor errors, ensuring the reliability of the geometric mapping.

[0085] In the above embodiments, the point cloud coordinate system is unified to the camera coordinate system through an extrinsic parameter matrix. Then, the projection from 3D to a normalized 2D plane is completed using a pinhole imaging model. Next, the intrinsic parameter matrix is ​​used to convert it into image pixel coordinates. Finally, effective geometric mapping relationships are generated through dual constraint filtering. This embodiment focuses on accurately establishing cross-modal spatial correlations. Each step is based on sensor calibration parameters, which not only solves the hardware differences between LiDAR and camera, but also eliminates invalid data caused by environmental interference through constraint filtering. This makes up for the shortcomings of existing methods that only establish coarse spatial correspondences, effectively improving the correlation accuracy of multimodal data in complex dynamic scenes. This provides a more stable foundation for environmental perception in fields such as autonomous driving, and helps to solve the problem of insufficient robustness of existing fusion methods.

[0086] In one embodiment, S21 includes:

[0087] S211. Rotate the lidar point cloud data based on the rotation matrix to obtain the rotated three-dimensional point coordinates.

[0088] Optionally, the rotation matrix R is a 3×3 orthogonal matrix pre-obtained through the sensor calibration process. Its elements are calculated from attitude parameters such as pitch, roll, and yaw angles between the LiDAR and the camera. Its core function is to eliminate coordinate system rotation deviations caused by differences in the installation angles of the two sensors. During the rotation transformation, the coordinate vector of each three-dimensional point in the LiDAR point cloud data is used. (Containing three spatial components x, y, and z) as input, and performs operations with the rotation matrix R according to the linear algebra matrix multiplication rules, i.e. In the specific calculation process, each row element of R is respectively compared with... Multiply the three components and sum them to obtain the rotated 3D point coordinate vector. This operation ensures that the spatial attitude of the LiDAR point cloud is consistent with the camera coordinate system. For example, it corrects angular deviations caused by the forward-tilted installation of the LiDAR and the horizontal installation of the camera, thus avoiding geometric misalignment during subsequent coordinate transformations.

[0089] S212. Based on the translation vector, perform a translation transformation on the coordinates of the rotated three-dimensional point to obtain the coordinates of the translated three-dimensional point.

[0090] Optionally, the translation vector t is a 3×1 vector synchronously acquired during sensor calibration. Its three components correspond to the positional offset of the origin of the LiDAR coordinate system relative to the origin of the camera coordinate system in the x, y, and z directions, respectively. This is used to compensate for the spatial distance difference caused by the different installation positions of the two sensors, such as the LiDAR being installed on the roof of the vehicle and the camera being installed behind the windshield. When performing the translation transformation, the rotated 3D point coordinate vector is used. As input, it is operated on with the translation vector t according to the vector addition rule, that is... Vector addition follows the principle of adding corresponding components, that is... The x-component is The sum of the x-components of t and t, and the same applies to the y and z components. Choosing the operation order of rotation before translation is to avoid the offset of the coordinate system origin due to translation, which would cause the rotation center to deviate, thus ensuring the transformation accuracy. This further eliminates the positional difference between the two coordinate systems, making the point cloud data completely aligned with the camera coordinate system in space.

[0091] S213. Store the translated 3D point coordinates as 3D point coordinates in the camera coordinate system; coordinate system transformation is calculated using the following formula:

[0092]

[0093] in, The point coordinate vector in the camera coordinate system. The coordinate vector of a point in the lidar coordinate system. For rotation matrix, It is a translation vector.

[0094] Optionally, the translated 3D point coordinate vector Store directly as 3D point coordinates in the camera coordinate system The data is stored using the same data structure as the original point cloud data from the LiDAR, such as a two-dimensional array or tensor. Each row represents a three-dimensional point, and each column corresponds to the x, y, and z components, ensuring compatibility and ease of data retrieval in subsequent steps. The complete calculation process for coordinate system transformation is performed using the formula: Clearly defined, among which, This represents the final point coordinate vector in the camera coordinate system. Let be the point coordinate vector in the initial lidar coordinate system. and The rotation matrix and translation vector obtained from the calibration are respectively used to fully reproduce the transformation logic of "first rotating to correct the attitude, then translating to compensate for the position". The step-by-step operation is standardized and mathematical, ensuring the reproducibility and accuracy of the transformation process.

[0095] In the above embodiment, the LiDAR point cloud is rotated based on the rotation matrix obtained from sensor calibration to correct the attitude differences between the two sensors; then, the rotated point cloud is translated using a translation vector to compensate for the position offset of the coordinate system origin; finally, the translated coordinates are stored as point coordinates in the camera coordinate system. This embodiment solves the coordinate incompatibility problem caused by hardware installation differences between LiDAR and camera by processing attitude and position deviations step by step, ensuring the accuracy of coordinate system transformation, effectively making up for the shortcomings of existing methods that only establish a coarse spatial correspondence, and improving the accuracy and reliability of multimodal data association in complex dynamic scenes.

[0096] In one embodiment, S23 includes:

[0097] S231. Scale the normalized imaging plane coordinates based on the focal length parameter to obtain the scaled image plane coordinates.

[0098] Optionally, focal length parameter and These are core components of camera intrinsics, pre-acquired through camera calibration, and correspond to the camera's effective focal length along the x and y axes of the image. Physically, they are key coefficients for converting dimensionless coordinates on the normalized imaging plane to the image pixel scale. During scaling operations, the normalized imaging plane coordinates (…) are used… , The input is , and scaling is achieved through coordinate-by-coordinate multiplication. The specific calculation method is as follows: , ,in and These are the image planar coordinates scaled along the x and y axes, respectively. Essentially, this maps optically normalized coordinates to the pixel dimensions of the digital image, ensuring that the projected positions of 3D points on the image match the actual pixel distribution and avoiding coordinate deviations caused by scale inconsistencies.

[0099] S232. Offset the scaled image plane coordinates based on the principal point coordinate parameters to obtain the offset image plane coordinates.

[0100] Optionally, principal point coordinates and Similarly obtained through camera calibration, these coordinates represent the pixel coordinates of the camera's optical center on the image plane, typically close to the center of the image resolution. Their function is to correct the positional deviation between the normalized image plane origin and the image pixel origin. The normalized image plane origin is located on the camera's optical axis, while the image pixel origin is usually at the upper left corner of the image; there is an inherent offset between the two. During offset processing, the scaled image plane coordinates (…) are used. , Using as input, offset correction is achieved through coordinate-by-coordinate addition. The specific calculation method is as follows: , ,in and These are the offset image plane coordinates. This process precisely compensates for the positional difference between the optical origin and the pixel origin, ensuring that the coordinates of the 3D point projected onto the image exactly correspond to the actual pixel positions, avoiding projection misalignment caused by origin offset, and further improving the accuracy of cross-modal spatial association.

[0101] S233. Store the offset image planar coordinates as two-dimensional pixel coordinates; the two-dimensional pixel coordinate transformation is calculated using the following formula:

[0102]

[0103] in,( , ) represents two-dimensional pixel coordinates, ( , () represents the normalized imaging plane coordinates. For focal length parameters, ( , () are the coordinate parameters of the main point.

[0104] Optionally, when performing the storage operation, the offset image plane coordinates are directly transferred ( , The coordinates are stored as two-dimensional pixel coordinates, and the storage format can be an array or a list, ensuring that each coordinate is associated with a corresponding 3D point in the camera coordinate system. The matrix formula for transforming two-dimensional pixel coordinates is essentially a linear transformation under homogeneous coordinates; the left side of the formula represents the homogeneous form of the two-dimensional pixel coordinates. , [1]^T, the right side represents the 3×3 camera intrinsic parameter matrix and the homogeneous normalized imaging plane coordinates. , The product of ,1]^T. In the intrinsic parameter matrix, and Located on the main diagonal, used to implement scaling; and Located in the first two rows of the third column of the matrix, this element is used for offsetting; the remaining off-diagonal zero elements ensure that coordinate transformations only act on the corresponding axes, without cross-interference. Through this matrix operation, all normalized coordinates can be converted to pixel coordinates in batches. The operation process strictly follows the principles of camera optical imaging, ensuring conversion accuracy and providing a precise spatial mapping basis for fusing visual and LiDAR information.

[0105] In the above embodiments, the normalized imaging plane coordinates are scaled to the pixel scale based on the focal length parameter, solving the scale mismatch problem between optical coordinates and pixel coordinates. Then, the scaled coordinates are offset-corrected using principal point coordinates, eliminating the positional deviation between the optical origin and the pixel origin. Finally, matrix operations are used to batch convert and store the two-dimensional pixel coordinates, achieving a precise association between the three-dimensional points in the camera coordinate system and the pixels of the RGB image. This embodiment uses pre-calibrated camera intrinsic parameters as its core, optimizing each step for key error sources in cross-modal spatial mapping. This effectively avoids mapping distortion caused by scale and origin offsets, ultimately establishing a reliable geometric correspondence. This significantly improves the robustness of multimodal data fusion in complex scenes and compensates for the deficiency of "coarse spatial correspondence."

[0106] In an optional embodiment, such as Figure 2 As shown, S6 includes:

[0107] S61. Perform channel concatenation on the two-dimensional visual semantic feature map and the two-dimensional point cloud feature map to obtain the concatenated feature map.

[0108] Optionally, both the two-dimensional visual semantic feature map and the two-dimensional point cloud feature map are in the form of a three-dimensional tensor structure, with the dimensional form uniformly H×W×C, where H represents the feature map height, W represents the width, and C represents the number of channels. The number of channels in the visual feature map is denoted as H×W×C. The number of channels in a point cloud feature map is denoted as , which carries semantic information such as texture and color. It is used to carry information such as geometric contours and spatial positions. During channel stitching, the two feature maps are integrated along the channel dimension (i.e., the third dimension) of the feature map, resulting in a feature map with dimensions H×W×(…). + In practice, it is necessary to ensure that the H and W of the two feature maps are completely consistent. This premise is guaranteed by the point cloud feature reprojection based on the geometric mapping relationship, so as to avoid splicing misalignment due to spatial size mismatch.

[0109] S62. Calculate the channel attention weights of the stitched feature map to obtain the channel attention weights of the image modality and the point cloud modality.

[0110] Optionally, a global average pooling operation is first performed on the stitched feature map. For each channel of the stitched feature map, the average value of all pixel values ​​in the H×W space is calculated, compressing the two-dimensional feature of each channel into a 1×1 scalar, resulting in a 1×1×( + The global response vector of each channel is calculated. This vector reflects the overall feature contribution of each channel. For example, in a strong light scene, the global response value of the image modal channel will be significantly lower than that of the point cloud modal channel. This vector is then input into a network consisting of two fully connected layers. The first fully connected layer compresses the channel dimension to a lower dimension (e.g., ...) through a linear transformation. + The second fully connected layer restores the dimension to 2, corresponding to the image modality and the point cloud modality respectively, and outputs two raw scores. Finally, the raw scores are normalized by the Softmax function to obtain two channel attention weights with values ​​between 0 and 1 and a sum of 1, denoted as α (image modality weight) and β (point cloud modality weight).

[0111] S63. Calculate the spatial attention weights of the stitched feature map to obtain the spatial attention weights of the image modality and the point cloud modality.

[0112] Optionally, using the stitched feature map as input, a 3×3 convolution kernel is first applied to it for convolution. During convolution, a zero-padding strategy is used to ensure that the H and W dimensions of the output feature map remain unchanged. This convolution operation can capture the local spatial context information of the feature map, such as identifying regions with clear textures or regions with obvious geometric features in the point cloud. After convolution, a single-channel feature map of dimension H×W×1 is obtained. Then, a Sigmoid activation function is used to map each pixel value in the feature map to the 0-1 interval, generating a spatial attention weight map of dimension H×W×1. Each element in this spatial attention weight map represents the feature importance of the corresponding spatial location. Further, this weight map is copied into two weight maps of the same dimension, serving as the spatial attention weight maps for the image modality and the point cloud modality, respectively denoted as . and And satisfy + =1, achieved through normalization. The above design can dynamically allocate the contribution of the two modalities for different spatial locations, such as in foggy scenes, where the contribution of the two modalities is adjusted for image edge regions. It will be higher than The clear area in the center of the image It will be higher than This solves the problem of fixed fusion strategies failing due to differences in local scenes.

[0113] S64. Based on channel attention weight and spatial attention weight, the two-dimensional visual semantic feature map and the two-dimensional point cloud feature map are weighted and fused to generate a multimodal fusion feature map.

[0114] Optionally, the two-dimensional visual semantic feature map is first multiplied element-wise with the corresponding channel attention weight α to enhance the feature contribution of important channels in the image modality and weaken the interference of unimportant channels; then the result is multiplied with the spatial attention weight map of the image modality. Element-wise multiplication is performed to further enhance spatially important features in the image modality, resulting in a visual feature map with both spatial and channel weights. Using the same logic, the 2D point cloud feature map is then compared with the channel attention weight β and the point cloud modality spatial attention weight map. Element-wise multiplication is performed sequentially to obtain the double-weighted point cloud feature map. Finally, the two double-weighted feature maps are added element-wise to obtain a feature map with dimensions H×W×. (or H×W×) The fusion of feature maps depends on whether the number of channels in the two modalities is consistent. If they are inconsistent, a 1×1 convolution can be used to unify the dimensions. This fusion process dynamically adjusts the contribution of the two modalities in the channel and spatial dimensions through dual weights. For example, in a backlit scene, the α and α of the image modalities are adjusted. Both decrease, and the β of the point cloud mode decreases. All are increased to ensure that the fusion features preferentially retain reliable modal information.

[0115] In the above embodiment, the dual-modal features of the image and point cloud are first integrated by channel stitching to provide a complete information foundation for attention calculation. Then, the global importance of the two modalities is learned through channel attention weights to adapt to global environmental changes such as lighting and weather. Next, the modal importance of local locations is learned through spatial attention weights to match local scene differences. Finally, a multimodal feature map is generated through dual-weighted fusion. This embodiment replaces the traditional fixed rules with a data-driven attention mechanism, effectively solving the defect of not being able to dynamically adjust the fusion weights. It can flexibly allocate the contribution of the two modalities according to the real-time modal reliability (such as unreliable vision under strong light and unreliable point clouds under fog), significantly improving the robustness and discriminativeness of the multimodal fusion feature map in complex and dynamic scenes. This provides more accurate and reliable feature support for environmental perception in fields such as autonomous driving and drone inspection, ensuring the security and perception accuracy of intelligent systems in edge scenarios.

[0116] In one embodiment, S62 includes:

[0117] S621. Perform global average pooling on the concatenated feature map to obtain the global context feature vector.

[0118] Optionally, the spliced ​​feature map is H×W×( + The three-dimensional tensor structure of the feature map, where H is the height of the feature map and W is the width. and These represent the number of channels in the 2D visual semantic feature map and the 2D point cloud feature map, respectively, carrying semantic and geometric information from two modalities. During global average pooling, for each channel of the concatenated feature map, all H×W pixels within that channel are traversed, and the arithmetic mean of all pixel values ​​is calculated. This compresses the 2D feature of each channel into a single scalar value, ultimately resulting in a 1×1×( + The global context feature vector is obtained by eliminating local fluctuations in spatial dimensions and preserving the global feature contribution of each channel. For example, in a scene where strong light causes image overexposure, the average pixel value of the image modal channel will be significantly lower than that of the point cloud modal channel, so that the vector can objectively reflect the global reliability difference between the two modalities.

[0119] S622. Process the global context feature vector with a fully connected layer to obtain the original channel scores.

[0120] Optionally, the global context feature vector is first input into the first fully connected layer, which is transformed by the following linear transformation formula: The vector dimension is compressed, where X is the global context feature vector. Here is the weight matrix of the first fully connected layer, with dimensions: [( + ) / k]×( + ), where k is the compression coefficient, usually taken as 4 to balance computational cost and feature representation capability. As the bias vector, output low-dimensional features Subsequently, on A ReLU activation function is introduced to filter out negative feature interference through the operation f(x) = max(0,x), thereby enhancing the nonlinear representation of effective features. The activated features are then input into the second fully connected layer, and a linear transformation is performed using the formula: Restore the dimension to 2, where The second layer weight matrix (dimension 2×[( + ) / k]), As the bias vector, the final output is the raw channel fraction of the image modality. Original channel fractions of point cloud modes The parameters of the two fully connected layers are optimized through model training. The training objective is to improve the perception accuracy after multimodal fusion, so that the output score can dynamically match scene changes, replace the score set by traditional fixed rules, and solve the defect that the fixed fusion architecture cannot adapt to scene differences.

[0121] S623. Normalize the original channel scores to obtain the channel attention weights; the normalization is calculated using the following formula:

[0122]

[0123] in, For the channel attention weights of the image modality, For the channel attention weights of the point cloud modality, and These are the original channel scores for the image modality and the point cloud modality, respectively.

[0124] Optionally, the Softmax function can be used to... and Normalization is performed according to a given formula, where e is the natural constant, the numerators are the exponents of the original fractions for the two modes, and the denominator is the sum of the exponents of the original fractions for the two modes. The result after the calculation is... and The values ​​are all between 0 and 1, and satisfy the following conditions: + =1, ensuring the rationality and exclusivity of weight allocation. For example, in foggy weather, the decrease in the signal-to-noise ratio of lidar leads to... Below After substituting into the formula It will be significantly higher than This allows the fusion process to focus on clear information from image modalities; however, in nighttime scenes, Lowering will The elevation emphasizes the geometric information of the cloud. The above normalization method dynamically adjusts the channel weights through data-driven approaches, rather than relying on preset rules, thus avoiding the adaptation failure of fixed weights when modal reliability changes abruptly.

[0125] In the above embodiment, global channel information of the stitched feature map is first extracted through global average pooling to objectively reflect the global reliability differences between the two modalities. Then, two fully connected layers are used to learn and adapt the original channel scores to different scenarios, breaking away from the limitations of traditional fixed rules. Finally, Softmax normalization is used to obtain a reasonably proportioned channel attention weight, realizing the dynamic allocation of the channel contributions of the two modalities. This embodiment replaces empirical fixed strategies with data-driven approaches, accurately responding to dynamic fluctuations in modal reliability caused by changes in ambient lighting and weather, effectively solving the deficiency of existing fusion methods in lacking the ability to dynamically adjust weights. It significantly improves the adaptability and robustness of multimodal data fusion at the channel level in complex and dynamic scenarios, ensuring the discriminability and stability of the final multimodal fused feature map.

[0126] In one embodiment, S64 includes:

[0127] S641. Multiply the spatial attention weights element-wise with the two-dimensional visual semantic feature map to obtain the spatially modulated visual features.

[0128] Optionally, the spatial attention weights exist in the form of a single-channel feature map of size H×W×1, where H is the feature map height and W is the width, and each element takes a value between 0 and 1, representing the feature importance of the corresponding spatial location; the two-dimensional visual semantic feature map is H×W× The three-dimensional tensor This represents the number of visual feature channels, carrying semantic information such as texture and color. During element-wise multiplication, the weight value at each position in the spatial attention weight map is multiplied by the feature values ​​of all channels at the corresponding position in the visual feature map. That is, for an element at coordinates (h, w, c) in the visual feature map... The result after multiplication is ,in This represents the weight value of the spatial attention weight map at position (h,w). The above process can specifically strengthen spatially reliable areas in visual features, such as areas with suitable lighting and clear textures, while weakening unreliable areas, such as backlit and overexposed areas.

[0129] S642. Multiply the spatial attention weights element-wise with the two-dimensional point cloud feature map to obtain the spatially modulated point cloud features.

[0130] Optionally, the dimensions of the spatial attention weight map and the two-dimensional point cloud feature map are H×W×1 and H×W×1, respectively. , The number of point cloud feature channels carries geometric contour and spatial location information. The H and W dimensions of both are guaranteed to be completely consistent by the point cloud reprojection based on geometric mapping. When performing element-wise multiplication, the same operational logic as for visual features is followed: the weight value at each (h,w) position in the spatial attention weight map is multiplied one by one with the feature values ​​of all channels at the corresponding position in the point cloud feature map to obtain the spatially modulated point cloud feature, whose dimensions remain H×W×. The process remains unchanged. It can highlight spatially reliable regions in point cloud features, such as unobstructed regions with clear geometric structures, and suppress unreliable regions, such as feature regions corresponding to noise points caused by rain and fog. For example, in a foggy scene, the weight value of the vehicle outline region in the point cloud feature map is higher, and the geometric features of this region are further enhanced after multiplication.

[0131] S643. Multiply the channel attention weights of the image modality by the spatially modulated visual features to obtain the channel- and spatially weighted visual features.

[0132] Optionally, the channel attention weights for the image modality are single scalar values. The value ranges from 0 to 1, calculated from the previous channel attention, and reflects the global importance of the visual modality. The spatially modulated visual features are H×W× The three-dimensional tensor, here The number of visual feature channels must be consistent with that of the point cloud feature channel. If the number of visual and point cloud feature channels were different beforehand, the number of visual feature channels can be adjusted in advance using a 1×1 convolution. (To ensure subsequent addition dimensions match). When performing scalar multiplication, All elements acting on the spatially modulated visual feature, i.e., elements at each (h, w, c) position in the feature. The result after multiplication is = × Adjusting the contribution of image modalities at a global level, for example, in bright light scenes, Smaller values ​​can reduce the overall weight of visual features, avoiding the negative impact of semantic information distortion caused by overexposure on the fusion result.

[0133] S644. Multiply the channel attention weights of the point cloud mode by the spatially modulated point cloud features to obtain the channel- and spatially weighted point cloud features.

[0134] Optionally, the channel attention weights of the point cloud modal are single scalar values. Values ​​range from 0 to 1, and The sum of these values ​​equals 1, reflecting the global importance of the point cloud mode. The spatially modulated point cloud feature dimensions are H×W× When performing scalar multiplication, the operational logic is completely consistent with that of scalar multiplication of visual features. Applying this to all elements of the spatially modulated point cloud features yields a point cloud feature with both channel and spatial weights, still maintaining the dimensions H×W×. ,Right now = × , This refers to the spatially modulated point cloud features. The overall contribution of point cloud modes can be strengthened or weakened at a global level; for example, in foggy scenarios where the signal-to-noise ratio of the lidar decreases only slightly. A larger value can increase the overall weight of point cloud features, ensuring that their geometric information plays a dominant role in the fusion process.

[0135] S645. Add the channel- and spatial dual-weighted visual features and the channel- and spatial dual-weighted point cloud features element by element to generate a multimodal fusion feature map.

[0136] Optionally, the visual features and point cloud features after dual weighting of channel and spatial dimensions are both unified to H×W× This satisfies the dimension matching requirement for element-wise addition. When performing element-wise addition, the two features must be on the same coordinate system. The element values ​​at each position are directly summed, i.e., the features are fused. The final generated dimension is H×W× Multimodal fusion feature maps integrate reliable local semantic information from image modalities with reliable local geometric information from point cloud modalities under the control of global channel weights. For example, in complex scenes, the fused feature map retains both the texture features of clear traffic signs in the image modalities and the precise three-dimensional contour features of vehicles in the point cloud modalities.

[0137] In the above embodiment, spatial local modulation of visual and point cloud features is first performed by element-wise multiplication to strengthen the reliable regions of each modality and suppress invalid regions. Then, scalar multiplication combined with channel attention weights is used to adjust the overall contribution of the two modalities at the global level to adapt to global changes in modal reliability under different environments. Finally, element-wise addition is used to achieve organic fusion of dual-weighted features, ensuring that the fusion result retains the effective information of both modalities. This embodiment replaces traditional fixed rules with dynamic weights, enabling real-time response to modal reliability fluctuations caused by factors such as ambient lighting, weather conditions, and occlusion. It effectively solves the problem of unstable perception accuracy and robustness of existing fusion methods in complex scenarios, ultimately providing more accurate and reliable multimodal fusion feature support for environmental perception in fields such as autonomous driving and mobile robots, and significantly improving the security and reliability of intelligent systems in edge scenarios.

[0138] In the aforementioned multimodal data fusion method, system, device, and medium, RGB images and LiDAR point cloud data are first acquired synchronously through timestamp matching to ensure that they correspond to the same spatiotemporal scene. Then, the point cloud is transformed to the camera coordinate system based on the sensor-calibrated extrinsic matrix. Perspective projection and pixel coordinate transformation are completed by combining the pinhole imaging model and the camera intrinsic matrix. Valid coordinates are selected through depth and image boundary constraints to establish a precise geometric mapping relationship. At the same time, the point cloud features are reprojected to the image space to achieve feature alignment. Subsequently, visual semantic features and point cloud 3D features are extracted respectively. Global modal weights are calculated through channel attention, and local position weights are calculated through spatial attention. The two features are subjected to dual processing of spatial modulation and channel weighting. Finally, the elements are added one by one to obtain the multimodal fusion feature map. Existing fixed-rule or simple weighted fusion methods cannot adapt to changes in ambient lighting, fluctuating weather conditions, and severe imbalances in modal reliability caused by occlusion, resulting in unstable perception accuracy and robustness. The above-mentioned technical solution, through a data-driven dynamic attention mechanism, can adjust the fusion weights in real time according to the global importance and local reliability of the two modalities, achieving intelligent fusion that "chooses the best" and effectively solves the defects of existing methods. It significantly improves the robustness and perception accuracy of multimodal data fusion in complex dynamic scenarios, providing key technical support for environmental perception in high-reliability intelligent systems in fields such as autonomous driving, mobile robots, and drone inspection.

[0139] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0140] Based on the same inventive concept, this application also provides a system for implementing the multimodal data fusion method described above. The solution provided by this system is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more multimodal data fusion system embodiments provided below can be found in the limitations of the multimodal data fusion method described above, and will not be repeated here.

[0141] In one exemplary embodiment, such as Figure 3 As shown, a multimodal data fusion system 10 is provided, comprising:

[0142] Data acquisition module 11 is used to acquire synchronized RGB image data and LiDAR point cloud data;

[0143] The geometric alignment module 12 is used to perform geometric alignment on the lidar point cloud data and establish a geometric mapping relationship between the three-dimensional points in the lidar point cloud data and the two-dimensional pixels in the RGB image data.

[0144] The two-dimensional visual feature extraction module 13 is used to extract visual features from RGB image data based on geometric mapping relationships and generate a two-dimensional visual semantic feature map.

[0145] The 3D point cloud feature extraction module 14 is used to extract 3D features from lidar point cloud data based on geometric mapping relationships and generate 3D point cloud features.

[0146] The two-dimensional reprojection module 15 is used to perform two-dimensional reprojection on the three-dimensional point cloud features based on geometric mapping relationships, and generate a two-dimensional point cloud feature map aligned with the two-dimensional visual semantic feature map space.

[0147] The multimodal fusion module 16 is used to perform adaptive attention fusion of the two-dimensional visual semantic feature map and the two-dimensional point cloud feature map to generate a multimodal fusion feature map.

[0148] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the multimodal data fusion method as described above.

[0149] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the multimodal data fusion method as described above.

[0150] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0151] The above-described embodiments are merely illustrative of several implementation methods of the embodiments of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this application, and these modifications and improvements all fall within the protection scope of the embodiments of this application.

Claims

1. A multimodal data fusion method, characterized in that, The method includes: S1. Acquire synchronized RGB image data and LiDAR point cloud data; S2. Perform geometric alignment on the lidar point cloud data to establish a geometric mapping relationship between the three-dimensional points in the lidar point cloud data and the two-dimensional pixels in the RGB image data. S3. Based on the geometric mapping relationship, visual features are extracted from the RGB image data to generate a two-dimensional visual semantic feature map; S4. Based on the geometric mapping relationship, perform three-dimensional feature extraction on the lidar point cloud data to generate three-dimensional point cloud features; S5. Based on the geometric mapping relationship, perform two-dimensional reprojection on the three-dimensional point cloud features to generate a two-dimensional point cloud feature map aligned with the two-dimensional visual semantic feature map space. S6. Perform adaptive attention fusion on the two-dimensional visual semantic feature map and the two-dimensional point cloud feature map to generate a multimodal fusion feature map.

2. The method according to claim 1, characterized in that, S2 includes: S21. Based on the external parameter matrix, perform coordinate system transformation on the lidar point cloud data to obtain the three-dimensional point coordinates in the camera coordinate system; S22. Based on the pinhole imaging model, perform perspective projection on the three-dimensional point coordinates in the camera coordinate system to obtain the normalized imaging plane coordinates. S23. Based on the camera intrinsic parameter matrix, perform pixel coordinate transformation on the normalized imaging plane coordinates to obtain two-dimensional pixel coordinates on the RGB image data; S24. Based on depth constraints and image boundary constraints, the two-dimensional pixel coordinates are filtered to obtain two-dimensional pixel coordinates that meet the validity conditions, and the geometric mapping relationship is generated by combining the corresponding three-dimensional point coordinates.

3. The method according to claim 2, characterized in that, S21 includes: S211. Rotate the lidar point cloud data based on the rotation matrix to obtain the rotated three-dimensional point coordinates; S212. Based on the translation vector, perform a translation transformation on the rotated three-dimensional point coordinates to obtain the translated three-dimensional point coordinates; S213. Store the translated 3D point coordinates as 3D point coordinates in the camera coordinate system; the coordinate system transformation is calculated using the following formula: in, The point coordinate vector in the camera coordinate system. This is the point coordinate vector in the lidar coordinate system. For rotation matrix, It is a translation vector.

4. The method according to claim 2, characterized in that, S23 includes: S231. Scale the normalized imaging plane coordinates based on the focal length parameter to obtain the scaled image plane coordinates; S232. Based on the principal point coordinate parameters, the scaled image plane coordinates are offset to obtain the offset image plane coordinates; S233. Store the offset image plane coordinates as the two-dimensional pixel coordinates; wherein, the two-dimensional pixel coordinate transformation is calculated using the following formula: in,( , ) represents two-dimensional pixel coordinates, ( , () represents the normalized imaging plane coordinates. For focal length parameters, ( , () are the coordinate parameters of the main point.

5. The method according to claim 1, characterized in that, S6 includes: S61. Perform channel concatenation on the two-dimensional visual semantic feature map and the two-dimensional point cloud feature map to obtain a concatenated feature map; S62. Calculate the channel attention weights of the stitched feature map to obtain the channel attention weights of the image modality and the point cloud modality; S63. Calculate the spatial attention weights of the stitched feature map to obtain the spatial attention weights of the image modality and the point cloud modality; S64. Based on the channel attention weight and spatial attention weight, the two-dimensional visual semantic feature map and the two-dimensional point cloud feature map are weighted and fused to generate the multimodal fusion feature map.

6. The method according to claim 5, characterized in that, S62 includes: S621. Perform global average pooling on the spliced ​​feature map to obtain a global context feature vector; S622. Perform fully connected layer processing on the global context feature vector to obtain the original channel scores; S623. Normalize the original channel scores to obtain the channel attention weights; the normalization is calculated using the following formula: in, For the channel attention weights of the image modality, For the channel attention weights of the point cloud modality, and These are the original channel scores for the image modality and the point cloud modality, respectively.

7. The method according to claim 6, characterized in that, S64 includes: S641. Multiply the spatial attention weights element-wise with the two-dimensional visual semantic feature map to obtain spatially modulated visual features. S642. Multiply the spatial attention weights element-wise with the two-dimensional point cloud feature map to obtain spatially modulated point cloud features. S643. Multiply the channel attention weights of the image modality by the spatially modulated visual features to obtain the channel- and spatially weighted visual features. S644. Multiply the channel attention weights of the point cloud mode by the spatially modulated point cloud features to obtain channel- and spatially weighted point cloud features. S645. The visual features weighted by both channel and space are added element-wise to the point cloud features weighted by both channel and space to generate the multimodal fusion feature map.

8. A multimodal data fusion system, characterized in that, The system includes: The data acquisition module is used to acquire synchronized RGB image data and LiDAR point cloud data; The geometric alignment module is used to perform geometric alignment on the lidar point cloud data and establish a geometric mapping relationship between the three-dimensional points in the lidar point cloud data and the two-dimensional pixels in the RGB image data. A two-dimensional visual feature extraction module is used to extract visual features from the RGB image data based on the geometric mapping relationship, and generate a two-dimensional visual semantic feature map; The three-dimensional point cloud feature extraction module is used to extract three-dimensional features from the lidar point cloud data based on the geometric mapping relationship, and generate three-dimensional point cloud features. A two-dimensional reprojection module is used to perform two-dimensional reprojection on the three-dimensional point cloud features based on the geometric mapping relationship, and generate a two-dimensional point cloud feature map aligned with the two-dimensional visual semantic feature map space. The multimodal fusion module is used to perform adaptive attention fusion on the two-dimensional visual semantic feature map and the two-dimensional point cloud feature map to generate a multimodal fusion feature map.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images