Method, system and device for dynamic fusion of multimodal perception data of embodied intelligent body

Abstract

The present invention discloses a method, system, and device for the dynamic fusion of multimodal perception data of an embodied intelligent body, including: S1. The embodied intelligent body collects image data in real time during motion; S2. Uses a multimodal large model to determine the point cloud filling density of the image data, and uses a mapping algorithm to fill the 3D point cloud of the camera image during data collection; S3. Generates a lidar feature map through point cloud data projection conversion; S4. Initializes and aligns the feature data of the multimodal large model; S5. Constructs a set of multi-scale, multi-channel, differentiable mutual information fine optimization methods to process the feature data; S6. Performs online monitoring and fusion self-correction operations of the multimodal large model; S7. Projects camera information onto the laser 3D point cloud, and continuously constructs a fused RGB-3D scene during the motion of the embodied intelligent body. The present invention can achieve accurate, robust, automatic calibration and fusion of external parameters between the lidar and the camera.

Description

Technical Field

[0001] The present invention relates to the field of data information processing technology, and in particular to a method, system and device for dynamically fusing multimodal perception data of an embodied intelligent body based on a multimodal large model. Background Art

[0002] With the widespread application of embodied intelligent bodies in service robots, autonomous driving, patrol and security, etc., the precise fusion of perception systems has become a key link in achieving high-level environmental understanding and autonomous behavioral decision-making. Among them, lidar and cameras, as the most commonly used heterogeneous sensor combinations, provide precise spatial structure and rich visual semantic information respectively. The accuracy of the external parameter calibration between the two directly affects the robustness and accuracy of core tasks such as map construction, target detection, and multimodal perception fusion.

[0003] Traditional lidar-camera extrinsic parameter calibration methods mostly rely on artificially constructed calibration plates or explicit geometric priors, such as special structures such as checkerboards, cubes, and cylinders. The extrinsic parameters are estimated by observing the projection relationship of the calibration plate in the field of view of the two sensors. Although such methods have high theoretical accuracy, they have significant limitations in actual deployment. First, they have strict requirements on environmental conditions, such as good lighting and unobstructed field of view. Second, the calibration operation requires the participation of professionals, the process is complex and difficult to promote in large-scale equipment deployment scenarios. In addition, the calibration plate must be located within the common field of view of the lidar and camera, which is often difficult to meet in the complex action scenarios of embodied intelligent bodies.

[0004] In recent years, with the development of deep learning, a large number of neural network-based calibration methods have emerged. For example, convolutional neural networks (CNNs) or Transformer structures are used to directly regress extrinsic parameters from image and point cloud data or estimate the correspondence between modalities. These methods can learn nonlinear mapping relationships through end-to-end training, and have certain noise resistance and strong generalization capabilities. However, such methods usually rely on a large amount of labeled training data, and have problems such as poor transferability, unexplainable network structure, and strong dependence on training scenarios. They are difficult to adapt to the long-term operation requirements of embodied intelligent bodies in heterogeneous scenarios and dynamic changes. In addition, most of these methods cannot be optimized online and have difficulty dealing with hardware error drift and time-accumulated errors.

[0005] At the same time, the rapid development of large language models and multimodal large models has provided a new paradigm for intelligent perception and reasoning tasks. Multimodal large models have the ability to align cross-modal information, understand data-driven patterns, and perform interactive task planning, which makes it possible to achieve higher-level automated calibration and fusion. For example, multimodal large models can combine inputs such as images and point clouds to autonomously generate data processing code, evaluation indicator analysis reports, and parameter adjustment suggestions, lowering the R&D threshold; at the same time, they can also serve as a "strategy scheduler" to fuse low-level mutual information estimation with high-level semantic alignment, thereby realizing a truly "self-supervised, adaptive, and self-repairing" sensor calibration and fusion mechanism in the embodied system. However, there is currently no working system in the existing technology that can introduce the understanding capabilities of multimodal large models into lidar-camera calibration and fusion to overcome the limitations of previous methods.

[0006] To this end, this application specifically proposes a method, system and device for dynamic fusion of multimodal perception data of embodied intelligent bodies to solve the above technical problems. Summary of the Invention

[0007] The main purpose of the present invention is to provide a method, system and equipment for dynamic fusion of multimodal perception data of an embodied intelligent body. The method utilizes the active perception and motion capabilities of the embodied intelligent body, combined with a large multimodal model to perform deep semantic understanding and feature extraction of the lidar point cloud feature map and camera image data, and obtains initialization alignment through adaptive matching of multimodal features. Then, a proposed multi-scale, multi-channel differentiable mutual information fine optimization method and online monitoring and fusion self-correction of the multimodal large model are utilized, as well as the scene exploration and data collection strategy of the embodied intelligent body are guided to achieve accurate, robust, automatic calibration and fusion of external parameters between the lidar and the camera, so as to solve the technical problems raised in the background technology.

[0008] The present invention adopts the following technical solutions to solve the above technical problems:

[0009] A dynamic fusion method for multimodal perception data of embodied intelligent agents can be applied to various complex indoor and outdoor scenarios, significantly reducing manual intervention and improving fusion efficiency and accuracy. It provides a highly reliable multi-sensor fusion perception foundation for embodied intelligent agents in fields such as autonomous driving, inspection robots, and service robots. The method specifically includes the following steps:

[0010] S1. The embodied agent uses lidar and cameras to actively perceive the environment and collect image data in real time during movement;

[0011] S2. Use a large multimodal model to determine the point cloud filling density of image data and use a mapping algorithm to fill the 3D point cloud of the camera image area during the lidar data acquisition process;

[0012] S3. Generate smooth, continuous and dense lidar feature maps through point cloud data projection transformation ;

[0013] S4. Initialize and align the feature data of the multimodal large model;

[0014] S5. Build a set of multi-scale, multi-channel, differentiable mutual information fine-tuning methods for generating image pyramids based on feature data, constructing mutual information objective functions, probability density estimation and gradient derivation, and multi-scale fine-tuning.

[0015] S6. Perform online monitoring and fusion self-correction of the multimodal large model to achieve online monitoring and adaptive adjustment of the data calibration results after fine optimization of mutual information;

[0016] S7. Based on the adaptively adjusted data, the camera information is projected onto the laser 3D point cloud, and the fused RGB-3D scene is continuously constructed during the movement of the embodied intelligent body.

[0017] Preferably, the specific operation process of step S2 includes:

[0018] S21. Modeling sensor trajectory using polynomial curves: Assume that the sensor is at time The posture of , the polynomial curve is expressed as in is the polynomial curve basis function, is the control point, is the number of control points;

[0019] S22. Perform distortion correction on the point cloud. The corrected point cloud is ,in is the original point cloud, It's time The sensor pose at the moment;

[0020] S23. Perform point cloud registration on the corrected point cloud and the map, and construct the point cloud matching error objective function: ,in is the matching error, is the number of point clouds;

[0021] S24. Minimize the objective function using nonlinear optimization methods , with the control points of the polynomial curve As the control variable, the optimized point cloud pose is obtained, inserted into the 3D point cloud and the global 3D point cloud is updated;

[0022] S25. The multimodal large model identifies the density of the 3D point cloud. When the specified requirements are met, stop filling the point cloud of the camera corresponding area. Otherwise, repeat steps S21 to S24.

[0023] Preferably, the specific operation process of the point cloud data projection conversion in step S3 includes:

[0024] S31. Projecting the dense 3D point cloud onto the 2D camera image plane using camera intrinsic parameters and extrinsic parameters estimated by mechanical measurement;

[0025] S32. After the external parameters are projected onto the image plane, for each camera pixel after projection , collect the feature values ​​of all laser radar points projected into its specified range, use the window kernel function to perform weighted average of the feature values ​​of the points within the specified range to form the feature value of the pixel, and finally obtain a dense laser radar feature map , used to compare with camera images Perform mutual information calculation;

[0026] That is: for each camera pixel , collect all projections to a small radius LiDAR points within The eigenvalue of the pixel is formed by weighted averaging the eigenvalues ​​of the points within the range using the window kernel function:

[0027]

[0028] in, is the window kernel function, is the lidar point , thus obtaining a smooth, continuous and dense lidar feature map , is the pixelated eigenvalue coordinate of the kth laser point after being projected onto the image plane, Represents the projection onto the image plane is the center and the radius is LiDAR points within .

[0029] Preferably, the specific operation process of initializing and aligning the feature data in step S4 includes:

[0030] Using the multimodal large model to train a set of cross-modal feature extraction and matching networks, the cross-modal feature extraction and matching networks are used to learn semantic and structural features extracted from lidar feature maps and camera images to generate initial correspondences;

[0031] Based on the initial correspondence, a set of regression heads are used to predict a rough 6-DOF initial external parameter ;

[0032] The network training of the multimodal large model is based on large-scale simulation or real-world data sets, and the loss function uses the Euclidean distance between the predicted external parameters and the true value, or the loss function uses mutual information as a weak supervision signal for end-to-end learning.

[0033] Preferably, the specific operation process of step S5 includes:

[0034] S51. is the laser radar feature map and camera images Construct a multi-scale image pyramid, each scale ,in is the coarsest scale, 0 is the original resolution, and the image resolution is ,scale Pixel value Obtained through downsampling operation;

[0035] S52. The multi-channel information of the lidar projection image is fused using a weighted mutual information method to construct a multi-channel mutual information objective function. The objective function is:

[0036]

[0037] in, 、 are the external parameters to be optimized, representing the rotation matrix and translation vector between the lidar and the camera, is the channel weight, set to the mean or obtained through learning; the number of projection images , 、 、 They are the depth map, reflection intensity map and semantic projection map of the lidar projection map, is the feature map and images The joint probability distribution of and They are feature maps and images The marginal probability distribution of

[0038] S53. For N pairs of registration points obtained by projection and sampling , , perform probability density estimation and gradient derivation operations;

[0039] S54. Based on the current scale differentiable mutual information value obtained from the above probability density estimation and gradient derivation operations and the gradient of the differentiable mutual information value with respect to the external parameter, perform a multi-scale fine optimization operation.

[0040] Preferably, the specific steps of performing probability density estimation and gradient derivation operations in step S53 include:

[0041] For N pairs of registration points obtained by projection and sampling , , calculate the feature map and images The marginal probability density function of is:

[0042]

[0043]

[0044] Constructing the joint probability density function, we have:

[0045]

[0046] in, 、 and All are kernel functions;

[0047] According to the chain rule, by calculating the mutual information about the external parameter Gradient , the calculation formula is:

[0048]

[0049] in, Through the projection equation and the chain rule, we can get It is the derivative of the mutual information with respect to the pixel value of the feature map, which is used to make the mutual information objective function differentiable, allowing the mutual information objective function to use the optimization algorithm of the gradient descent type.

[0050] Preferably, the specific operation process of performing the multi-scale fine optimization operation in step S54 includes:

[0051] S541. From the coarsest scale Start with the initial external parameters ;

[0052] S542. In scale When , repeat the following steps to reach the maximum number of iterations or convergence:

[0053] a. Through the S3 step, the lidar point cloud uses the current external parameters , projected onto the image plane of the current scale , generate lidar feature map ;

[0054] b. Through step S53, calculate the differentiable mutual information value of the current scale ;

[0055] c. Through step S53, calculate the gradient of the differentiable mutual information value with respect to the external parameter ;

[0056] d. Use the optimizer to update external parameters, including: ,in, is the updated external reference data, is the current external parameter data before updating, is the external parameter update amount, which is obtained by the optimizer based on the gradient calculation;

[0057] S543. Use the optimized external parameters of the current scale as the next finer scale The initial values ​​are: ,in, is the external parameter after optimization of the current scale, For the next finer scale Initial value of external parameter;

[0058] S544. Repeat the above steps S542 to S543 until the scale , that is, to the finest scale.

[0059] Preferably, the specific operation process of step S6 includes:

[0060] S61. Use the multimodal large model to continuously monitor the alignment quality of the lidar and camera data under the current external parameters by setting the mutual information threshold and error threshold;

[0061] S62. Execute multimodal large model assisted error diagnosis operation.

[0062] Preferably, the specific operation process of step S62 includes:

[0063] S621. When calibration drift is detected, use the multimodal large model to obtain current sensor data, environmental information, historical calibration data, monitoring data, and embodied agent motion state information;

[0064] S622. Use the multimodal large model as a high-level intelligent diagnostic tool to infer and identify the causes of data drift.

[0065] S623. Output a diagnostic report and corresponding suggestions for solving the specified problem based on the reasoning and recognition results of the multimodal large model;

[0066] S624. Based on the specified problem-solving suggestions output by the multimodal large model, re-trigger the S54 step to execute a new multi-scale, multi-channel, differentiable mutual information fine optimization process and algorithm parameter adjustment process.

[0067] On the other hand, the present invention further discloses an embodied intelligent body multi-sensor dynamic fusion system, which is used to execute any of the above-mentioned embodied intelligent body multi-modal perception data dynamic fusion methods, comprising:

[0068] embodied agents with lidar and cameras;

[0069] The data transmission module is located on the embodied intelligent body and is used to transmit the collected data of the lidar and camera to the data processing module;

[0070] The data processing module has a large multimodal model that is used to construct a lidar feature map based on the data collected by the lidar and camera, and perform data processing operations as well as online monitoring and fusion self-correction operations to obtain finely optimized environmental data.

[0071] In another aspect, the present invention further discloses a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of the above method.

[0072] On the other hand, the present invention further discloses a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0073] As can be seen from the above technical solutions, the present invention provides a method, system, and device for dynamically fusing multimodal perception data of an embodied intelligent body. Compared with the existing technology, the present invention has the following advantages:

[0074] 1. The method of the present invention breaks the dependence of traditional radar-camera calibration on conditions such as calibration plates, good lighting, and manual operation, and realizes an automatic calibration process without the need for calibration plates and manual intervention, significantly improving the practicality and deployment efficiency of the method.

[0075] 2. This invention uses a large multimodal model to guide embodied intelligent agents to perform autonomous perception and environmental exploration. By combining dense laser point cloud filling with image registration, it can construct a continuous, smooth, and dense radar feature map, greatly improving the accuracy of modal alignment and providing high-quality input for high-precision extrinsic parameter estimation.

[0076] 3. By introducing a multi-scale, multi-channel, and differentiable mutual information optimization model, the present invention can further integrate multiple radar feature channels such as depth maps, reflection intensity maps, and semantic maps, and construct a differentiable objective function, so that the extrinsic parameter estimation process can converge efficiently through optimization algorithms such as gradient descent, thereby improving optimization accuracy and stability.

[0077] 4. Compared with existing deep learning calibration methods, this invention combines feature extraction and matching of multimodal semantic and structural information in the initial alignment stage, without the need for large amounts of manually labeled data. Therefore, it has stronger scene generalization capabilities and cross-domain adaptability, and is suitable for embodied intelligent agents to operate in unknown or dynamic environments.

[0078] 5. By setting up an online monitoring and self-correction mechanism based on a multimodal large model, the present invention can monitor the calibration status in real time, automatically identify error drifts that may be caused by collisions, deformations, environmental interference, etc., and trigger adaptive recalibration and parameter correction, thereby ensuring calibration accuracy and system robustness during long-term operation.

[0079] 6. By using the external reference results to construct a fused RGB-3D scene graph, the present invention achieves high-quality multimodal perception fusion data, provides reliable support for the environmental understanding and autonomous decision-making of embodied intelligent bodies, and promotes the realization of an integrated closed loop of perception-fusion-understanding.

[0080] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become easy to understand through the following description. Of course, it is not necessary to achieve all of the above-mentioned advantages simultaneously in order to implement any product of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are intended to explain the present invention and do not constitute an undue limitation of the present invention. In the accompanying drawings:

[0082] Figure 1 It is a schematic diagram of the overall operation process of the present invention;

[0083] Figure 2 This is a schematic diagram of a high-definition camera capturing images according to a specific embodiment of the present invention;

[0084] Figure 3 This is a schematic diagram of laser radar trajectory point cloud modeling according to a specific embodiment of the present invention;

[0085] Figure 4 This is a schematic diagram of the fused RGB-3D scene image in a specific embodiment of the present invention. Figure 1 ;

[0086] Figure 5 This is a schematic diagram of the fused RGB-3D scene image in a specific embodiment of the present invention. Figure 2 . DETAILED DESCRIPTION

[0087] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. In the absence of conflict, the embodiments in this application and the features in the embodiments can be combined with each other. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0088] In the embodiment, see Figures 1 to 5 .

[0089] like Figure 1 As shown in the figure, an embodiment of the present invention proposes a method for dynamic fusion of multimodal perception data of an embodied intelligent body. It is based on the dynamic fusion of embodied intelligent body lidar and camera of a multimodal large model. The core steps include data preprocessing and feature generation, rough initial alignment, fine optimization of multi-scale differentiable information, and auxiliary decision-making and online monitoring of the multimodal large model. It can be applied to various complex indoor and outdoor scenes. Specifically, it includes the following steps:

[0090] S1. The embodied agent uses lidar and cameras to actively perceive the environment and collect image data in real time during movement, such as Figure 2 The picture shown is that the embodied intelligent body collects images through a high-definition camera, and the images change as the body moves.

[0091] S2. Use the multimodal large model to determine the point cloud filling density of the image data. Since the lidar points of a single frame are sparse, the mapping algorithm is used to fill the 3D point cloud of the camera image area during the lidar data acquisition process. Then, the multimodal large model is used to determine the point cloud filling density. Combined with the camera image guidance, the embodied intelligent agent is controlled to perform lidar scanning of the target area to obtain a dense 3D point cloud, such as Figure 3 The following is shown by modeling the laser radar trajectory: Figure 2 The dense point cloud of the acquired area is equivalent to a 3D reconstruction of the area, which is used in the subsequent steps to generate a smooth, continuous, and dense lidar feature map. The specific operation process includes:

[0092] S21. Modeling sensor trajectory using polynomial curves: Assume that the sensor is at time The posture of , the polynomial curve is expressed as in is the polynomial curve basis function, is the control point, is the number of control points;

[0093] S22. Perform distortion correction on the point cloud. The corrected point cloud is ,in is the original point cloud, It's time The sensor pose at the moment;

[0094] S23. Perform point cloud registration on the corrected point cloud and the map, and construct the point cloud matching error objective function: ,in is the matching error, is the number of point clouds;

[0095] S24. Minimize the objective function using nonlinear optimization methods , with the control points of the polynomial curve As the control variable, the optimized point cloud pose is obtained, inserted into the 3D point cloud and the global 3D point cloud is updated;

[0096] S25. The multimodal large model identifies the density of the 3D point cloud. When the specified requirements are met, stop filling the point cloud of the camera corresponding area. Otherwise, repeat steps S21 to S24.

[0097] At this time, the multimodal large model is used to guide the embodied intelligent agent to perform autonomous perception and environmental exploration. Combined with dense laser point cloud filling and image registration, a continuous, smooth, and dense radar feature map can be constructed, which greatly improves the accuracy of modal alignment and provides high-quality input for high-precision external parameter estimation.

[0098] S3. Generate smooth, continuous and dense lidar feature maps through point cloud data projection transformation , where the specific operation process of point cloud data projection conversion includes:

[0099] S31. Projecting the dense 3D point cloud onto the 2D camera image plane using camera intrinsic parameters and extrinsic parameters estimated by mechanical measurement;

[0100] S32. After the external parameters are projected onto the image plane, for each camera pixel after projection , collect the feature values ​​of all laser radar points projected into its specified range, use the window kernel function to perform weighted average of the feature values ​​of the points within the specified range to form the feature value of the pixel, and finally obtain a dense laser radar feature map , used to compare with camera images Perform mutual information calculation;

[0101] That is: for each camera pixel , collect all projections to a small radius LiDAR points within The eigenvalue of the pixel is formed by weighted averaging the eigenvalues ​​of the points within the range using the window kernel function:

[0102]

[0103] in, is the window kernel function, is the lidar point , thus obtaining a smooth, continuous and dense lidar feature map , is the pixelated eigenvalue coordinate of the kth laser point after being projected onto the image plane, Represents the projection onto the image plane is the center and the radius is LiDAR points within .

[0104] S4. Perform feature data initialization alignment of the multimodal large model. The specific operation process of feature data initialization alignment includes:

[0105] A set of cross-modal feature extraction and matching networks are trained using a large multimodal model. The cross-modal feature extraction and matching networks are used to learn semantic and structural features extracted from lidar feature maps and camera images to generate initial correspondences.

[0106] Based on the initial correspondences, these correspondences are robustly estimated by RANSAC or directly predicted by a regression head MLP to obtain a rough 6-DOF initial external parameter ;

[0107] The network training of the multimodal large model is based on large-scale simulation or real-world data sets, and the loss function uses the Euclidean distance between the predicted external parameters and the true value, or the loss function uses mutual information as a weak supervision signal for end-to-end learning.

[0108] S5. Build a set of multi-scale, multi-channel, differentiable mutual information fine-tuning optimization methods for generating image pyramids based on feature data, constructing mutual information objective functions, probability density estimation and gradient derivation, and multi-scale fine-tuning. The specific operation process includes:

[0109] S51. is the laser radar feature map and camera images Construct a multi-scale image pyramid, each scale ,in is the coarsest scale, 0 is the original resolution, and the image resolution is ,scale Pixel value Obtained through downsampling operation;

[0110] S52. The LiDAR projection image is not limited to the depth map but also includes the reflection intensity map and the semantic projection map. Therefore, we use weighted mutual information to fuse the multi-channel information of the LiDAR projection image to construct a multi-channel mutual information objective function. The objective function is:

[0111]

[0112] in, 、 are the external parameters to be optimized, representing the rotation matrix and translation vector between the lidar and the camera, is the channel weight, set to the mean or obtained through learning; the number of projection images , 、 、 They are the depth map, reflection intensity map and semantic projection map of the lidar projection map, is the feature map and images The joint probability distribution of and They are feature maps and images The marginal probability distribution of

[0113] S53. For N pairs of registration points obtained by projection and sampling , , perform probability density estimation and gradient derivation operations. The specific steps for performing probability density estimation and gradient derivation operations include:

[0114] For N pairs of registration points obtained by projection and sampling , , calculate the feature map and images The marginal probability density function of is:

[0115]

[0116]

[0117] Constructing the joint probability density function, we have:

[0118]

[0119] in, 、 and All are kernel functions;

[0120] According to the chain rule, by calculating the mutual information about the external parameter Gradient , the calculation formula is:

[0121]

[0122] in, Through the projection equation and the chain rule, we can get It is the derivative of the mutual information with respect to the pixel value of the feature map, which is used to make the mutual information objective function differentiable, allowing the mutual information objective function to use the optimization algorithm of the gradient descent type;

[0123] S54. Based on the current scale differentiable mutual information value obtained from the above-mentioned probability density estimation and gradient derivation operations and the gradient of the differentiable mutual information value with respect to the external parameter, a multi-scale fine optimization operation is performed. The specific operation process of performing the multi-scale fine optimization operation at this time includes:

[0124] S541. From the coarsest scale Start with the initial external parameters ;

[0125] S542. In scale When , repeat the following steps to reach the maximum number of iterations or convergence:

[0126] a. Through the S3 step, the lidar point cloud uses the current external parameters , projected onto the image plane of the current scale , generate lidar feature map ;

[0127] b. Through step S53, calculate the differentiable mutual information value of the current scale ;

[0128] c. Through step S53, calculate the gradient of the differentiable mutual information value with respect to the external parameter ;

[0129] d. Use the optimizer to update external parameters, including: ,in, is the updated external reference data, is the current external parameter data before updating, is the external parameter update amount, which is obtained by the optimizer based on the gradient calculation;

[0130] S543. Use the optimized external parameters of the current scale as the next finer scale The initial values ​​are: ,in, is the external parameter after optimization of the current scale, For the next finer scale Initial value of external parameter;

[0131] S544. Repeat the above steps S542 to S543 until the scale , that is, to the finest scale.

[0132] In summary, this method can further integrate multiple radar feature channels such as depth map, reflection intensity map and semantic map by introducing a multi-scale, multi-channel and differentiable mutual information optimization model, and construct a differentiable objective function, so that the extrinsic parameter estimation process can converge efficiently through optimization algorithms such as gradient descent to improve the optimization accuracy and stability. Compared with the existing deep learning calibration method, this method combines the feature extraction and matching of multimodal semantic and structural information in the initial alignment stage, and does not require a large amount of manually labeled data. Therefore, it has stronger scene generalization ability and cross-domain adaptability, and is suitable for embodied intelligent bodies to operate in unknown or dynamic environments.

[0133] S6. Perform online monitoring and fusion self-correction of the multimodal large model to achieve online monitoring and adaptive adjustment of the data calibration results after fine optimization of mutual information. The specific operation process includes:

[0134] S61. Use the multimodal large model to continuously monitor the alignment quality of the lidar and camera data under the current external parameters by setting the mutual information threshold and error threshold;

[0135] S62. Execute multimodal large model assisted error diagnosis operation. The specific operation process includes:

[0136] S621. When calibration drift is detected, use the multimodal large model to obtain current sensor data, environmental information, historical calibration data, monitoring data, and embodied agent motion state information;

[0137] S622. Use the multimodal large model as a high-level intelligent diagnostic tool to infer and identify the causes of data drift.

[0138] S623. Output a diagnostic report and corresponding suggestions for solving the specified problem based on the reasoning and recognition results of the multimodal large model;

[0139] S624. Based on the specified problem-solving suggestions output by the multimodal large model, re-trigger the S54 step to execute a new multi-scale, multi-channel, differentiable mutual information fine optimization process and algorithm parameter adjustment process.

[0140] At this time, by setting up an online monitoring and self-correction mechanism based on a multimodal large model, it is possible to monitor the calibration status in real time, automatically identify error drifts that may be caused by collisions, deformations, environmental interference, etc., and trigger adaptive recalibration and parameter correction, thereby ensuring calibration accuracy and system robustness during long-term operation.

[0141] S7. After obtaining the final lidar and camera external parameters based on the adaptively adjusted data, the camera information is projected onto the laser 3D point cloud, giving the embodied intelligent body the ability to perceive the fused environmental data. During the movement of the embodied intelligent body, the fused RGB-3D scene is continuously constructed, such as Figure 4 、 5 As shown in the figure, after obtaining the fusion parameters, the embodied intelligent agent obtains a globally consistent fusion perception of scene images and distances by collecting data from cameras and lidars during movement or task execution. This is similar to the three-dimensional high-definition scene seen by the human eye, with both high-definition images and a sense of distance, thus providing a good decision-making path for the multimodal large model.

[0142] Therefore, this method can achieve high-quality multimodal perception fusion data by using the external reference results to construct a fused RGB-3D scene graph, providing reliable support for the environmental understanding and autonomous decision-making of embodied intelligent agents, and promoting the realization of an integrated closed loop of perception-fusion-understanding.

[0143] In the specific implementation process, based on a set of standard scenes, the RGB-3D scene construction operation was performed using the technical means disclosed in existing literature and this application. The comparison of the accuracy of randomly selected point clouds is shown in the following table:

[0144]

[0145] Based on the data in the above table, combined Figure 4 、 Figure 5 The test results in the scene show that compared with the method in the above paper, this method can achieve higher calibration and fusion accuracy, and the translation error and rotation error are reduced by 27.45% to 80.42% and 18.79% to 77.21% respectively.

[0146] In summary, this method breaks the traditional radar-camera calibration's reliance on conditions such as calibration plates, good lighting, and manual operation, and realizes an automatic calibration process without the need for calibration plates and manual intervention, significantly improving the method's practicality and deployment efficiency.

[0147] On the other hand, the present invention further discloses an embodied intelligent body multi-sensor dynamic fusion system, which is used to implement the embodied intelligent body multi-modal perception data dynamic fusion method proposed in the above embodiment, comprising:

[0148] embodied agents with lidar and cameras;

[0149] The data transmission module is located on the embodied intelligent body and is used to transmit the collected data of the lidar and camera to the data processing module;

[0150] The data processing module has a large multimodal model that is used to construct a lidar feature map based on the data collected by the lidar and camera, and perform data processing operations as well as online monitoring and fusion self-correction operations to obtain finely optimized environmental data.

[0151] In another aspect, the present invention further discloses a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of the above method.

[0152] On the other hand, the present invention further discloses a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0153] In another embodiment provided in the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute any one of the methods, systems, and devices for dynamic fusion of multimodal perception data of an embodied intelligent body in the above-mentioned embodiments.

[0154] It is understandable that the system provided by the embodiment of the present invention corresponds to the method provided by the embodiment of the present invention, and the explanation, examples and beneficial effects of the relevant contents can refer to the corresponding parts of the above method.

[0155] The embodiment of the present application further provides an electronic device, comprising a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus.

[0156] Memory for storing computer programs;

[0157] The processor is used to implement the above-mentioned method, system and device for dynamic fusion of multimodal perception data of an embodied intelligent body when executing the program stored in the memory.

[0158] The communication bus mentioned in the above electronic device can be a peripheral component interconnect standard bus or an extended industry standard architecture bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc.

[0159] The communication interface is used for communication between the above electronic device and other devices.

[0160] The memory may include a random access memory, or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0161] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor, etc.; it can also be a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component.

[0162] In the above embodiments, all or part of the embodiments can be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, hard disk, tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0163] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

[0164] In addition, it should be noted that if the embodiments of the present invention involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative position relationship, movement status, etc. between the components in a certain specific posture. If the specific posture changes, the directional indications will also change accordingly.

[0165] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. In addition, the meaning of "and / or" appearing in the full text includes three parallel schemes. Taking "A and / or B" as an example, it includes scheme A, or scheme B, or schemes in which A and B are satisfied at the same time. In addition, in the embodiments of the present invention, "multiple" refers to more than two. In addition, the technical solutions between the various embodiments can be combined with each other, but it must be based on the ability of ordinary technicians in this field to implement. When the combination of technical solutions is mutually contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

Claims

1. A method for dynamic fusion of multimodal perception data of an embodied intelligent agent, characterized by: The following steps are involved: S1. The embodied agent uses lidar and cameras to actively perceive the environment and collect image data in real time during movement; S2. Use a large multimodal model to determine the point cloud filling density of image data and use a mapping algorithm to fill the 3D point cloud of the camera image area during the lidar data acquisition process; S3. Generate smooth, continuous and dense lidar feature maps through point cloud data projection transformation ; S4. Initialize and align the feature data of the multimodal large model; S5. Build a set of multi-scale, multi-channel, differentiable mutual information fine-tuning methods for generating image pyramids based on feature data, constructing mutual information objective functions, probability density estimation and gradient derivation, and multi-scale fine-tuning. S6. Perform online monitoring and fusion self-correction of the multimodal large model to achieve online monitoring and adaptive adjustment of the data calibration results after fine optimization of mutual information; S7. Based on the adaptively adjusted data, the camera information is projected onto the laser 3D point cloud, and the fused RGB-3D scene is continuously constructed during the movement of the embodied intelligent body.

2. The method for dynamic fusion of multimodal perception data of an embodied intelligent agent according to claim 1, wherein: The specific operation process of the point cloud data projection conversion in step S3 includes: Project the dense 3D point cloud onto the 2D camera image plane using camera intrinsic parameters and extrinsic parameters estimated by mechanical measurements; After the external parameters are projected onto the image plane, for each camera pixel after projection , collect the feature values ​​of all laser radar points projected into its specified range, use the window kernel function to perform weighted average of the feature values ​​of the points within the specified range to form the feature value of the pixel, and finally obtain a dense laser radar feature map , used to compare with camera images Perform mutual information calculation.

3. The method for dynamic fusion of multimodal perception data of an embodied intelligent agent according to claim 2, wherein: The specific operation process of the feature data initialization alignment in step S4 includes: Using the multimodal large model to train a set of cross-modal feature extraction and matching networks, the cross-modal feature extraction and matching networks are used to learn semantic and structural features extracted from lidar feature maps and camera images to generate initial correspondences; Based on the initial correspondence, a set of regression heads are used to predict a rough 6-DOF initial external parameter ; The network training of the multimodal large model is based on large-scale simulation or real-world data sets, and the loss function uses the Euclidean distance between the predicted external parameters and the true value, or the loss function uses mutual information as a weak supervision signal for end-to-end learning.

4. The method for dynamic fusion of multimodal perception data of an embodied intelligent agent according to claim 3, wherein: The specific operation process of step S5 includes: S51. is the laser radar feature map and camera images Construct a multi-scale image pyramid, each scale ,in is the coarsest scale, 0 is the original resolution, and the image resolution is ,scale Pixel value Obtained through downsampling operation; S52. The multi-channel information of the lidar projection image is fused using a weighted mutual information method to construct a multi-channel mutual information objective function. The objective function is: in, 、 are the external parameters to be optimized, representing the rotation matrix and translation vector between the lidar and the camera, is the channel weight, set to the mean or obtained through learning; the number of projection images , 、 、 They are the depth map, reflection intensity map and semantic projection map of the lidar projection map, is the feature map and images The joint probability distribution of and They are feature maps and images The marginal probability distribution of S53. For N pairs of registration points obtained by projection and sampling , , perform probability density estimation and gradient derivation operations; S54. Based on the current scale differentiable mutual information value obtained from the above probability density estimation and gradient derivation operations and the gradient of the differentiable mutual information value with respect to the external parameter, perform a multi-scale fine optimization operation.

5. The method for dynamic fusion of multimodal perception data of an embodied intelligent agent according to claim 4, characterized in that: The specific steps of performing probability density estimation and gradient derivation operations in step S53 include: For N pairs of registration points obtained by projection and sampling , , calculate the feature map and images The marginal probability density function of is: Constructing the joint probability density function, we have: in, 、 and All are kernel functions; According to the chain rule, by calculating the mutual information about the external parameter Gradient , the calculation formula is: in, Through the projection equation and the chain rule, we can get It is the derivative of the mutual information with respect to the pixel value of the feature map, which is used to make the mutual information objective function differentiable, allowing the mutual information objective function to use the optimization algorithm of the gradient descent type.

6. The method for dynamic fusion of multimodal perception data of an embodied intelligent agent according to claim 5, characterized in that: The specific operation process of performing the multi-scale fine optimization operation in step S54 includes: S541. From the coarsest scale Start with the initial external parameters ; S542. In scale When , repeat the following steps to reach the maximum number of iterations or convergence: a. Through the S3 step, the lidar point cloud uses the current external parameters , projected onto the image plane of the current scale , generate lidar feature map ; b. Through step S53, calculate the differentiable mutual information value of the current scale ; c. Through step S53, calculate the gradient of the differentiable mutual information value with respect to the external parameter ; d. Use the optimizer to update external parameters, including: ,in, is the updated external reference data, is the current external parameter data before updating, is the external parameter update amount, which is obtained by the optimizer based on the gradient calculation; S543. Use the optimized external parameters of the current scale as the next finer scale The initial values ​​are: ,in, is the external parameter after optimization of the current scale, For the next finer scale Initial value of external parameter; S544. Repeat the above steps S542 to S543 until the scale , that is, to the finest scale.

7. The method for dynamic fusion of multimodal perception data of an embodied intelligent agent according to claim 4, characterized in that: The specific operation process of step S6 includes: S61. Use the multimodal large model to continuously monitor the alignment quality of the lidar and camera data under the current external parameters by setting the mutual information threshold and error threshold; S62. Execute multimodal large model assisted error diagnosis operation.

8. The method for dynamic fusion of multimodal perception data of an embodied intelligent agent according to claim 7, wherein: The specific operation process of step S62 includes: S621. When calibration drift is detected, use the multimodal large model to obtain current sensor data, environmental information, historical calibration data, monitoring data, and embodied agent motion state information; S622. Use the multimodal large model as a high-level intelligent diagnostic tool to infer and identify the causes of data drift. S623. Output a diagnostic report and corresponding suggestions for solving the specified problem based on the reasoning and recognition results of the multimodal large model; S624. Based on the specified problem-solving suggestions output by the multimodal large model, re-trigger the S54 step to execute a new multi-scale, multi-channel, differentiable mutual information fine optimization process and algorithm parameter adjustment process.

9. An embodied intelligent body multi-sensor dynamic fusion system, used to execute the embodied intelligent body multi-modal perception data dynamic fusion method according to any one of claims 1 to 8, characterized in that: include: embodied agents with lidar and cameras; The data transmission module is located on the embodied intelligent body and is used to transmit the collected data of the lidar and camera to the data processing module; The data processing module has a large multimodal model that is used to construct a lidar feature map based on the data collected by the lidar and camera, and perform data processing operations as well as online monitoring and fusion self-correction operations to obtain finely optimized environmental data.

10. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 8.

Images