A Multimodal Dynamic Weight Adaptive Fusion Acquisition Method and System for Embossed Robots
By employing a multi-factor cross-correction and dynamic weight generation strategy, combined with an incremental dynamic weight function and an inverse reinforcement learning algorithm, the problems of single-dimensional weight evaluation and unbalanced response in existing multimodal fusion weights are solved, enabling high-precision, safe, and flexible fusion of embodied robots in complex environments.
Patent Information
- Application Number
- CN202610834109.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2026-06-10
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2046-06-10
AI Technical Summary
Existing multimodal fusion technologies cannot comprehensively reflect a robot's motion operation capabilities, the degree of environmental change, and the urgency of the task. Furthermore, the weighting evaluation dimension is singular, making it impossible to make optimal decisions in complex environments. They also lack the ability to balance historical information and instantaneous response, and the fusion results cannot effectively interact with the planning module.
A multi-factor cross-correction and dynamic weight generation strategy is adopted to evaluate operational margin, environmental roughness and task urgency factors in real time. Feedback components are generated through factor graph planning to achieve adaptive adjustment of fusion weights. Combined with incremental dynamic weight function and inverse reinforcement learning algorithm, the stability and agility of weights are ensured.
It achieves accurate and reasonable allocation of fusion weights in complex environments, improving robot task safety and environmental adaptability. The fusion results are optimized in conjunction with the planning module, enhancing task safety and responsiveness under low operational margin conditions.
Smart Images

Figure CN122401431B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of robot intelligent perception and control technology, and in particular to a method and system for multimodal dynamic weight adaptive fusion acquisition of embodied robots. Background Technology
[0002] When embodied robots perform tasks in complex and dynamic environments, they need to simultaneously process multimodal data from various sensors, including vision, depth, touch, force, and inertia. Effectively fusing heterogeneous information from multiple sources with vastly different physical properties to generate robust and accurate environmental perception representations is one of the key technologies for improving the task execution capabilities of embodied robots. Currently, mainstream fusion methods can be categorized into three types based on the timing of fusion: early fusion, mid-stage fusion, and late-stage fusion. Early fusion directly concatenates the raw data and feeds it into a unified network, ignoring the differences in noise levels between different modes. Mid-stage fusion weights and combines features extracted from each mode at the feature level, and is the most widely used approach. Late-stage fusion performs a voting-based synthesis after each mode makes independent decisions. In recent years, dynamic weight fusion methods have improved the fusion effect to some extent by introducing attention mechanisms or gating networks to adaptively adjust the weight allocation of each mode based on the quality of the input data.
[0003] However, existing fusion technologies still have insurmountable shortcomings. First, most dynamic weighting schemes rely on a single evaluation metric, such as signal timing synchronization, illumination intensity, or signal-to-noise ratio, which cannot comprehensively reflect the robot's real needs in a multi-factor coupled environment. When a robot faces multiple challenges simultaneously, such as insufficient operational margin, drastic environmental changes, and increased task urgency, weight allocation based on a single metric is unlikely to make optimal decisions. Second, existing weight update mechanisms generally lack the ability to balance historical information and instantaneous response. Reset updates based entirely on current evaluation results are prone to drastic weight fluctuations due to occasional noise, while gating networks trained entirely offline are slow to respond to sudden environmental changes, failing to balance the stability and agility of weights. Third, current multimodal fusion systems lack information interaction with robot motion planning. The fusion result only serves as a one-way input to the planning module, and the kinematic constraint information generated during planning execution cannot be used to adjust the fusion weights. This results in the fusion system maintaining an unreasonable weight configuration even when the robot is in extreme joint poses or has limited end-effector accessibility.
[0004] Therefore, there is an urgent need in this field for a multimodal adaptive fusion method that can comprehensively reflect the robot's motion operation capabilities, the degree of environmental change, and the urgency of the task, and achieve synergy between weighted smooth response and planning information feedback. Summary of the Invention
[0005] The purpose of this invention is to provide a method and system for adaptive fusion acquisition of multimodal dynamic weights for embodied robots, so as to solve the technical problem that the multimodal fusion weight evaluation dimension in the prior art is single and cannot comprehensively reflect the robot's operating state and environmental changes.
[0006] To achieve the above objectives, this invention provides a multimodal dynamic weight adaptive fusion acquisition method for embodied robots, comprising the following steps: Step S1: Collect multimodal environmental perception data of the embodied robot; Step S2: Before fusion, the channel quality of each modality data is evaluated in real time, multiple evaluation factors are generated, and the multiple evaluation factors are cross-corrected to output the corrected evaluation factors. Step S3: Based on the corrected evaluation factors, calculate the fusion weights of each modality using a dynamic weight generation strategy; Step S4: Based on the fusion weights, perform weighted fusion on the collected multimodal environmental perception data and output fused perception information; Step S5: Based on the factor graph, perform real-time planning on the fused perception information, generate robot action sequences and convert them into control commands. At the same time, extract the motion constraint information generated during the planning process to form feedback components, and inject the feedback components back into the calculation model of the operation margin factor in the evaluation factors in step S2.
[0007] Preferably, the multimodal environmental perception data includes visual data, depth data, tactile data, force data, and inertial measurement data.
[0008] Preferably, the multiple evaluation factors include an operational margin factor, an environmental roughness factor, a task urgency factor, and a tactile-mechanical joint factor; The operating margin factor is denoted as , For time variables, The condition number of the robot's Jacobian matrix is calculated using the following formula: ; in, Let Jacobian matrix be the end effector of the robot. for The condition number of The normalization constant is It is a small positive constant; Environmental roughness factor is denoted as The calculation formula is based on visual data, depth data, and inertial measurement data: ; in, This is the normalized difference in visual brightness entropy extracted from visual data. This is the normalized variance of the depth gradient accumulation extracted from depth data. This is the normalized variance of the motion acceleration variance extracted from inertial measurement data. , , These are weighting coefficients; The task urgency factor is denoted as The calculation is based on the remaining time and progress of the task, using the following formula: ; in, The remaining executable time, For the total planned duration, This represents the percentage of progress already completed. It is a positive deviation coefficient; The tactile-force association factor is denoted as Based on tactile and force data, the calculation formula is as follows: ; in, The variance of the pressure distribution of the tactile sensor array. The time derivative of a six-dimensional force / torque sensor. Let the time derivative be the modulus. , This is the normalization coefficient.
[0009] Preferably, the cross-correction in step S2 is based on the operating margin factor. Environmental roughness factor Task urgency factor and tactile-mechanical association factors The calculations include the following three cases, and the corrected operating margin factor, corrected environmental roughness factor, and corrected task urgency factor are denoted as follows: , , : Scenario 1: When Below the first preset threshold and Higher than the second preset threshold At that time, an amplification correction was applied to the environmental roughness factor to obtain... ; Scenario 2: When Higher than the third preset threshold and Higher than the fourth preset threshold At that time, a saturation acceleration correction is applied to the task urgency factor to obtain... ; Scenario 3: When Below and Below the fifth preset threshold At that time, an inverse decay correction is applied to the operating margin factor to obtain... ; Among them, the tactile-mechanical joint factor The strength of the anti-phase attenuation correction used for dynamically adjusting case three: when Higher than the sixth preset threshold When adjusting the safety attenuation factor in the inverse attenuation correction, this safety attenuation factor is denoted as... ; When multiple conditions are met simultaneously, the correction is performed sequentially according to the priority order of condition 2, condition 1, and condition 3.
[0010] The preferred formula for calculating the amplification correction is: ; in, This is the environmental sensitivity amplification factor; The formula for calculating saturation acceleration correction is: ; in, For acceleration coefficient, This represents the upper limit of urgency. The basic calculation formula for anti-phase attenuation correction is: ; When touch-force association factor hour, Replace with And the replaced Calculations for inverse phase decay correction; corrected operating margin factor The range of values is , It is the minimum effective operating margin factor.
[0011] Preferably, the dynamic weight generation strategy in step S3 includes: First, an incremental dynamic weighting function is used, based on the modified operating margin factor. Corrected environmental roughness factor and the revised task urgency factor Calculate the baseline weights for each mode, the first... The baseline weights for each mode are denoted as: , This indicates the modality number, corresponding to the visual modality, depth modality, tactile modality, force modality, and inertial measurement modality; Then, through an experience-guided inverse reinforcement learning algorithm, the corrected operating margin factor is used. Corrected environmental roughness factor Revised task urgency factor and the tactile-mechanical association factor generated in step S2 As state input, with baseline weights As prior knowledge, output the weight offset, the first... The weight offset of each mode is denoted as ; The weight offset is added to the baseline weight to obtain the final fused weight. .
[0012] Preferably, the formula for calculating the incremental dynamic weight function is as follows: ; in, , , All are activation functions. , , For adjustment coefficients, Incremental coefficient; For the previous moment The baseline weights for each modality.
[0013] Preferred, experience-guided inverse reinforcement learning algorithms include: The reward shaping module based on noise contrast estimation constructs an auxiliary reward function using the noise contrast estimation method, which is used to provide auxiliary reward signals for weight adjustment actions under sparse task reward conditions. A hierarchical policy network, consisting of a high-level network and a low-level network, wherein the high-level network is based on a modified operating margin factor. Corrected environmental roughness factor Revised task urgency factor and tactile-mechanical association factors The output mode selection gating signal allows the lower-level network to output fine-grained weight offsets on the subset of modes selected by the higher-level network. ; The experience replay pool stores historical experience tuples, each containing the corrected operational margin factor for the current time step. Corrected environmental roughness factor Revised task urgency factor and tactile-mechanical association factors Cross-correction records, weight offsets The network parameters are updated by weighting and replaying stored experience tuples based on a priority sampling strategy, and the reward value is calculated.
[0014] This invention also provides a multimodal dynamic weight adaptive fusion acquisition system for embodied robots, comprising: The multimodal data acquisition module is used to collect multimodal environmental perception data of the embodied robot; The evaluation and cross-correction module is used to evaluate the channel quality of each modality data in real time before fusion, generate multiple evaluation factors, cross-correct the multiple evaluation factors, and output the corrected evaluation factors. The dynamic weight generation module is used to calculate the fusion weight of each modality based on the corrected evaluation factors and a dynamic weight generation strategy. The weighted fusion module is used to perform weighted fusion of the collected multimodal environmental perception data according to the fusion weights and output fused perception information. The planning and feedback module is used to perform real-time planning based on the fused perception information using factor graphs, generate robot action sequences and convert them into control commands, extract motion constraint information generated during the planning process to form feedback components, and inject the feedback components back into the operation margin factor calculation model in the evaluation and cross correction module.
[0015] Therefore, the present invention employs the above-mentioned multimodal dynamic weight adaptive fusion acquisition method and system for embodied robots, and the beneficial technical effects are as follows: (1) This invention evaluates the operational margin factor, environmental roughness factor, and task urgency factor in real time before fusion, and performs cross-correction based on the coupling relationship of the three factors, overcoming the shortcomings of existing technologies that rely on a single evaluation index (such as illumination intensity or signal-to-noise ratio) resulting in a single dimension of weight allocation. When the robot is in a low operational margin and the environment changes drastically, the influence of the environmental roughness factor is enhanced by amplification correction; when the task is urgent and the environment is harsh, the weight of the low-latency mode is rapidly increased by saturation acceleration correction. The cross-coupling between the three factors enables the fused weight to comprehensively reflect the robot's motion capability, environmental dynamic characteristics, and task progress constraints, making the weight allocation in complex unstructured scenarios more accurate and reasonable.
[0016] (2) This invention uses an incremental dynamic weight function to calculate the baseline weight, retains historical weight components to ensure a smooth transition of weights, and simultaneously outputs a weight offset by superimposing it on the baseline weight through an experience-guided inverse reinforcement learning algorithm, enabling the system to respond quickly to sudden environmental changes or unexpected disturbances. The dual-source generation architecture of the baseline weight and offset solves the contradictions in the prior art where completely reset updates are susceptible to noise interference and offline training gating networks have lag in response. It ensures the stability of weight adjustment and meets the need for rapid reconstruction in emergency situations.
[0017] (3) This invention extracts motion constraint information from the factor graph planning process as feedback components and injects it back into the calculation model of the operation margin factor, enabling the adjustment of the fusion weights to respond in real time to changes in the robot's own kinematic state. When the robot approaches the joint limit pose or the end-effector accessibility is limited, the operation margin factor is automatically corrected, and the fusion weights tilt towards force and tactile modes, thereby guiding the planning module to generate a safer action sequence. This mechanism overcomes the technical defects of the prior art where the fusion result is unidirectionally input into the planning module and the planning information cannot be used to optimize the fusion parameters in reverse, forming a collaborative working mode of fusion-planning-feedback-refusion, which effectively improves the task safety of the robot under low operation margin conditions. Attached Figure Description
[0018] Figure 1 This is a flowchart of the multimodal dynamic weight adaptive fusion acquisition method for embodied robots according to the present invention; Figure 2 This is an architecture diagram of the multimodal dynamic weight adaptive fusion acquisition system for the embodied robot of the present invention; Figure 3 A comparison of the visual weight response of different methods under sudden environmental changes; Figure 4 This is a comparison chart of fusion accuracy under different operating margins. Detailed Implementation
[0019] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.
[0020] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.
[0021] Example 1 This embodiment builds an embodied robot model in the ROS (Robot Operating System) simulation environment. The robot is equipped with an RGB-D camera, an array of tactile sensors, a six-dimensional force / torque sensor, and an inertial measurement unit. The task scenario is: the robot performs a sequence of tasks—"approaching the target object → grasping → transporting"—in an unknown, unstructured environment. The ambient lighting exhibits periodic abrupt changes, and the ground contains irregular obstacles. The simulation duration is 30 seconds, and the sampling frequency is 50Hz. Figure 1 The overall process of this invention has been demonstrated, and the implementation process will be described in detail below with reference to specific numerical values.
[0022] Step S1: Collect multimodal environmental perception data of the embodied robot.
[0023] Multimodal environment perception data includes: visual data (RGB images, resolution) ), depth data (depth map, resolution) ), tactile data ( The system uses an array of tactile sensors to output pressure distribution matrix, force data (six-dimensional force / torque, sampling frequency 200Hz), and inertial measurement data (three-axis acceleration and angular velocity, sampling frequency 100Hz). All sensor data is synchronized via hardware timestamps and cached in a circular queue for subsequent steps.
[0024] Step S2: Before fusion, the channel quality of each modality data is evaluated in real time, multiple evaluation factors are generated, and the multiple evaluation factors are cross-corrected to output the corrected evaluation factors.
[0025] Multiple assessment factors include operational margin factor, environmental roughness factor, task urgency factor, and tactile-mechanical joint factor.
[0026] (1) The operating margin factor is denoted as , For time variables, The condition number of the robot's Jacobian matrix is calculated using the following formula: ; in, Let Jacobian matrix be the end effector of the robot. for The condition number of The normalization constant is It is a small positive constant; when At that time, Set to the preset minimum effective weight threshold , This is the preset upper limit threshold for the security domain.
[0027] Pick =1.0, =0.01, =5.0, =0.05. The operating margin factor changes with joint motion in the simulation, for example, in... Time measured =4.6, calculated as follows =0.217; in time =8.2 (entering the singular region), triggering an emergency reset. =0.05.
[0028] (2) The environmental roughness factor is denoted as The calculation formula is based on visual data, depth data, and inertial measurement data: ; in, This is the normalized difference in visual brightness entropy extracted from visual data. This is the normalized variance of the depth gradient accumulation extracted from depth data. This is the normalized variance of the motion acceleration variance extracted from inertial measurement data. , , For weighting coefficients; when , , When any one of them exceeds the corresponding preset mutation threshold, the weighting coefficient corresponding to that one increases instantaneously by a preset slope multiple.
[0029] Pick =0.4, =0.4, =0.2, mutation fold is 2.0. In time, =0.32, =0.45, =0.18, not exceeding the threshold, calculated as follows =0.344. In Moment (sudden change in light intensity) =0.85 (exceeds the threshold) (Temporarily changed to 0.8) =0.62, =0.44, calculated as follows =0.676.
[0030] (3) The task urgency factor is denoted as The calculation is based on the remaining time and progress of the task, using the following formula: ; in, The remaining executable time, For the total planned duration, This represents the percentage of progress already completed. This is a positive deviation coefficient. Take... =0.5. In At the current time, 20 seconds remain, progress is 0.4, calculated as follows: =1.8. In At the given time, with 2 seconds remaining and a completion rate of 0.9, the calculation yields... =15.05.
[0031] (4) The tactile-mechanical association factor is denoted as Based on tactile and force data, the calculation formula is as follows: ; in, The variance of the pressure distribution of the tactile sensor array. The time derivative of a six-dimensional force / torque sensor. Let the time derivative be the modulus. , This is the normalization coefficient. Let it be... =0.7, =0.3. In time, =0.12, =0.25, calculated as follows =0.540. In time, =0.32, =0.78, calculated as follows =0.614.
[0032] (5) Cross-correction based on operating margin factor Environmental roughness factor Task urgency factor and tactile-mechanical association factors The calculations include the following three cases, and the corrected operating margin factor, corrected environmental roughness factor, and corrected task urgency factor are denoted as follows: , , : Scenario 1: When Below the first preset threshold (Set to 0.3) and Higher than the second preset threshold When set to 0.5, an amplification correction is applied to the environmental roughness factor, resulting in... .
[0033] The formula for calculating magnification correction is: ; in, =1.0 is the environmental sensitivity amplification factor.
[0034] Scenario 2: When Higher than the third preset threshold (Set to 1.5) and Higher than the fourth preset threshold When set to 0.6, a saturation acceleration correction is applied to the task urgency factor, resulting in... .
[0035] The formula for calculating saturation acceleration correction is: ; in, =0.8 is the acceleration coefficient. =20.0 is the upper limit of urgency.
[0036] Scenario 3: When Below and Below the fifth preset threshold When set to 0.8, an inverse decay correction is applied to the operating margin factor, resulting in... .
[0037] The basic calculation formula for anti-phase attenuation correction is: ; When touch-force association factor At that time, safety attenuation coefficient (Set to 0.6, base value) Replace with And the replaced Calculations for inverse phase decay correction; corrected operating margin factor The range of values is , It is the minimum effective operating margin factor.
[0038] Among them, the tactile-mechanical joint factor The strength of the anti-phase attenuation correction used for dynamically adjusting case three: when Higher than the sixth preset threshold When set to 0.6, adjust the safety attenuation coefficient in the inverse attenuation correction. This safety attenuation coefficient is denoted as... ; When multiple conditions are met simultaneously, the correction is performed sequentially according to the priority order of condition 2, condition 1, and condition 3.
[0039] time: =0.217<0.3, =0.344<0.5, =1.8>1.5 but If the value is less than 0.6, no conditions are met, and no correction will be performed. =0.217, =0.344, =1.8.
[0040] time: =0.05<0.3, =0.676>0.5, =15.05>1.5 and If the value is greater than 0.6, both Case 1 and Case 2 are satisfied. Case 2 will be executed first based on priority. =16.57; Execution Scenario 1: =1.239, truncated to 1.0.
[0041] because =0.614>0.6, Replace with =0.6×1.614≈0.968, although Case three was not triggered, but the subsequent anti-phase decay will use this value. Ultimately... =0.05, =1.0, =16.57.
[0042] Before fusion, factors reflecting robot operational margin, roughness factor reflecting the severity of environmental changes, urgency factor reflecting the remaining task time pressure, and joint factor integrating tactile and force information were evaluated in real time. Cross-correction was then performed based on the coupling relationship between these four factors. This mechanism overcomes the shortcomings of traditional methods that rely solely on a single evaluation index (such as illumination intensity or signal-to-noise ratio), leading to a one-sided weight allocation dimension. When the robot has insufficient operational margin and the environment changes drastically, cross-correction amplifies the impact of the environmental roughness factor; when the task is urgent and the environment is harsh, cross-correction saturates and accelerates the task urgency factor, rapidly increasing the weight of low-latency modes. Through this multi-factor cross-coupling, the corrected evaluation factors output in step S2 comprehensively reflect the robot's motion capabilities, environmental dynamics, and task progress constraints, thus providing a more accurate and reasonable basis for subsequent weight allocation in complex unstructured scenarios.
[0043] Step S3: Based on the corrected evaluation factors, calculate the fusion weights of each modality using a dynamic weight generation strategy.
[0044] (1) The dynamic weight generation strategy includes: first, using an incremental dynamic weight function, based on the corrected operating margin factor output in step S2. Corrected environmental roughness factor and the revised task urgency factor Calculate the baseline weights for each mode, the first... The baseline weights for each mode are denoted as: , This indicates the modality number, corresponding to the visual modality, depth modality, tactile modality, force modality, and inertial measurement modality.
[0045] The formula for calculating the incremental dynamic weight function is: ; in, , , All are activation functions. =0.25、 =0.35、 =0.4 is the adjustment coefficient. Incremental coefficient, The value range is 0.3 to 0.7, and in this embodiment it is set to 0.5; For the previous moment The baseline weights for each modality.
[0046] (2) Then, through an experience-guided inverse reinforcement learning algorithm, the corrected operating margin factor is used. Corrected environmental roughness factor Revised task urgency factor and the tactile-mechanical association factor generated in step S2 As state input, with baseline weights As prior knowledge, output the weight offset, the first... The weight offset of each mode is denoted as .
[0047] Experience-guided inverse reinforcement learning algorithms include: The reward shaping module based on noise contrast estimation constructs an auxiliary reward function using the noise contrast estimation method, which provides an auxiliary reward signal for weight adjustment actions under sparse task reward conditions.
[0048] Discriminator The formula is: ; in, The feature output of the current policy network, The output is a characteristic of the noise distribution, and the state is... ,action , This is the gating vector (1 indicates activation). This represents the weight offset; the offset for inactive modes is 0. (Auxiliary reward function) Defined as: ; Total Rewards ,in, The reward for successful task completion is sparse; +1 for success, 0 otherwise. =0.3.
[0049] A hierarchical policy network, comprising a high-level network and a low-level network, wherein the high-level network is based on the modified operating margin factor input in step S2. Corrected environmental roughness factor Revised task urgency factor and tactile-mechanical association factors The output mode selection gating signal allows the lower-level network to output fine-grained weight offsets on the subset of modes selected by the higher-level network. .
[0050] The experience replay pool stores historical experience tuples, each containing the corrected operational margin factor for the current time step. Corrected environmental roughness factor Revised task urgency factor and tactile-mechanical association factors Cross-correction records, weight offsets The network parameters are updated by weighting and replaying stored experience tuples based on a priority sampling strategy, and the reward value is calculated.
[0051] The weight offset is added to the baseline weight to obtain the final fused weight. .
[0052] This step employs a dual-source weight generation strategy: First, an incremental dynamic weight function is used, combining the weight information from the previous time step with the currently corrected evaluation factor, to calculate the baseline weights for each modality, ensuring a smooth transition of weights over time and avoiding drastic fluctuations caused by occasional noise. Then, an experience-guided inverse reinforcement learning algorithm is used, with the corrected evaluation factor as the state input and the baseline weights as prior knowledge, to output a fine-grained weight offset. This offset is then superimposed on the baseline weights to obtain the final fused weights. This strategy balances the contradiction between historical information and instantaneous response: the baseline weights retain stable components from past time steps, while the inverse reinforcement learning module can quickly respond to environmental changes or sudden disturbances, outputting adaptive offsets. Compared to completely reset weight updates or offline-trained gated networks, this step ensures the stability of weight adjustments while meeting the need for rapid reconstruction of the fusion strategy in emergency situations, significantly improving the robustness of multimodal fusion in dynamic environments.
[0053] Step S4: Weight the data from each modality according to the final fusion weights and output the fused perception information. Perform posterior consistency verification: Repeat the experiment 20 times during the robot health phase and calculate the end-effector position estimation error. Standard deviation: ; in, The number of repeated trials during the healthy phase. =20, The mean is the measured value. =0.0018, which is less than the preset threshold of 0.002, so the increment coefficient is maintained. =0.5 remains unchanged. If If it's too large, reduce it appropriately. (For every time the threshold is exceeded by 100%, Multiply by 0.9 to achieve a smooth downward adjustment.
[0054] Step S5: Construct state nodes (robot pose, joint angles, end effector velocity) and observation factor nodes (visual pose constraints, tactile contact constraints, inertial pre-integration factors) based on the factor graph. Use the fused perception information output from Step S4 as input to the observation factor nodes to perform factor graph optimization and generate a motion sequence (joint angle trajectory). During optimization, extract motion constraint information: end effector reachability margin. =0.2, the index of balanced motion capability in all directions =0.15. This forms the operational margin factor for the injection of feedback components. The computational model, with the Jacobian condition number corrected: ; Pick =0.5, =0.8, so that the next time step It reflects kinematic constraints more sensitively.
[0055] To verify the effectiveness of this invention, comparative experiments were conducted under the same simulation conditions. The visual weight responses of the proposed method, the fixed-weight method, and the single-factor dynamic weight method under sudden environmental changes were tested, as well as the fusion accuracy of the proposed method and the method without cross-correction under different operational margins. The experimental results are as follows.
[0056] Figure 3 The results show that when a sudden change in illumination occurs within 12 seconds, the visual weight of the method in this invention rapidly decreases from 0.50 to 0.35, and further decreases to 0.12 under external impact, effectively avoiding the misleading influence of low-quality visual data on fusion. In contrast, the fixed weight method remains constant at 0.60, while the single-factor dynamic weight method, although decreasing in weight, exhibits a significantly slower response. After illumination recovery, the visual weight of the method in this invention gradually recovers to 0.45, demonstrating good adaptive recovery capability.
[0057] Figure 4 The results show that as the operating margin factor decreases from 0.5 to 0.05, the fusion accuracy of the method of this invention decreases from 0.96 to 0.55, while the accuracy of the method without cross-correction decreases sharply from 0.94 to 0.22. This is especially true in the low operating margin region ( <0.2), the cross correction mechanism of the present invention significantly suppresses the accuracy decay, and verifies the effective improvement of the force and tactile modal weights by the anti-phase decay correction, thereby ensuring the safety of the robot in the vicinity of singular poses.
[0058] Table 1 presents the correction results under three typical operating conditions. For Table 1, the classification criteria for each factor are first clarified: an operational margin factor less than 0.3 is "low margin," and greater than or equal to 0.3 is "normal margin"; an environmental roughness factor less than or equal to 0.5 is "low roughness," greater than 0.5 and less than or equal to 0.6 is "medium roughness," and greater than 0.6 is "high roughness"; a task urgency factor less than or equal to 0.8 is "low urgency," and greater than 1.5 is "high urgency"; a tactile-force joint factor greater than 0.6 is "high tactile-force response." All the above thresholds are derived from the parameters preset in step S2.
[0059] Table 1. Factor correction results in three typical scenarios
[0060] As shown in Table 2, in the initial stage of the task (0-10 seconds), the environment was stable, and visual and depth weights dominated (0.45-0.50 and 0.25-0.30, respectively), while tactile, force, and inertial weights were relatively low. After a sudden change in lighting at 12 seconds, the visual weight rapidly decreased to 0.35, while the tactile and force weights began to rise (to 0.20 and 0.15, respectively), indicating that the system shifted trust from visual to tactile and force modalities. After 14 seconds, the visual weight further decreased to 0.18, while the tactile and force weights rose to 0.30 and 0.25, respectively. At the 18-second external impact, the visual weight dropped to a minimum of 0.12, the inertial weight rose to a maximum of 0.13, and the tactile and force weights remained high. After the impact, the visual weight gradually recovered to 0.40, and the weights of each modality tended to be balanced. The entire process demonstrates the real-time adaptive capability of this invention to the environment and motion state.
[0061] Table 2. Examples of modal weight changes over time (partial time points throughout the task).
[0062] Example 2 like Figure 2 As shown, the embodied robot multimodal dynamic weight adaptive fusion acquisition system includes: The multimodal data acquisition module is used to collect multimodal environmental perception data of the embodied robot; The evaluation and cross-correction module is used to evaluate the channel quality of each modality data in real time before fusion, generate multiple evaluation factors, cross-correct the multiple evaluation factors, and output the corrected evaluation factors. The dynamic weight generation module is used to calculate the fusion weight of each modality based on the corrected evaluation factors and a dynamic weight generation strategy. The weighted fusion module is used to perform weighted fusion of the collected multimodal environmental perception data according to the fusion weights and output fused perception information. The planning and feedback module is used to perform real-time planning based on the fused perception information using factor graphs, generate robot action sequences and convert them into control commands, extract motion constraint information generated during the planning process to form feedback components, and inject the feedback components back into the operation margin factor calculation model in the evaluation and cross correction module.
[0063] It is worth noting that all contents not described in detail in this invention are existing technologies and are well known to those skilled in the art.
[0064] Therefore, the present invention employs the above-mentioned multimodal dynamic weight adaptive fusion acquisition method and system for embodied robots, which has the advantages of high fusion accuracy, strong environmental adaptability, agile weight response, and good coordination with motion planning.
[0065] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.
Claims
1. A method for adaptive fusion acquisition of multimodal dynamic weights in embodied robots, characterized in that, Includes the following steps: Step S1: Collect multimodal environmental perception data of the embodied robot; Step S2: Before fusion, the channel quality of each modality data is evaluated in real time, multiple evaluation factors are generated, and the multiple evaluation factors are cross-corrected to output the corrected evaluation factors. Multiple assessment factors include operational margin factor, environmental roughness factor, task urgency factor, and tactile-mechanical joint factor; The operating margin factor is denoted as , For time variables, The condition number of the robot's Jacobian matrix is calculated using the following formula: ; in, Let Jacobian matrix be the end effector of the robot. for The condition number of The normalization constant is It is a small positive constant; Environmental roughness factor is denoted as The calculation formula is based on visual data, depth data, and inertial measurement data: ; in, This is the normalized difference in visual brightness entropy extracted from visual data. This is the normalized variance of the depth gradient accumulation extracted from depth data. This is the normalized variance of the motion acceleration variance extracted from inertial measurement data. , , These are weighting coefficients; The task urgency factor is denoted as The calculation is based on the remaining time and progress of the task, using the following formula: ; in, The remaining executable time, For the total planned duration, This represents the percentage of progress already completed. It is a positive deviation coefficient; The tactile-force association factor is denoted as Based on tactile and force data, the calculation formula is as follows: ; in, The variance of the pressure distribution of the tactile sensor array. The time derivative of a six-dimensional force / torque sensor. Let the time derivative be the modulus. , These are the normalization coefficients; Step S3: Based on the corrected evaluation factors, calculate the fusion weights of each modality using a dynamic weight generation strategy; Step S4: Based on the fusion weights, perform weighted fusion on the collected multimodal environmental perception data and output fused perception information; Step S5: Based on the factor graph, perform real-time planning on the fused perception information, generate robot action sequences and convert them into control commands. At the same time, extract the motion constraint information generated during the planning process to form feedback components, and inject the feedback components back into the calculation model of the operation margin factor in the evaluation factors in step S2.
2. The multimodal dynamic weight adaptive fusion acquisition method for embodied robots according to claim 1, characterized in that, Multimodal environmental perception data includes visual data, depth data, tactile data, force data, and inertial measurement data.
3. The multimodal dynamic weight adaptive fusion acquisition method for embodied robots according to claim 2, characterized in that, The cross-correction in step S2 is based on the operating margin factor. Environmental roughness factor Task urgency factor and tactile-mechanical association factors The calculations include the following three cases, and the corrected operating margin factor, corrected environmental roughness factor, and corrected task urgency factor are denoted as follows: , , : Scenario 1: When Below the first preset threshold and Higher than the second preset threshold At that time, an amplification correction was applied to the environmental roughness factor to obtain... ; Scenario 2: When Higher than the third preset threshold and Higher than the fourth preset threshold At that time, a saturation acceleration correction is applied to the task urgency factor to obtain... ; Scenario 3: When Below and Below the fifth preset threshold At that time, an inverse decay correction is applied to the operating margin factor to obtain... ; in Tactile-force association factor The strength of the anti-phase attenuation correction used for dynamically adjusting case three: when Higher than the sixth preset threshold When adjusting the safety attenuation factor in the inverse attenuation correction, this safety attenuation factor is denoted as... ; When multiple conditions are met simultaneously, the correction is performed sequentially according to the priority order of condition 2, condition 1, and condition 3.
4. The multimodal dynamic weight adaptive fusion acquisition method for embodied robots according to claim 3, characterized in that, The formula for calculating magnification correction is: ; in, This is the environmental sensitivity amplification factor; The formula for calculating saturation acceleration correction is: ; in, For acceleration coefficient, This represents the upper limit of urgency. The basic calculation formula for anti-phase attenuation correction is: ; When touch-force association factor hour, Replace with And the replaced Calculations for inverse phase decay correction; corrected operating margin factor The range of values is , It is the minimum effective operating margin factor.
5. The multimodal dynamic weight adaptive fusion acquisition method for embodied robots according to claim 1, characterized in that, The dynamic weight generation strategy in step S3 includes: First, an incremental dynamic weighting function is used, based on the modified operating margin factor. Corrected environmental roughness factor and the revised task urgency factor Calculate the baseline weights for each mode, the first... The baseline weights for each mode are denoted as: , This indicates the modality number, corresponding to the visual modality, depth modality, tactile modality, force modality, and inertial measurement modality; Then, through an experience-guided inverse reinforcement learning algorithm, the corrected operating margin factor is used. Corrected environmental roughness factor Revised task urgency factor and the tactile-mechanical association factor generated in step S2 As state input, with baseline weights As prior knowledge, output the weight offset, the first... The weight offset of each mode is denoted as ; The weight offset is added to the baseline weight to obtain the final fused weight. .
6. The multimodal dynamic weight adaptive fusion acquisition method for embodied robots according to claim 5, characterized in that, The formula for calculating the incremental dynamic weight function is: ; in, , , All are activation functions. , , For adjustment coefficients, Incremental coefficient; For the previous moment The baseline weights for each modality.
7. The multimodal dynamic weight adaptive fusion acquisition method for embodied robots according to claim 6, characterized in that, Experience-guided inverse reinforcement learning algorithms include: The reward shaping module based on noise contrast estimation constructs an auxiliary reward function using the noise contrast estimation method, which is used to provide auxiliary reward signals for weight adjustment actions under sparse task reward conditions. A hierarchical policy network, consisting of a high-level network and a low-level network, wherein the high-level network is based on a modified operating margin factor. Corrected environmental roughness factor Revised task urgency factor and tactile-mechanical association factors The output mode selection gating signal allows the lower-level network to output fine-grained weight offsets on the subset of modes selected by the higher-level network. ; The experience replay pool stores historical experience tuples, each containing the corrected operational margin factor for the current time step. Corrected environmental roughness factor Revised task urgency factor and tactile-mechanical association factors Cross-correction records, weight offsets The network parameters are updated by weighting and replaying stored experience tuples based on a priority sampling strategy, and the reward value is calculated.
8. A multimodal dynamic weighted adaptive fusion acquisition system for embodied robots, characterized in that, The method for implementing the multimodal dynamic weight adaptive fusion acquisition method for embodied robots as described in any one of claims 1-7 includes: The multimodal data acquisition module is used to collect multimodal environmental perception data of the embodied robot; The evaluation and cross-correction module is used to evaluate the channel quality of each modality data in real time before fusion, generate multiple evaluation factors, cross-correct the multiple evaluation factors, and output the corrected evaluation factors. The dynamic weight generation module is used to calculate the fusion weight of each modality based on the corrected evaluation factors and a dynamic weight generation strategy. The weighted fusion module is used to perform weighted fusion of the collected multimodal environmental perception data according to the fusion weights and output fused perception information. The planning and feedback module is used to perform real-time planning based on the fused perception information using factor graphs, generate robot action sequences and convert them into control commands, extract motion constraint information generated during the planning process to form feedback components, and inject the feedback components back into the operation margin factor calculation model in the evaluation and cross correction module.
Citation Information
Patent Citations
Industrial robot adaptive control method and system based on multi-modal sensor fusion
CN121515213A
