Simulation scene physical parameter calibration method and device based on multi-modal data

Abstract

The application discloses a simulation scene physical parameter calibration method and device based on multi-modal data, comprising: writing current simulation physical parameter information into a simulation scene to perform a target contact operation task, and collecting a current multi-modal simulation response; calculating a physical simulation error between the current multi-modal simulation response and a multi-modal real response; based on a preset correction algorithm, processing the current simulation physical parameter information and the current multi-modal simulation response to obtain a simulation physical parameter correction amount, and correcting the current simulation physical parameter information according to the simulation physical parameter correction amount, minimum boundaries and maximum boundaries of each simulation physical parameter; when the physical simulation error is greater than a preset error threshold, taking the corrected simulation physical parameter information as new current simulation physical parameter information, returning to perform the first step until the error is less than or equal to the preset error threshold, or when the number of iterations is reached, taking the latest obtained corrected simulation physical parameter information as target simulation physical parameter information.

Description

Technical Field

[0001] This application relates to the field of simulation technology, and more specifically, to a method and apparatus for calibrating physical parameters of a simulation scene based on multimodal data. Background Technology

[0002] In embodied intelligence scenarios, physical parameter calibration is typically performed in a simulation environment before the calibrated parameters are transferred to the real device. This process reduces wear and tear on the real device and the cost of manual trial and error, while improving the training efficiency of contact and manipulation tasks. However, whether a simulation scenario can effectively support real-world tasks depends on whether the simulated physical parameters accurately reflect the physical characteristics of the real-world scenario. For contact and manipulation tasks, the physical interaction process in the real-world scenario is usually characterized by multiple coupled factors, and the accuracy of its representation in the simulation scenario will affect the operation results.

[0003] Existing methods for calibrating physical parameters in simulation scenarios for contact and manipulation tasks typically rely on single-modal data to calibrate the physical parameters within the simulation scenario. While these methods can reduce the difference between the simulation and real-world scenarios under stable observation conditions, their reliance on a single source of observational information makes it difficult to accurately characterize the actual physical interaction processes. In contact and manipulation tasks, the interaction between the executor and the environment is usually simultaneously influenced by factors such as mass, friction, contact stiffness, and damping. Relying solely on single-modal data for simulation parameter calibration makes it difficult to accurately distinguish the impact of different physical factors on the task results, easily leading to inaccurate parameter correction and thus discrepancies between the calibrated simulation and the real-world scenario. Summary of the Invention

[0004] This application provides a method and apparatus for calibrating physical parameters of a simulation scene based on multimodal data, which can solve the problem of low accuracy in calibrating physical parameters in a simulation scene based on single-modal data.

[0005] The specific technical solution is as follows: In a first aspect, embodiments of this application provide a method for calibrating physical parameters of a simulation scene based on multimodal data, the method comprising: The current simulated physical parameter information is written into the simulation scene of the embodied intelligent device, the target contact operation task is executed, and the current multimodal simulation response is collected. The target contact operation task is the same as the contact operation task in the real scene. Calculate the current physical simulation error between the current multimodal simulation response and the actual multimodal response; Based on a preset correction algorithm, the current simulation physical parameter information and the current multimodal simulation response are processed to obtain the simulation physical parameter correction amount corresponding to the current simulation physical parameter information. Then, based on the simulation physical parameter correction amount and the minimum and maximum boundaries of each simulation physical parameter, the current simulation physical parameter information is corrected to obtain the corrected simulation physical parameter information. If the current physical simulation error is greater than a preset error threshold, the corrected simulation physical parameter information is used as the new current simulation physical parameter information. The process returns to write the current simulation physical parameter information into the simulation scene of the embodied intelligent device and execute the target contact operation task until the current physical simulation error is less than or equal to the preset error threshold, or the target number of iterations is reached. In this case, the most recently obtained corrected simulation physical parameter information is used as the target simulation physical parameter information.

[0006] In one possible implementation, the preset correction algorithm includes a multimodal physical parameter deviation network, which includes a multimodal deviation encoding module, an intermediate fusion module, and a parameter correction module. Based on a preset correction algorithm, the current simulation physical parameter information and the current multimodal simulation response are processed to obtain the simulation physical parameter correction amount corresponding to the current simulation physical parameter information, including: The current multimodal simulation response is encoded using the two fully connected layers of the multimodal deviation coding module to obtain the current multimodal simulation coding features; The intermediate fusion module is used to fuse the current multimodal simulation coding features with the current simulation physical parameter information to obtain the current multimodal simulation fusion features. The fully connected layer in the intermediate fusion module is used to map the current multimodal simulation fusion features into the current multimodal fusion intermediate features. The three fully connected layers of the parameter correction module are used to correct the current multimodal fusion intermediate features to obtain the initial simulation physical parameter correction amount corresponding to the current simulation physical parameter information; The output range limiting layer of the parameter correction module is used to limit the range of the initial simulation physical parameter correction amount, and the final simulation physical parameter correction amount is output.

[0007] In one possible implementation, the output range limiting layer of the parameter correction module is used to limit the range of the initial simulation physical parameter correction amount, and the final simulation physical parameter correction amount is output, including: Using a preset range limitation formula, calculate the final simulation physical parameter correction amount corresponding to the current simulation physical parameter information; The preset range limitation formula includes: ; Among them, the S represents the final simulation physical parameter correction amount corresponding to the current simulation physical parameter information. S is used to limit the maximum correction range of each simulation physical parameter in a single iteration. Z represents the initial simulation physical parameter correction amount.

[0008] In one possible implementation, the training loss function of the multimodal physical parameter deviation network includes a parameter correction loss and a physical simulation error loss. The parameter correction loss is used to constrain the amount of simulation physical parameter correction output by the multimodal physical parameter deviation network to be close to the training label, and the physical simulation error loss is used to constrain the multimodal simulation response obtained based on the corrected simulation physical parameter information to be close to the multimodal real response. The training label is the difference between the simulated physical parameter information and the preferred physical parameter information used in each training session; The method for determining the preferred physical parameter information includes: Each initial simulation physical parameter in the initial simulation physical parameter information of the simulation scene is perturbed to generate multiple candidate simulation physical parameter information; Each candidate simulation physical parameter information is written into the simulation scene, the target contact operation task is executed, and multimodal simulation response is collected; Calculate the physical simulation error between each multimodal simulation response and the actual multimodal response; The candidate simulation physical parameter information with the smallest physical simulation error is selected as the preferred physical parameter information.

[0009] In one possible implementation, the simulated physical parameter information includes at least one of the following: mass of the manipulated object, equivalent moment of inertia of the manipulated object, contact stiffness, contact damping, static friction coefficient, and dynamic friction coefficient; and / or, The current multimodal simulation response includes at least two of the following: current simulation visual response, current simulation force response, current simulation tactile response, current simulation motion trajectory response, current simulation joint state response, current simulation sound data response, and current simulation workpiece deformation response; correspondingly, the multimodal real response includes at least two of the following: real visual response, real force response, real tactile response, real motion trajectory response, real joint state response, real sound data response, and real workpiece deformation response.

[0010] Secondly, embodiments of this application provide a simulation scene physical parameter calibration device based on multimodal data, the device comprising: The simulation unit is used to write the current simulation physical parameter information into the simulation scene of the embodied intelligent device, execute the target contact operation task, and collect the current multimodal simulation response. The target contact operation task is the same as the contact operation task in the real scene. The calculation unit is used to calculate the current physical simulation error between the current multimodal simulation response and the actual multimodal response; The correction amount determination unit is used to process the current simulation physical parameter information and the current multimodal simulation response based on a preset correction algorithm to obtain the simulation physical parameter correction amount corresponding to the current simulation physical parameter information; The correction unit is used to correct the current simulation physical parameter information according to the correction amount of the simulation physical parameters, the minimum boundary and the maximum boundary of each simulation physical parameter, so as to obtain the corrected simulation physical parameter information. The setting unit is also used to, when the current physical simulation error is greater than a preset error threshold, take the corrected simulation physical parameter information as the new current simulation physical parameter information and return to the execution unit; The parameter determination unit is used to take the most recently obtained corrected simulation physical parameter information as the target simulation physical parameter information until the current physical simulation error is less than or equal to the preset error threshold, or until the target number of iterations is reached.

[0011] In one possible implementation, the preset correction algorithm includes a multimodal physical parameter deviation network, which includes a multimodal deviation encoding module, an intermediate fusion module, and a parameter correction module. The correction amount determination unit includes: The multimodal deviation coding module is used to encode the current multimodal simulation response using its own two fully connected layers to obtain the current multimodal simulation coding features; The intermediate fusion module is used to fuse the current multimodal simulation coding features with the current simulation physical parameter information to obtain the current multimodal simulation fusion features, and to use the fully connected layer in the intermediate fusion module to map the current multimodal simulation fusion features into the current multimodal fusion intermediate features. The parameter correction module is used to correct the current multimodal fusion intermediate features using its own three fully connected layers to obtain the initial simulation physical parameter correction amount corresponding to the current simulation physical parameter information; and to limit the range of the initial simulation physical parameter correction amount using the output range limiting layer to output the final simulation physical parameter correction amount.

[0012] In one possible implementation, the parameter correction module is used to calculate the final simulation physical parameter correction amount corresponding to the current simulation physical parameter information using a preset range limitation formula. The preset range limitation formula includes: ; Among them, the S represents the final simulation physical parameter correction amount corresponding to the current simulation physical parameter information. S is used to limit the maximum correction range of each simulation physical parameter in a single iteration. Z represents the initial simulation physical parameter correction amount.

[0013] In one possible implementation, the training loss function of the multimodal physical parameter deviation network includes a parameter correction loss and a physical simulation error loss. The parameter correction loss is used to constrain the amount of simulation physical parameter correction output by the multimodal physical parameter deviation network to be close to the training label, and the physical simulation error loss is used to constrain the multimodal simulation response obtained based on the corrected simulation physical parameter information to be close to the multimodal real response. The training label is the difference between the simulated physical parameter information and the preferred physical parameter information used in each training session; The parameter determination unit is further configured to perturb each initial simulation physical parameter in the initial simulation physical parameter information of the simulation scene to generate multiple candidate simulation physical parameter information; write each candidate simulation physical parameter information into the simulation scene, execute the target contact operation task, and collect multimodal simulation responses; calculate the physical simulation error between each multimodal simulation response and the multimodal real response; and select the candidate simulation physical parameter information with the smallest physical simulation error as the preferred physical parameter information.

[0014] In one possible implementation, the simulated physical parameter information includes at least one of the following: mass of the manipulated object, equivalent moment of inertia of the manipulated object, contact stiffness, contact damping, static friction coefficient, and dynamic friction coefficient; and / or, The current multimodal simulation response includes at least two of the following: current simulation visual response, current simulation force response, current simulation tactile response, current simulation motion trajectory response, current simulation joint state response, current simulation sound data response, and current simulation workpiece deformation response; correspondingly, the multimodal real response includes at least two of the following: real visual response, real force response, real tactile response, real motion trajectory response, real joint state response, real sound data response, and real workpiece deformation response.

[0015] Thirdly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the method as described in any possible implementation of the first aspect.

[0016] Fourthly, embodiments of this application provide an electronic device, which includes: One or more processors; The processor is coupled to a storage device for storing one or more programs; When one or more programs are executed by one or more processors, the electronic device performs the method as described in any possible implementation of the first aspect.

[0017] Fifthly, embodiments of this application provide a computer program product containing instructions that, when executed on a computer or processor, cause the computer or processor to perform the method described in any possible implementation of the first aspect.

[0018] The simulation scene physical parameter calibration method and apparatus based on multimodal data provided in this application introduces the physical simulation error between the multimodal simulation response and the multimodal real response as feedback, and uses a preset correction algorithm combined with parameter boundary constraints to iteratively correct the simulation physical parameters. This can effectively distinguish the influence of multiple factors such as mass, friction, stiffness, and damping on contact operation tasks, avoiding the parameter identification ambiguity caused by the single information in traditional single-modal calibration methods. At the same time, by iterating repeatedly until the physical simulation error converges to within the threshold or reaches the target number of iterations, the final target simulation physical parameters can more accurately reflect the physical characteristics of the real scene, thereby significantly reducing the dynamic deviation between the simulation and the real environment, improving the effectiveness of the calibrated simulation scene in supporting embodied intelligent tasks (such as contact and operation), and reducing the secondary calibration cost when migrating real equipment.

[0019] Furthermore, the technical effects achieved by the embodiments of this application may also include: 1. This application embodiment constructs a multimodal physical parameter deviation network including a multimodal deviation coding module, an intermediate fusion module, and a parameter correction module. First, the multimodal deviation coding module encodes the current multimodal simulation response to obtain the current multimodal simulation coding features. Then, the intermediate fusion module fuses the current multimodal simulation coding features with the current simulation physical parameter information, and after passing through a fully connected layer, obtains the current multimodal fusion intermediate features. Finally, the parameter correction module corrects and limits the output of the current multimodal fusion intermediate features to obtain the final simulation physical parameter correction amount. Instead of simply splicing or processing the multimodal data separately, this approach further solves the technical problem of difficulty in uniformly modeling different modal data and ultimately achieves the effect of enabling multimodal deviation information to jointly participate in the simulation physical parameter correction.

[0020] 2. In the training of the multimodal physical parameter deviation network, this application embodiment combines the optimization of parameter correction loss and physical simulation error loss. This ensures the network is simultaneously subject to two complementary constraints during training: First, the parameter correction loss directly supervises the network output correction amount to approximate the training label composed of the difference between the simulation parameters and the optimized parameters, guaranteeing clear prior guidance for the direction and magnitude of parameter correction. Second, the physical simulation error loss indirectly supervises the approximation between the multimodal simulation response generated by the corrected parameters and the actual multimodal response, forcing the network to learn the physical parameter adjustment rules that truly reduce the dynamic differences between simulation and reality. This dual-loss collaborative training mechanism effectively overcomes the limitations of relying solely on parameter labels or response errors. It enables the network not only to fit known optimized parameter samples but also to output reasonable parameter correction amounts in unseen scenarios through the generalization characteristics of physical simulation errors, thereby significantly improving the accuracy and robustness of simulation physical parameter calibration. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0022] Figure 1 A flowchart illustrating a method for calibrating physical parameters of a simulation scene based on multimodal data, provided in an embodiment of this application; Figure 2 A flowchart illustrating another method for calibrating physical parameters of a simulation scene based on multimodal data, provided in this application embodiment; Figure 3This is a block diagram illustrating the composition of a simulation scene physical parameter calibration device based on multimodal data, as provided in an embodiment of this application. Detailed Implementation

[0023] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0024] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The terms "comprising" and "having," and any variations thereof, in the embodiments and drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0025] This application presents a flowchart illustrating a method for calibrating physical parameters of a simulation scene based on multimodal data. This method can be applied to electronic devices or computer equipment. The following is a detailed explanation... Figure 1 and Figure 2 The method can be explained and may include the following steps: S110: Write the current simulation physical parameter information into the simulation scene of the embodied intelligent device, execute the target contact operation task, and collect the current multimodal simulation response.

[0026] The simulated physical parameter information used in the first execution of this application embodiment is the initial simulated physical parameter information. The simulated physical parameter information includes at least one of the following: the mass of the manipulated object, the equivalent moment of inertia of the manipulated object, contact stiffness, contact damping, static friction coefficient, and dynamic friction coefficient. The target contact operation task is the same as the contact operation task in the real scenario.

[0027] The current multimodal simulation response includes at least two of the following: current simulated visual response, current simulated force response, current simulated tactile response, current simulated motion trajectory response, current simulated joint state response, current simulated sound data response, and current simulated workpiece deformation response. Correspondingly, the multimodal realistic response includes at least two of the following: realistic visual response, realistic force response, realistic tactile response, realistic motion trajectory response, realistic joint state response, realistic sound data response, and realistic workpiece deformation response. The multimodal data combination can be adjusted according to sensor configuration and task type. For example, in scenarios without tactile sensors, visual data, force data, and joint state data can be used; in flexible workpiece manipulation scenarios, workpiece deformation data can be further introduced. All of the above data combination methods can be used to expand the observation dimensions of the physical interaction process.

[0028] Real-world scenarios (i.e.) Figure 2 The actual operational scenario (in the context of the simulation) includes a robotic arm, an end effector, the object being manipulated, a worktable, and multimodal sensors. Multimodal sensors include vision devices, force sensors, and tactile sensors. Vision devices are used to acquire the pose changes of the object being manipulated and the end effector; force sensors are used to acquire the force response during contact; and tactile sensors are used to acquire the contact area, contact pressure, and slippage state.

[0029] The simulation scenario establishes corresponding robot arm models, end effector models, workpiece models, worktable models, and contact models based on the real scenario, and sets initial physical parameters.

[0030] In real-world scenarios, a robotic arm is controlled to perform contact operations such as grasping, pushing, pulling, assembling, and inserting / removing, while simultaneously acquiring multimodal real-world responses, including real visual responses, real force responses, and real tactile responses. Real visual responses, after extraction, can form multi-dimensional visual features (e.g., 12-dimensional visual features), including the three-dimensional posture of the workpiece and end effector, the relative position of the workpiece and end effector, and the workpiece displacement before and after contact. Real force responses, after statistical analysis, also form multi-dimensional force features (e.g., 8-dimensional force features), including three-dimensional contact force, three-dimensional contact torque, peak contact force, and average contact force. Real tactile responses, after contact area analysis, form multi-dimensional tactile features (e.g., 8-dimensional force features), including contact area, contact center position, maximum contact pressure, average contact pressure, pressure change rate, and slip indication. After repeatedly performing the contact operation process, multimodal data under different conditions are obtained. The acquired real visual, force, and tactile responses are then synchronized in time, standardized in coordinates, have outliers removed, and normalized to form a physical parameter sample library.

[0031] S120: Calculate the current physical simulation error between the current multimodal simulation response and the current multimodal real response.

[0032] The same contact operation tasks as in the real scene are performed in the simulation scenario, and the simulated visual response, simulated force response, and simulated tactile response are collected. The multimodal data from the real scene are aligned with the multimodal data from the simulation scenario, and the current physical simulation error between the current multimodal simulated response and the multimodal real response is calculated. Visual deviation features can be 12-dimensional, reflecting the degree of inconsistency in three-dimensional pose, relative position, and displacement changes; force deviation features can be 8-dimensional, reflecting the degree of inconsistency in contact force and torque; tactile deviation features can be 8-dimensional, reflecting the degree of inconsistency in contact area, contact pressure, and slip state. To obtain scalar error, normalized mean square processing is performed on the visual deviation features, force deviation features, and tactile deviation features respectively to obtain the visual deviation scalar. Force perception deviation scalar and tactile deviation scalar Among them, each deviation scalar is used to characterize the average degree of deviation between the simulated response and the actual response under the corresponding mode.

[0033] The current physical simulation error E can be obtained by weighting each modal deviation. Taking three deviations as an example, the current physical simulation error E can be: ; in, This is a visual bias. For force perception deviation, This is due to tactile bias. , , These are the weights corresponding to visual bias, force bias, and tactile bias, respectively.

[0034] S130: Based on the preset correction algorithm, process the current simulation physical parameter information and the current multimodal simulation response to obtain the simulation physical parameter correction amount corresponding to the current simulation physical parameter information, and correct the current simulation physical parameter information according to the simulation physical parameter correction amount, the minimum boundary and the maximum boundary of each simulation physical parameter to obtain the corrected simulation physical parameter information.

[0035] The preset correction algorithm includes a multimodal physical parameter deviation network, which includes a multimodal deviation encoding module, an intermediate fusion module, and a parameter correction module.

[0036] The current simulation physical parameter information and the current multimodal simulation response are input into the multimodal physical parameter deviation network, and the simulation physical parameter correction amount corresponding to the current simulation physical parameter information is output. The specific implementation of this step may include: encoding the current multimodal simulation response using two fully connected layers of the multimodal deviation encoding module to obtain the current multimodal simulation encoded features; fusing the current multimodal simulation encoded features with the current simulation physical parameter information using the intermediate fusion module to obtain the current multimodal simulation fused features, and mapping the current multimodal simulation fused features to the current multimodal fused intermediate features using the fully connected layers in the intermediate fusion module; correcting the current multimodal fused intermediate features using three fully connected layers of the parameter correction module to obtain the initial simulation physical parameter correction amount corresponding to the current simulation physical parameter information; and limiting the range of the initial simulation physical parameter correction amount using the output range limiting layer of the parameter correction module to output the final simulation physical parameter correction amount.

[0037] Each module is described in detail below: 1. Multimodal deviation coding module The multimodal deviation coding module includes multiple coding branches, with one branch corresponding to each modality. Examples include visual coding, force coding, and tactile coding branches. For instance, the visual deviation coding branch takes 12-dimensional visual simulation deviation features as input, while the force and tactile deviation coding branches take 8-dimensional force simulation deviation features and 8-dimensional tactile simulation deviation features as input, respectively. Each coding branch employs two fully connected layers, first mapping the input features to 32-dimensional intermediate features, and then mapping them to 16-dimensional simulation coding features. A ReLU activation function is applied after each fully connected layer to enhance the network's ability to express nonlinear contact physical deviations.

[0038] 2. Intermediate Fusion Module In the process of using an intermediate fusion module to fuse the current multimodal simulation coding features with the current simulation physical parameter information to obtain the current multimodal simulation fusion features, the fusion methods include, but are not limited to, splicing, weighted attention fusion, gated fusion, or hierarchical fusion, as long as they can uniformly express the deviation information of different modes with the current simulation physical parameters and be used to output the physical parameter correction amount.

[0039] Taking the aforementioned three modalities as examples, the intermediate fusion module concatenates the 16-dimensional visual simulation coding features, 16-dimensional force simulation coding features, 16-dimensional tactile simulation coding features, and 6-dimensional current simulation physical parameter information to obtain a 54-dimensional fused feature. The intermediate fusion module uses a fully connected layer to uniformly represent the 54-dimensional fused feature, mapping it to a 64-dimensional intermediate fused feature. This intermediate fusion method allows the deviation information from the visual, force, and tactile modalities to jointly participate in the correction of the simulation physical parameters, avoiding the dominance of single-modal data in the calibration results.

[0040] 3. Parameter Correction Module Continuing with the aforementioned three modes as examples, the parameter correction module takes 64-dimensional fused intermediate features as input and outputs 6-dimensional simulation physical parameter corrections through three fully connected layers. The first fully connected layer maps the 64-dimensional fused intermediate features to 64-dimensional parameter correction features, the second fully connected layer maps the 64-dimensional parameter correction features to 32-dimensional parameter correction features, and the third fully connected layer maps the 32-dimensional parameter correction features to 6-dimensional simulation physical parameter corrections. The output 6-dimensional simulation physical parameter corrections correspond to the corrections for the mass of the manipulated object, the equivalent moment of inertia of the manipulated object, the contact stiffness, the contact damping, the static friction coefficient, and the dynamic friction coefficient, respectively.

[0041] The final layer of the parameter correction module is the output range limitation layer. For example, the hyperbolic tangent function tanh can be used to limit the output range, preventing excessively large single physical parameter corrections from causing simulation instability. Specifically, a preset range limitation formula can be used to calculate the final simulation physical parameter correction amount corresponding to the current simulation physical parameter information.

[0042] The preset range limitation formula includes: ; in, S represents the final simulation physical parameter correction amount corresponding to the current simulation physical parameter information. S is used to limit the maximum correction range of each simulation physical parameter in a single iteration, and Z represents the initial simulation physical parameter correction amount.

[0043] The limits for each parameter are different. For example, for mass and equivalent moment of inertia, the maximum correction is set to 15% of the current parameter value; for contact stiffness and contact damping, the maximum correction is set to 20% of the current parameter value; and for static friction coefficient and dynamic friction coefficient, the maximum correction is set to 15% of the current parameter value.

[0044] After obtaining the simulation physics parameter correction values, the current simulation physics parameter information can be corrected based on these correction values ​​and the minimum and maximum boundaries of each simulation physics parameter, thus obtaining the corrected simulation physics parameter information. The specific calculation formula can be: ; in, This represents the simulation physics parameter information used in the (r+1)th iteration, i.e., the corrected simulation physics parameter information. This represents the simulation physics parameter information used in the r-th iteration, i.e., the current simulation physics parameter information. This indicates the final simulation physics parameter correction amount corresponding to the current simulation physics parameter information. and These represent the minimum and maximum boundaries of each simulation physics parameter, respectively. These parameter information can all be in multi-dimensional vector form, with each dimension representing a parameter. This represents the boundary truncation function, used to ensure that the updated physical parameters are within a physically reasonable range.

[0045] It should be added that the preset correction algorithm can also include optimization methods such as Bayesian optimization, evolutionary algorithms, and particle swarm optimization to directly search for simulation physics parameters. This type of alternative method uses the deviation of multimodal physical response as the evaluation criterion for the physical consistency between the simulation scene and the real scene, and obtains the final calibration parameters through iteration. It can also serve as an alternative to the overall implementation path of this scheme.

[0046] S140: If the current physical simulation error is greater than the preset error threshold, the corrected simulation physical parameter information is used as the new current simulation physical parameter information, and the process returns to step S110 until the current physical simulation error is less than or equal to the preset error threshold, or the target number of iterations is reached, at which point the most recently obtained corrected simulation physical parameter information is used as the target simulation physical parameter information.

[0047] If the current physical simulation error exceeds a preset error threshold, the corrected simulation physical parameter information is used as the new current simulation physical parameter information. The process returns to steps S110-S130, iteratively updating the current simulation physical parameter information to recalculate the current physical simulation error between the current multimodal simulation response and the actual multimodal response. This continues until the current physical simulation error is less than or equal to the preset error threshold. At this point, the most recently obtained corrected simulation physical parameter information is used as the target simulation physical parameter information. The preset error threshold can be an empirical value, for example, 8% of the initial simulation physical parameter information.

[0048] If the target number of iterations is reached, but the current physical simulation error exceeds a preset error threshold, then the corrected simulation physical parameter information obtained from the last iteration is directly used as the target simulation physical parameter information. The target simulation physical parameter information is the final required simulation physical parameter information. The target number of iterations can be an empirical value, such as 20 iterations.

[0049] The simulation scene physical parameter calibration method based on multimodal data provided in this application introduces the physical simulation error between the multimodal simulation response and the multimodal real response as feedback, and uses a preset correction algorithm combined with parameter boundary constraints to iteratively correct the simulation physical parameters. This can effectively distinguish the influence of multiple factors such as mass, friction, stiffness, and damping on contact operation tasks, avoiding the parameter identification ambiguity caused by the single information in traditional single-modal calibration methods. At the same time, by iterating repeatedly until the physical simulation error converges to within the threshold or reaches the target number of iterations, the final target simulation physical parameters can more accurately reflect the physical characteristics of the real scene, thereby significantly reducing the dynamic deviation between the simulation and the real environment, improving the effectiveness of the calibrated simulation scene in supporting embodied intelligent tasks (such as contact and operation), and reducing the secondary calibration cost when migrating real equipment.

[0050] Furthermore, this embodiment of the application constructs a multimodal physical parameter deviation network including a multimodal deviation coding module, an intermediate fusion module, and a parameter correction module. First, the multimodal deviation coding module encodes the current multimodal simulation response to obtain the current multimodal simulation coding features. Then, the intermediate fusion module fuses the current multimodal simulation coding features with the current simulation physical parameter information, and after passing through a fully connected layer, obtains the current multimodal fusion intermediate features. Finally, the parameter correction module corrects and limits the output of the current multimodal fusion intermediate features to obtain the final simulation physical parameter correction amount. Instead of simply splicing or processing the multimodal data separately, this approach further solves the technical problem of the difficulty in uniformly modeling different modal data and ultimately achieves the effect of enabling multimodal deviation information to participate in the simulation physical parameter correction.

[0051] In one possible implementation, the training loss function of the multimodal physical parameter deviation network includes parameter correction loss and physical simulation error loss. The parameter correction loss is used to constrain the amount of simulation physical parameter correction output by the multimodal physical parameter deviation network to be close to the training label. The physical simulation error loss is used to constrain the multimodal simulation response obtained based on the corrected simulation physical parameter information to be close to the multimodal real response. The training label is the difference between the simulation physical parameter information used in each training session and the preferred physical parameter information.

[0052] The method for determining the preferred physical parameter information includes: perturbing each initial simulation physical parameter in the initial simulation physical parameter information of the simulation scene to generate multiple candidate simulation physical parameter information, for example, the candidate simulation physical parameters are sampled within ±20% of the initial simulation physical parameters; writing each candidate simulation physical parameter information into the simulation scene, executing the target contact operation task, and collecting multimodal simulation responses; calculating the physical simulation error between each multimodal simulation response and the multimodal real response; and selecting the candidate simulation physical parameter information with the smallest physical simulation error as the preferred physical parameter information.

[0053] The training loss function can be a weighted sum of parameter correction loss and physical simulation error loss. Parameter correction loss can be the mean square error or mean absolute error between the simulated physical parameter correction amount output by the multimodal physical parameter bias network and the training labels; physical simulation error loss can be the mean square error between the multimodal simulated response obtained based on the corrected simulated physical parameter information and the multimodal real response.

[0054] For example, the multimodal physical parameter bias network can be implemented in the PyTorch deep learning framework and trained on an NVIDIA V100 GPU with 16GB of VRAM. The learning rate is set to 1e-4, the weight decay is set to 1e-5, the batch size is set to 64, and the maximum number of training epochs is set to 100.

[0055] In this embodiment, when training a multimodal physical parameter deviation network, the parameter correction loss and the physical simulation error loss are jointly optimized. This ensures the network is simultaneously subjected to two complementary constraints during training: First, the parameter correction loss directly supervises the network output correction amount to approximate the training label composed of the difference between the simulation parameters and the optimized parameters, guaranteeing clear prior guidance for the direction and magnitude of parameter correction. Second, the physical simulation error loss indirectly supervises the approximation between the multimodal simulation response generated by the corrected parameters and the real multimodal response, forcing the network to learn the physical parameter adjustment rules that truly reduce the dynamic differences between simulation and reality. This dual-loss collaborative training mechanism effectively overcomes the limitations of relying solely on parameter labels or response errors. It enables the network not only to fit known optimized parameter samples but also to output reasonable parameter correction amounts in unseen scenarios through the generalization characteristics of physical simulation errors, thereby significantly improving the accuracy and robustness of simulation physical parameter calibration.

[0056] Based on the above method embodiments, another embodiment of this application provides a simulation scene physical parameter calibration device based on multimodal data, such as... Figure 3 As shown, the device includes: The simulation unit 210 is used to write the current simulation physical parameter information into the simulation scene of the embodied intelligent device, execute the target contact operation task, and collect the current multimodal simulation response. The target contact operation task is the same contact operation task as the real scene. The calculation unit 220 is used to calculate the current physical simulation error between the current multimodal simulation response and the multimodal real response; The correction amount determination unit 230 is used to process the current simulation physical parameter information and the current multimodal simulation response based on a preset correction algorithm to obtain the simulation physical parameter correction amount corresponding to the current simulation physical parameter information. The correction unit 240 is used to correct the current simulation physical parameter information according to the correction amount of the simulation physical parameters, the minimum boundary and the maximum boundary of each simulation physical parameter, so as to obtain the corrected simulation physical parameter information. The setting unit 250 is also used to, when the current physical simulation error is greater than a preset error threshold, use the corrected simulation physical parameter information as the new current simulation physical parameter information and return to the execution unit; The parameter determination unit 260 is used to take the most recently obtained corrected simulation physical parameter information as the target simulation physical parameter information until the current physical simulation error is less than or equal to the preset error threshold, or until the target number of iterations is reached.

[0057] In one possible implementation, the preset correction algorithm includes a multimodal physical parameter deviation network, which includes a multimodal deviation encoding module, an intermediate fusion module, and a parameter correction module. The correction amount determination unit 230 includes: The multimodal deviation coding module is used to encode the current multimodal simulation response using its own two fully connected layers to obtain the current multimodal simulation coding features; The intermediate fusion module is used to fuse the current multimodal simulation coding features with the current simulation physical parameter information to obtain the current multimodal simulation fusion features, and to use the fully connected layer in the intermediate fusion module to map the current multimodal simulation fusion features into the current multimodal fusion intermediate features. The parameter correction module is used to correct the current multimodal fusion intermediate features using its own three fully connected layers to obtain the initial simulation physical parameter correction amount corresponding to the current simulation physical parameter information; and to limit the range of the initial simulation physical parameter correction amount using the output range limiting layer to output the final simulation physical parameter correction amount.

[0058] In one possible implementation, the parameter correction module is used to calculate the final simulation physical parameter correction amount corresponding to the current simulation physical parameter information using a preset range limitation formula. The preset range limitation formula includes: ; Among them, the S represents the final simulation physical parameter correction amount corresponding to the current simulation physical parameter information. S is used to limit the maximum correction range of each simulation physical parameter in a single iteration. Z represents the initial simulation physical parameter correction amount.

[0059] In one possible implementation, the training loss function of the multimodal physical parameter deviation network includes a parameter correction loss and a physical simulation error loss. The parameter correction loss is used to constrain the amount of simulation physical parameter correction output by the multimodal physical parameter deviation network to be close to the training label, and the physical simulation error loss is used to constrain the multimodal simulation response obtained based on the corrected simulation physical parameter information to be close to the multimodal real response. The training label is the difference between the simulated physical parameter information and the preferred physical parameter information used in each training session; The parameter determination unit 260 is further configured to perturb each initial simulation physical parameter in the initial simulation physical parameter information of the simulation scene to generate multiple candidate simulation physical parameter information; write each candidate simulation physical parameter information into the simulation scene, execute the target contact operation task, and collect multimodal simulation responses; calculate the physical simulation error between each multimodal simulation response and the multimodal real response; and select the candidate simulation physical parameter information with the smallest physical simulation error as the preferred physical parameter information.

[0060] In one possible implementation, the simulated physical parameter information includes at least one of the following: mass of the manipulated object, equivalent moment of inertia of the manipulated object, contact stiffness, contact damping, static friction coefficient, and dynamic friction coefficient; and / or, The current multimodal simulation response includes at least two of the following: current simulation visual response, current simulation force response, current simulation tactile response, current simulation motion trajectory response, current simulation joint state response, current simulation sound data response, and current simulation workpiece deformation response; correspondingly, the multimodal real response includes at least two of the following: real visual response, real force response, real tactile response, real motion trajectory response, real joint state response, real sound data response, and real workpiece deformation response.

[0061] The simulation scene physical parameter calibration device based on multimodal data provided in this application introduces the physical simulation error between the multimodal simulation response and the multimodal real response as feedback, and uses a preset correction algorithm combined with parameter boundary constraints to iteratively correct the simulation physical parameters. It can effectively distinguish the influence of multiple factors such as mass, friction, stiffness, and damping on contact operation tasks, and avoid the parameter identification ambiguity caused by the single information in traditional single-modal calibration methods. At the same time, by iterating repeatedly until the physical simulation error converges to within the threshold or reaches the target number of iterations, the final target simulation physical parameters can more accurately reflect the physical characteristics of the real scene, thereby significantly reducing the dynamic deviation between the simulation and the real environment, improving the effectiveness of the calibrated simulation scene in supporting embodied intelligent tasks (such as contact and operation), and reducing the secondary calibration cost when migrating real equipment.

[0062] Based on the above method embodiments, another embodiment of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any of the above embodiments.

[0063] Based on the above method embodiments, another embodiment of this application provides an electronic device or computer device, including: One or more processors; The processor is coupled to a storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the electronic device or computer device performs the method as described in any of the above embodiments.

[0064] Based on the above embodiments, another embodiment of this application provides a computer program product, which includes instructions that, when executed on a computer or processor, cause the computer or processor to perform the method described in any of the above embodiments.

[0065] The above-described apparatus embodiments correspond to the method embodiments and have the same technical effects. For detailed descriptions, please refer to the method embodiments. The apparatus embodiments are derived from the method embodiments; detailed descriptions can be found in the method embodiments section, and will not be repeated here. Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of one embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing this application.

[0066] Those skilled in the art will understand that the modules in the apparatus of the embodiments can be distributed in the apparatus of the embodiments as described in the embodiments, or they can be located in one or more devices different from this embodiment with corresponding changes. The modules of the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0067] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for calibrating physical parameters of a simulation scene based on multimodal data, characterized in that, The method includes: The current simulated physical parameter information is written into the simulation scene of the embodied intelligent device, the target contact operation task is executed, and the current multimodal simulation response is collected. The target contact operation task is the same as the contact operation task in the real scene. Calculate the current physical simulation error between the current multimodal simulation response and the actual multimodal response; Based on a preset correction algorithm, the current simulation physical parameter information and the current multimodal simulation response are processed to obtain the simulation physical parameter correction amount corresponding to the current simulation physical parameter information. Then, based on the simulation physical parameter correction amount and the minimum and maximum boundaries of each simulation physical parameter, the current simulation physical parameter information is corrected to obtain the corrected simulation physical parameter information. If the current physical simulation error is greater than a preset error threshold, the corrected simulation physical parameter information is used as the new current simulation physical parameter information. The process returns to write the current simulation physical parameter information into the simulation scene of the embodied intelligent device and execute the target contact operation task until the current physical simulation error is less than or equal to the preset error threshold, or the target number of iterations is reached. In this case, the most recently obtained corrected simulation physical parameter information is used as the target simulation physical parameter information.

2. The method according to claim 1, characterized in that, The preset correction algorithm includes a multimodal physical parameter deviation network, which includes a multimodal deviation encoding module, an intermediate fusion module, and a parameter correction module. Based on a preset correction algorithm, the current simulation physical parameter information and the current multimodal simulation response are processed to obtain the simulation physical parameter correction amount corresponding to the current simulation physical parameter information, including: The current multimodal simulation response is encoded using the two fully connected layers of the multimodal deviation coding module to obtain the current multimodal simulation coding features; The intermediate fusion module is used to fuse the current multimodal simulation coding features with the current simulation physical parameter information to obtain the current multimodal simulation fusion features. The fully connected layer in the intermediate fusion module is used to map the current multimodal simulation fusion features into the current multimodal fusion intermediate features. The three fully connected layers of the parameter correction module are used to correct the current multimodal fusion intermediate features to obtain the initial simulation physical parameter correction amount corresponding to the current simulation physical parameter information; The output range limiting layer of the parameter correction module is used to limit the range of the initial simulation physical parameter correction amount, and the final simulation physical parameter correction amount is output.

3. The method according to claim 2, characterized in that, The output range limiting layer of the parameter correction module is used to limit the range of the initial simulation physical parameter correction amount, and the final simulation physical parameter correction amount is output, including: Using a preset range limitation formula, calculate the final simulation physical parameter correction amount corresponding to the current simulation physical parameter information; The preset range limitation formula includes: ； Among them, the S represents the final simulation physical parameter correction amount corresponding to the current simulation physical parameter information. S is used to limit the maximum correction range of each simulation physical parameter in a single iteration. Z represents the initial simulation physical parameter correction amount.

4. The method according to claim 1, characterized in that, The training loss function of the multimodal physical parameter deviation network includes parameter correction loss and physical simulation error loss. The parameter correction loss is used to constrain the amount of simulation physical parameter correction output by the multimodal physical parameter deviation network to be close to the training label. The physical simulation error loss is used to constrain the multimodal simulation response obtained based on the corrected simulation physical parameter information to be close to the multimodal real response. The training label is the difference between the simulated physical parameter information and the preferred physical parameter information used in each training session; The method for determining the preferred physical parameter information includes: Each initial simulation physical parameter in the initial simulation physical parameter information of the simulation scene is perturbed to generate multiple candidate simulation physical parameter information; Each candidate simulation physical parameter information is written into the simulation scene, the target contact operation task is executed, and multimodal simulation response is collected; Calculate the physical simulation error between each multimodal simulation response and the actual multimodal response; The candidate simulation physical parameter information with the smallest physical simulation error is selected as the preferred physical parameter information.

5. The method according to any one of claims 1-4, characterized in that, The simulated physical parameter information includes at least one of the following: mass of the manipulated object, equivalent moment of inertia of the manipulated object, contact stiffness, contact damping, static friction coefficient, and dynamic friction coefficient. And / or, The current multimodal simulation response includes at least two of the following: current simulation visual response, current simulation force response, current simulation tactile response, current simulation motion trajectory response, current simulation joint state response, current simulation sound data response, and current simulation workpiece deformation response; correspondingly, the multimodal real response includes at least two of the following: real visual response, real force response, real tactile response, real motion trajectory response, real joint state response, real sound data response, and real workpiece deformation response.

6. A device for calibrating physical parameters of a simulation scene based on multimodal data, characterized in that, The device includes: The simulation unit is used to write the current simulation physical parameter information into the simulation scene of the embodied intelligent device, execute the target contact operation task, and collect the current multimodal simulation response. The target contact operation task is the same as the contact operation task in the real scene. The calculation unit is used to calculate the current physical simulation error between the current multimodal simulation response and the actual multimodal response; The correction amount determination unit is used to process the current simulation physical parameter information and the current multimodal simulation response based on a preset correction algorithm to obtain the simulation physical parameter correction amount corresponding to the current simulation physical parameter information; The correction unit is used to correct the current simulation physical parameter information according to the correction amount of the simulation physical parameters, the minimum boundary and the maximum boundary of each simulation physical parameter, so as to obtain the corrected simulation physical parameter information. The setting unit is also used to, when the current physical simulation error is greater than a preset error threshold, take the corrected simulation physical parameter information as the new current simulation physical parameter information and return to the execution unit; The parameter determination unit is used to take the most recently obtained corrected simulation physical parameter information as the target simulation physical parameter information until the current physical simulation error is less than or equal to the preset error threshold, or until the target number of iterations is reached.

7. The apparatus according to claim 6, characterized in that, The preset correction algorithm includes a multimodal physical parameter deviation network, which includes a multimodal deviation encoding module, an intermediate fusion module, and a parameter correction module. The correction amount determination unit includes: The multimodal deviation coding module is used to encode the current multimodal simulation response using its own two fully connected layers to obtain the current multimodal simulation coding features; The intermediate fusion module is used to fuse the current multimodal simulation coding features with the current simulation physical parameter information to obtain the current multimodal simulation fusion features, and to use the fully connected layer in the intermediate fusion module to map the current multimodal simulation fusion features into the current multimodal fusion intermediate features. The parameter correction module is used to correct the current multimodal fusion intermediate features using its own three fully connected layers to obtain the initial simulation physical parameter correction amount corresponding to the current simulation physical parameter information; and to limit the range of the initial simulation physical parameter correction amount using the output range limiting layer to output the final simulation physical parameter correction amount.

8. The apparatus according to claim 7, characterized in that, The parameter correction module is used to calculate the final simulation physical parameter correction amount corresponding to the current simulation physical parameter information using a preset range limitation formula. The preset range limitation formula includes: ； Among them, the S represents the final simulation physical parameter correction amount corresponding to the current simulation physical parameter information. S is used to limit the maximum correction range of each simulation physical parameter in a single iteration. Z represents the initial simulation physical parameter correction amount.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-5.

10. An electronic device, characterized in that, The electronic device includes: One or more processors; The processor is coupled to a storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the electronic device performs the method as described in any one of claims 1-5.

Images