Simulation model construction method and device, electronic equipment and readable storage medium

Abstract

The invention provides a simulation model construction method and device, electronic equipment and a computer readable storage medium, and relates to the technical field of robotics.The method comprises the steps that first action data of a first object is obtained, and the first object is a physical entity in the real world; according to the first action data and a digital twinborn model, obtaining a driving parameter of a second object, the second object being a digital twinborn body of the first object; driving and controlling the second object to move in the virtual simulation environment based on the driving parameter to obtain second action data of the second object; and adjusting the digital twin model according to the first action data and the second action data to obtain a simulation control model of the robot. The control model optimization of the robot is completed in the virtual simulation environment, the reasonability and learning efficiency of the posture action of the robot can be improved, the overall working cost in the control model optimization process can be reduced to a certain degree, and the control model optimization method has remarkable economical efficiency and practicability.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robots, and more particularly, to a simulation model construction method and device, electronic equipment and computer readable storage medium in the technical field of robots. BACKGROUND

[0002] Currently, when optimizing a control model for controlling a robot, a real robot is generally relied on, and the robot is driven to perform complex actions and interactive tasks through collected user action data, so as to gradually complete iterative optimization of the control model.

[0003] However, the real robot generally has problems such as complex mechanical structure, high freedom of movement, and many vulnerable parts, which limits its operation reliability and increases maintenance costs. In the process of imitative learning, frequent trial and error operations may cause hardware failures, such as joint motor overload, transmission mechanism wear, and sensor failure. Once a failure occurs, not only does it need to be repaired, but it may also cause the model optimization process to be forced to stop and training data to be lost, thereby significantly prolonging the optimization cycle of the control model and reducing the overall learning efficiency. SUMMARY

[0004] Embodiments of the present application provide a simulation model construction method, device, electronic equipment and computer readable storage medium, which realize the optimization of the control model of the robot in a virtual simulation environment, can not only improve the rationality of the posture action of the robot and the learning efficiency, but also can reduce the overall working cost in the process of optimizing the control model to a certain extent, and has significant economy and practicality.

[0005] In a first aspect, a simulation model construction method is provided, applied to a robot simulation system, and the method includes: obtaining first action data of a first object, the first object being a physical entity in a real world; obtaining driving parameters of a second object according to the first action data and a digital twin model, the second object being a digital twin of the first object; driving the second object to move in a virtual simulation environment based on the driving parameters to obtain second action data of the second object; and adjusting the digital twin model according to the first action data and the second action data to obtain a simulation control model of a robot.

[0006] In a possible implementation manner, the first object has a plurality of movable joints, and the obtaining of the first action data of the first object includes: obtaining the first action data of the first object through a wearable device; the wearable device includes a virtual reality head-mounted display device, a posture sensor device, and a data glove device; the first action data includes position information, speed information, and movable joint information of the first object moving in the real world, and the movable joint information includes angle information and force information of the movable joints.

[0007] In a possible implementation, the driving parameter of the second object is obtained according to the first action data and the digital twin model, including: pre-processing the first action data by the digital twin model to obtain pre-processed data, the pre-processing including at least one of data checking, data filtering, and data format conversion; converting the pre-processed data from a first coordinate system in which the first object is located to a second coordinate system in which the second object is located based on an initial conversion matrix in the digital twin model to obtain conversion data, the initial conversion matrix being a coordinate transformation matrix between the first coordinate system and the second coordinate system; and generating the driving parameter according to the conversion data.

[0008] In a possible implementation, the driving parameter is generated according to the conversion data, including: obtaining a target action trajectory of the second object according to position coordinates of the first object at different time points in the conversion data, the target action trajectory including a target action direction; obtaining a target posture of the second object at different time points according to angles of each movable joint of the first object at different time points in the conversion data; obtaining a target action strength of the second object at different time points according to strengths of each movable joint of the first object at different time points in the conversion data; determining a target action speed of the second object at different time points according to displacements between position coordinates of the first object at adjacent time points in the conversion data and corresponding time intervals; and obtaining the driving parameter according to the target action trajectory, the target posture, the target action strength, and the target action speed.

[0009] In a possible implementation, the simulation control model of the robot is obtained by adjusting the digital twin model according to the first action data and the second action data, including: determining a deviation between the first action data and the second action data; and adjusting an initial conversion matrix in the digital twin model according to the deviation to obtain the simulation control model.

[0010] In a possible implementation, the simulation control model is obtained by adjusting the initial conversion matrix in the digital twin model according to the deviation, including: in a case where the deviation is located in a preset deviation interval, determining the current digital twin model as the simulation control model.

[0011] In a possible implementation, the adjusting the initial transformation matrix in the digital twin model according to the deviation to obtain the simulation control model includes: in a case where the deviation is located outside a preset deviation interval, adjusting the initial transformation matrix in the digital twin model according to the deviation to obtain a target transformation matrix; taking the target transformation matrix as the initial transformation matrix; obtaining first action data of the first object; pre-processing the first action data by using the digital twin model to obtain pre-processed data; converting the pre-processed data from a first coordinate system in which the first object is located to a second coordinate system in which the second object is located based on the initial transformation matrix to obtain converted data; generating a driving parameter according to the converted data; driving the second object to move in the virtual simulation environment based on the driving parameter to obtain second action data of the second object; and determining the deviation between the first action data and the second action data until the deviation is located in the preset deviation interval, to obtain the simulation control model.

[0012] In a possible implementation, the initial transformation matrix includes an initial rotation matrix and an initial translation vector, and the adjusting the initial transformation matrix in the digital twin model according to the deviation to obtain the target transformation matrix includes: constructing a deviation function of the initial transformation matrix based on the deviation, unknowns in the deviation function including a target rotation matrix and a target translation vector; substituting position coordinates in the first action data and the second action data into the deviation function, and solving the deviation function to obtain the target rotation matrix and the target translation vector; taking the target rotation matrix as the initial rotation matrix, and taking the target translation vector as the initial translation vector to obtain the target transformation matrix.

[0013] In a possible implementation, the determining the deviation between the first action data and the second action data includes: determining, according to a same movable joint corresponding to the first object and the second object, a difference value between a first position coordinate and a second position coordinate of the same movable joint at different time points, to obtain a plurality of difference values, wherein the first position coordinate is a position coordinate of the same movable joint in the first action data, and the second position coordinate is a position coordinate of the same movable joint in the second action data; and determining an average value of the plurality of difference values to obtain the deviation.

[0014] In a second aspect, a simulation model construction apparatus is provided, applied to a robot simulation system, and includes: a data obtaining module, configured to obtain first action data of a first object, the first object being a physical entity in a real world; a parameter generating module, configured to obtain a driving parameter of a second object according to the first action data and a digital twin model, the second object being a digital twin of the first object; an action driving module, configured to drive and control the second object to move in a virtual simulation environment based on the driving parameter, to obtain second action data of the second object; The model adjusting module is configured to adjust the digital twin model according to the first action data and the second action data, to obtain the simulation control model of the robot.

[0015] In a third aspect, an electronic device is provided, including a memory and a processor. The memory is configured to store executable program code, and the processor is configured to invoke and run the executable program code from the memory, so that the electronic device executes the simulation model construction method in the first aspect or any possible implementation manner of the first aspect.

[0016] In a fourth aspect, a computer program product is provided, which includes computer program code. When the computer program code is run on a computer, the computer program code causes the computer to execute the simulation model construction method in the first aspect or any possible implementation manner of the first aspect.

[0017] In a fifth aspect, a computer-readable storage medium is provided, which stores computer program code. When the computer program code is run on a computer, the computer program code causes the computer to execute the simulation model construction method in the first aspect or any possible implementation manner of the first aspect.

[0018] The technical solution of the present application includes the following beneficial effects: The present application collects first action data of a first object in a real world, and then generates driving parameters for driving a second object based on the first action data by a digital twin model, and drives the second object to restore the action of the first object based on the driving parameters, and adjusts the digital twin model according to second action data during the action of the second object and the first action data, thereby obtaining a simulation control model of a robot. Since the second action data participating in the adjustment of the digital twin model comes from the second object, and the second object is a virtual object based on a virtual simulation environment, when generating the simulation control model of the robot, there is no need to actually manufacture a robot entity, and there is no need to bear high physical manufacturing costs and maintenance costs. Therefore, compared with the traditional method of relying on a robot entity for model optimization, the simulation model construction method provided by the embodiments of the present application can significantly reduce the optimization cost of the simulation control model of the robot. When the method is applied to the control model optimization process of the robot, there is no need to specially design and manufacture a real robot entity, and the control model optimization of the robot can be completed in a virtual simulation environment, thereby reducing the human, material and time costs in the model optimization process, and having significant economic and practicality. Since the present application does not need to rely on a real robot entity in the process of optimizing the control model of the robot, the problem of model optimization interruption caused by robot hardware failure or maintenance demand is avoided, thereby effectively shortening the model optimization period. Therefore, the present application not only improves the rationality of the robot posture action, but also improves the learning efficiency of the robot behavior learning. Attached Figure Description

[0019] Figure 1 A schematic flowchart of a simulation model construction method provided in an embodiment of this application is shown; Figure 2 An architecture diagram of the robot simulation system provided in an embodiment of this application is shown; Figure 3 A schematic diagram illustrating the actions of the second object in replicating those of the first object is shown. Figure 4 An exemplary hardware connection architecture diagram for collecting first action data provided in an embodiment of this application is shown; Figure 5 This paper illustrates another exemplary hardware connection architecture diagram for collecting first action data provided in an embodiment of this application; Figure 6 This paper shows another schematic flowchart of a simulation model construction method provided in an embodiment of the present application; Figure 7 This paper shows a schematic diagram of the structure of a simulation model building device provided in an embodiment of this application; Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0020] The technical solutions in this application will be clearly and thoroughly described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. "And / or" in the text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.

[0021] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

[0022] Currently, when optimizing the control model used to control robots, it is usually necessary to rely on real robots and drive them to perform complex actions and interactive tasks by collecting user action data, thereby gradually completing the iterative optimization of the control model.

[0023] However, real robots generally have problems such as complex mechanical structure, high degree of freedom of movement, and many vulnerable parts, which limit the operation reliability and increase the maintenance cost. During the imitation learning process, frequent trial-and-error operations may cause hardware failures, such as joint motor overload, transmission mechanism wear, sensor failure, and the like. Once a failure occurs, not only is maintenance required, but the model optimization process may also be forced to stop and training data may be lost, thereby significantly prolonging the optimization period of the control model and reducing the overall learning efficiency.

[0024] Based on the above problems, the embodiment of the present application provides a simulation model construction method, device, electronic equipment and computer readable storage medium. The present application constructs a digital twin corresponding to a physical entity in the real world in a virtual simulation environment, acquires first action data of the physical entity in the real world, and then generates driving parameters for driving the digital twin based on the first action data by the digital twin model, and drives the digital twin to restore the action of the physical entity based on the driving parameters, and adjusts the digital twin model according to the second action data during the action of the digital twin and the first action data, until the restoration degree of the action of the physical entity by the digital twin reaches the set expected value, and stops adjusting the digital twin model, thereby obtaining a simulation control model of the robot, and then controlling the robot based on the simulation control model. Since the second action data participating in the adjustment of the digital twin model comes from the digital twin corresponding to the physical entity, and the digital twin is a virtual object based on the virtual simulation environment, when generating the simulation control model of the robot, there is no need to actually manufacture the robot entity, and there is no need to bear high physical manufacturing cost and maintenance cost. Therefore, compared with the traditional method of relying on the robot entity to optimize the model, on the one hand, the simulation control model of the robot can be optimized at a significantly reduced cost, and on the other hand, since the present application does not need to rely on the real robot entity to optimize the control model, the problem of model optimization interruption caused by robot hardware failure or maintenance demand is avoided, thereby effectively shortening the model optimization period. Therefore, the present application not only improves the rationality of the robot posture action, but also improves the learning efficiency of the robot behavior learning.

[0025] The following is an embodiment of a simulation model construction method provided in the present application.

[0026] Figure 1 A schematic flowchart of a simulation model construction method provided by the embodiment of the present application is shown. As shown in Figure 1 The simulation model construction method provided by the embodiment of the present application is applied to a robot simulation system, and the robot simulation system is mounted on an electronic device with certain computing power and display function, such as a computer.

[0027] The simulation model construction method provided in this application includes the following schemes: S110: Obtain the first action data of the first object, where the first object is a physical entity in the real world.

[0028] In an exemplary embodiment, the first object is a physical entity in the real world. The first object can be a physical entity with living characteristics, such as a human or an animal, or a physical entity without living characteristics, such as a bionic animal toy or a doll.

[0029] When the first object takes action, the action data of the first object is acquired to obtain the first action data. This first action data can be generated by the first object's own active movement, or it can be generated after the first object is manually assisted in performing the action. In other words, the first action data can be generated by the first object's autonomous action, or it can be generated through manual manipulation or guidance of the first object's action.

[0030] S120: Based on the first action data and the digital twin model, obtain the driving parameters of the second object, which is the digital twin of the first object.

[0031] like Figure 2 As shown, Figure 2 An architecture diagram of the robot simulation system provided in this application embodiment is shown. The robot simulation system runs on an electronic device and includes a digital twin model, a physics engine, and a visualization module. The second object is a digital twin of the first object created in a virtual simulation environment. For example, the first object can be a human or an animal, such as a dog or a horse, and the second object is a virtual robot. The visualization module is used to visualize the second object. The digital twin model is pre-set in the robot simulation system and is a model used to generate the driving parameters of the digital twin. The physics engine is used to simulate the real world and control the second object to simulate the actions and behaviors of the first object. For example, if the second object is a virtual robot, the physics engine can control the movements of the virtual robot's head, eyes, arms, hands, legs, feet, etc., to reproduce the actions and behaviors of the first object.

[0032] After acquiring the first action data of the first object, the first action data is input into the digital twin model. Based on the first action data, the digital twin model plans the driving parameters for driving the second object, so that the second object can reproduce the action behavior of the first object. The driving parameters include the target action trajectory, target posture, target action force, and target action speed of the second object.

[0033] S130: Drive the second object to move in the virtual simulation environment based on the driving parameters, and obtain the second motion data of the second object.

[0034] The virtual simulation environment simulates the actual environment of the first object in the real world, for example, the first object stands straight against a wall in the real world, and the second object also stands against the corresponding virtual wall in the same posture in the virtual simulation environment. In some embodiments, the virtual environment is a simulation scene of a robot operation, such as different operation scenes of outdoor, home, factory, etc.

[0035] After obtaining the driving parameters of the second object, the digital twin model inputs the driving parameters of the second object into the physics engine, and the physics engine controls the movement of the second object in the virtual simulation environment according to the driving parameters to restore the action behavior of the first object. As shown in Figure 3 Figure 3 A schematic diagram showing that the second object restores the action behavior of the first object is shown. Wherein 100 represents the first object, specifically a dog, 200 represents the display screen of the electronic device, and 300 represents the second object, specifically a virtual dog robot. Among them, in the process of the movement of the second object, the action data of the second object is obtained, and the second action data is obtained.

[0036] S140: Adjust the digital twin model according to the first action data and the second action data to obtain a simulation control model of the robot.

[0037] After obtaining the second action data, it is judged whether the restoration degree of the action behavior of the first object by the second object reaches the expected value according to the first action data and the second action data. If the restoration degree reaches the expected value, the digital twin model is taken as the simulation control model of the robot; if the restoration degree does not reach the expected value, the digital twin model is adjusted once, and S110-S140 are continuously executed until the restoration degree reaches the expected value, and then the adjustment of the digital twin model is stopped, and the stopped digital twin model is taken as the simulation control model of the robot.

[0038] ​This application collects first motion data of a first object in the real world, then uses a digital twin model to generate driving parameters for a second object based on this first motion data. The second object is then driven by these driving parameters to recreate the actions of the first object. The digital twin model is adjusted based on the second motion data during the second object's actions and the first motion data, thereby obtaining a robot simulation control model. Since the second motion data used to adjust the digital twin model comes from the second object, and the second object is a virtual object based on a virtual simulation environment, there is no need to actually manufacture a physical robot when generating the robot simulation control model, nor to incur high physical manufacturing and maintenance costs. Therefore, compared to traditional methods that rely on physical robot entities for model optimization, the simulation model construction method provided in this application significantly reduces the optimization cost of the robot's simulation control model. Applying this method to the optimization process of the robot's control model eliminates the need for specially designed and manufactured physical robot entities; the robot's control model optimization can be completed in a virtual simulation environment, reducing the manpower, material resources, and time costs in the model optimization process, demonstrating significant economic efficiency and practicality. Since this application does not rely on a real robot entity during the optimization of the robot's control model, it avoids the problem of model optimization interruption due to robot hardware failure or maintenance needs, thereby effectively shortening the model optimization cycle. Therefore, this application not only improves the rationality of the robot's posture and movements but also enhances the learning efficiency of the robot's behavior learning.

[0039] In one possible implementation, the first object has multiple movable joints, including shoulder joints, elbow joints, wrist joints, hip joints, knee joints, ankle joints, etc. With multiple movable joints, the first object can perform multi-degree-of-freedom movements; that is, the first object can be a human, an animal (e.g., a horse), etc.

[0040] When the first object has multiple movable joints, the above-mentioned acquisition of the first motion data of the first object includes the following steps: like Figure 4 As shown, Figure 4 An exemplary hardware connection architecture diagram for acquiring first motion data provided in an embodiment of this application is shown. The wearable device includes a virtual reality head-mounted display, a posture sensor device, and a data glove device, and is communicatively connected to the electronic device. The wearable device acquires first motion data of a first object, sends the first motion data to the electronic device, the electronic device inputs the first motion data into a digital twin model, and the digital twin model outputs driving parameters for a second object.

[0041] It can be understood that the wearable device is worn on the first object in advance. For example, the first object is a human, and the virtual reality head-mounted display device is worn on the head of the human, the posture sensor device is worn on the shoulder joint, elbow joint, knee joint, etc. of the human, and the data glove device is worn on the hand of the human. After the human performs an action, the wearable device can obtain the action data of each part of the human, and then fuse the action data of each part to obtain the first action data. The first action data includes position information, speed information and movable joint information of the first object in the real world, and the movable joint information includes angle information and force information of the movable joint.

[0042] As shown in Figure 5 , Figure 5 Another exemplary hardware connection architecture diagram for collecting first action data provided by the embodiment of the application is shown, and the wearable device is in communication connection with the electronic device. The wearable device can further include a micro mechanical arm, and the micro mechanical arm includes a double-arm and double-hand part. In the case that the first object is a human, the arms and hands of the human are worn in the micro mechanical arm, and when the arms and hands of the human move, the micro mechanical arm also moves synchronously, and the action data of the arms and hands of the human is collected by the sensor built-in the micro mechanical arm, and the action data of the arms and hands of the human is fused to obtain the first action data. A micro mechanical arm can also be designed for animals other than humans according to actual needs, for example, the first object is a dog, and the micro mechanical arm includes forelimbs, hindlimbs and feet. The micro mechanical arm is worn on the dog, and the sensor built-in the micro mechanical arm collects the action data of the forelimbs, hindlimbs and feet of the dog, and the action data is fused to obtain the first action data. The micro mechanical arm sends the first action data to the electronic device, and the electronic device inputs the first action data into the digital twin model, and the digital twin model outputs the driving parameter of the second object.

[0043] In a possible implementation, in the case that the first object has a plurality of movable joints, the above-mentioned obtaining the driving parameter of the second object according to the first action data and the digital twin model includes the following steps: The first action data is preprocessed by the digital twin model to obtain preprocessed data, and the preprocessing includes at least one of data verification, data filtering and data format conversion; The preprocessed data is converted from a first coordinate system in which the first object is located to a second coordinate system in which the second object is located based on an initial conversion matrix in the digital twin model to obtain conversion data, and the initial conversion matrix is a coordinate transformation matrix between the first coordinate system and the second coordinate system; The driving parameter is generated according to the conversion data.

[0044] After obtaining the first action data, in order to improve the calculation efficiency of the driving parameter, the first action data is input into the digital twin model, and the digital twin model pre-processes the first action data, the pre-processing including at least one of data verification, data filtering and data format conversion. Among them, the data verification of the first action data is to verify the accuracy of the data, so as to ensure the accuracy of the driving parameter calculation; the data filtering of the first action data is to filter out noise interference, improve data quality and processing speed; the data format conversion of the first action data is to unify the data structure, simplify the processing logic and improve the overall data processing efficiency.

[0045] After the digital twin model pre-processes the first action data, pre-processed data is obtained. The digital twin model converts the pre-processed data from the first coordinate system in which the first object is located to the second coordinate system in which the second object is located based on an initial conversion matrix in the digital twin model, to obtain conversion data, and the initial conversion matrix is a coordinate transformation matrix between the first coordinate system and the second coordinate system. Specifically, the first coordinate system is represented as A, the second coordinate system is represented as B, and the initial conversion matrix is represented as M, that is, the coordinate transformation matrix between the first coordinate system and the second coordinate system. The initial conversion matrix M0 includes an initial rotation matrix R0 and an initial translation vector T0, that is, M0=[R0, T0]. Any one data in the pre-processed data is represented as D1, the data converted to the second coordinate system is represented as D2, then D2=R0xD1+T0, and based on the data conversion method, the converted pre-processed data can be obtained, that is, the conversion data corresponding to the pre-processed data in the second coordinate system is obtained. After the digital twin model converts the conversion data, the driving parameter for driving the second object is generated based on the conversion data.

[0046] In one possible implementation, the above generation of the driving parameter according to the conversion data includes the following steps: According to the position coordinates of the first object at different time points in the conversion data, a target action trajectory of the second object is obtained, and the target action trajectory includes a target action direction; According to the angles of each movable joint of the first object at different time points in the conversion data, a target posture of the second object at different time points is obtained; According to the force generated by each movable joint of the first object at different time points in the conversion data, a target action force of the second object at different time points is obtained; According to the displacement between the position coordinates of the first object at adjacent time points and the corresponding time interval, a target action speed of the second object at different time points is determined; According to the target action trajectory, the target posture, the target action force and the target action speed, the driving parameter is obtained.

[0047] The driving parameters include a target motion trajectory, a target posture, a target motion strength, and a target motion speed of the second object.

[0048] The target motion trajectory is obtained by: obtaining position coordinates of the first object at different time points in the conversion data to obtain a plurality of position coordinates; and connecting the plurality of position coordinates in chronological order to obtain the target motion trajectory of the second object, wherein the target motion trajectory of the second object includes a target motion direction of the second object, and the target motion direction includes a plurality of sub-directions, and each sub-direction is a vector direction from a position coordinate of a previous time point to a position coordinate of a next time point.

[0049] The target posture is obtained by: obtaining an initial angle of each movable joint of the second object in an initial posture; obtaining target angles of the same movable joint at different time points from the conversion data to obtain a plurality of target angles; calculating an angle difference between each target angle and the initial angle; correcting the initial angle according to the angle difference to obtain a corrected angle, corrected angle = initial angle + angle difference, thereby obtaining the corrected angles of the second object at different time points, and further obtaining the target posture of the second object at different time points based on the corrected angles of the second object at different time points.

[0050] The target motion strength is obtained by: obtaining the strength of each movable joint of the first object at different time points from the conversion data to obtain a plurality of strengths; and taking the strength generated at each time point as a target motion strength of the second object at a corresponding time point in chronological order, thereby obtaining the target motion strength of the second object at different time points.

[0051] The target motion speed is obtained by: obtaining a displacement between position coordinates of the first object at adjacent time points and a corresponding time interval from the conversion data; calculating a ratio of the displacement to the time interval to obtain the target motion speed of the second object at different time points, i.e., target motion speed = displacement / time interval.

[0052] After the digital twin model obtains the target motion trajectory, the target posture, the target motion strength, and the target motion speed, the driving parameters are obtained, and then the target motion trajectory, the target posture, and the target motion strength and the target motion speed are taken as control targets of the second object to control the motion of the second object in the virtual simulation environment to restore the motion behavior of the first object.

[0053] In one possible implementation, the method for adjusting the digital twin model according to the first motion data and the second motion data to obtain a simulation control model of the robot includes the following steps: determining a deviation between the first motion data and the second motion data; According to the deviation, the initial transformation matrix in the digital twin model is adjusted to obtain a simulation control model.

[0054] After obtaining the second action data, a deviation between the first action data and the second action data is calculated. The deviation between the first action data and the second action data is determined by determining a difference value between a first position coordinate and a second position coordinate of a same movable joint at different time instants according to the same movable joint corresponding to the first object and the second object, obtaining a plurality of difference values, and determining an average value of the plurality of difference values. The first position coordinate is a position coordinate of the same movable joint in the first action data, and the second position coordinate is a position coordinate of the same movable joint in the second action data. The first action data is used to generate driving parameters of the second object, and the first action data is obtained before the second action data, so the acquisition time of the first action data is earlier than the acquisition time of the second action data, that is, a time stamp corresponding to the first position coordinate is earlier than a time stamp corresponding to the second position coordinate.

[0055] The same movable joint corresponding to the first object and the second object is denoted as joint Gi, a position coordinate of the joint Gi in the first action data is obtained, and a first position coordinate of the joint Gi is obtained. The first position coordinate of the joint Gi at each time instant is denoted as , t represents a time instant, and the first position coordinate of the joint Gi at different time instants includes , and there are Y time instants. A position coordinate of the joint Gi in the second action data is obtained, and a second position coordinate of the joint Gi is obtained. The second position coordinate of the joint Gi at each time instant is denoted as , and the second position coordinate of the joint Gi at different time instants includes For each time instant t (t = 1, 2,..., Y), a difference value ΔX between the first position coordinate and the second position coordinate at each time instant t is calculated, , a sequence of a plurality of difference values is obtained, that is, {ΔX (1) , ΔX (2) ,..., ΔX (Y)}, and an average value of the deviations is taken to obtain the deviation between the first action data and the second action data, that is, .

[0056] After the deviation between the first action data and the second action data is obtained, it is determined whether the restoration degree of the action behavior of the second object to the first object reaches an expected value according to the deviation. If the restoration degree reaches the expected value, the digital twin model is taken as the simulation control model of the robot; if the restoration degree does not reach the expected value, the digital twin model is adjusted once, and steps S110-S140 are continuously executed until the restoration degree of the action behavior of the second object to the first object reaches the expected value according to the deviation, and then the adjustment of the digital twin model is stopped, and the digital twin model stopped from being adjusted is taken as the simulation control model of the robot.

[0057] In a possible implementation manner, the step of adjusting the initial conversion matrix in the digital twin model according to the deviation to obtain the simulation control model comprises the following steps: determining whether the deviation is located in a preset deviation interval; in a case where the deviation is located in the preset deviation interval, determining the current digital twin model as the simulation control model; in a case where the deviation is located outside the preset deviation interval (i.e., the deviation is not located in the preset deviation interval), adjusting the initial conversion matrix in the digital twin model according to the deviation to obtain a target conversion matrix; taking the target conversion matrix as the initial conversion matrix; obtaining the first action data of the first object; preprocessing the first action data by using the digital twin model to obtain preprocessed data; converting the preprocessed data from a first coordinate system in which the first object is located to a second coordinate system in which the second object is located based on the initial conversion matrix to obtain the conversion data; generating the driving parameter according to the conversion data; driving and controlling the second object to move in a virtual simulation environment based on the driving parameter to obtain the second action data of the second object; determining a deviation between the first action data and the second action data, and returning to execute the step of determining whether the deviation is located in the preset deviation interval until the deviation is located in the preset deviation interval, and the simulation control model is obtained.

[0058] After the deviation between the first action data and the second action data is obtained, whether the deviation is located in a preset deviation interval is judged to determine whether the restoration degree of the action behavior of the second object to the first object reaches an expected value. If the deviation is located in the preset deviation interval, it indicates that the restoration degree of the action behavior of the second object to the first object reaches the expected value, and the current digital twin model is determined as the simulation control model. If the deviation is not located in the preset deviation interval, it indicates that the restoration degree of the action behavior of the second object to the first object does not reach the expected value, and the initial transformation matrix in the digital twin model is adjusted based on the deviation to obtain a target transformation matrix, that is, the target transformation matrix is the adjusted initial transformation matrix. Then, the target transformation matrix is taken as a new initial transformation matrix, and the steps from the step of obtaining the first action data of the first object are executed again to obtain a newly obtained deviation, then the newly obtained deviation is used to update the last obtained deviation, and then whether the updated deviation is located in the preset deviation interval is judged. If the updated deviation is not located in the preset deviation interval, the steps from the step of obtaining the first action data of the first object are continuously executed to continuously adjust the initial transformation matrix. If the updated deviation is located in the preset deviation interval, the adjustment of the initial transformation matrix is stopped, and then the current digital twin model is determined as the simulation control model, so that the optimization of the simulation control model is realized, that is, the optimization of the simulation control model is completed.

[0059] In a possible implementation manner, the initial transformation matrix M0 includes an initial rotation matrix R0 and an initial translation vector T0, and the adjusting of the initial transformation matrix in the digital twin model according to the deviation to obtain a target transformation matrix includes the following scheme: a deviation function of the initial transformation matrix M0 is constructed based on the deviation, unknowns in the deviation function include a target rotation matrix R1 and a target translation vector T1; the position coordinates in the first action data and the second action data are substituted into the deviation function, and the deviation function is solved to obtain the target rotation matrix R1 and the target translation vector T1; the target rotation matrix R1 is taken as the initial rotation matrix R0, and the target translation vector T1 is taken as the initial translation vector T0 to obtain the target transformation matrix M1.

[0060] a data set of the first action data is represented as P s , a data set of the second action data is represented as P b , , i =1, 2,..., Y; and an error function is represented as E(R1, T1), so that: (1) wherein, represents and The error between them.

[0061] The dataset P of the first action data s And the dataset P of the second action data b Substitute the data into the above formula (1), and then use Singular Value Decomposition (SVD) to solve the error function of the substituted data, thereby obtaining the target rotation matrix R1 and the target translation vector T1. The specific solution process includes: 1. Calculate the dataset P s First center of mass C s And calculate the dataset P b Second center of mass C b ,Right now: (2) (3).

[0062] 2. For dataset P s and dataset P b Decentralization yields a decentralized dataset P. s and dataset P b , respectively, are represented as dataset P' s and P' b Decentralization refers to calculating the first average of the x-axis coordinates, the second average of the y-axis coordinates, and the third average of the z-axis coordinates in the dataset. For each data point, the first average is subtracted from the x-axis coordinates, the second average is subtracted from the y-axis coordinates, and the third average is subtracted from the z-axis coordinates, thus achieving data decentralization. Each data point in the dataset is processed in this way, thereby achieving dataset decentralization.

[0063] 3. Calculate the covariance matrix H. (4).

[0064] 4. Perform SVD decomposition on the covariance matrix H. (5), where U and V represent orthogonal matrices (for real matrices, they are identity matrices), and V represent the left singular vector and the right singular vector, respectively. This represents singular values.

[0065] 5. Calculate the target rotation matrix R1, R1=VU T .

[0066] 6. Calculate the target translation vector T1, T1=C b -R1×C s .

[0067] After obtaining the target rotation matrix R1 and the target translation vector T1, the target rotation matrix R1 is used to replace the initial rotation matrix R0, and the target translation vector T1 is used to replace the initial translation vector T0, to obtain a target conversion matrix M1, i.e., M1 = [R1, T1], so as to realize adjustment of the initial conversion matrix M0.

[0068] The following is another embodiment of a simulation model construction method provided in the specification.

[0069] Figure 6 Another schematic flowchart of a simulation model construction method provided by an embodiment of the application is shown in FIG. 13. Figure 6 As shown in FIG. 13, the simulation model construction method provided by the embodiment of the application includes the following scheme. S110: Obtain first action data of a first object, the first object being a physical entity in a real world; S120: Obtain driving parameters of a second object according to the first action data and a digital twin model, the second object being a digital twin of the first object; S130: Drive and control the second object to move in a virtual simulation environment based on the driving parameters to obtain second action data of the second object; S141: Determine a deviation between the first action data and the second action data; S142: Determine whether the deviation is located in a preset deviation interval, if not, perform S143, and if yes, perform S145; S143: Adjust an initial conversion matrix in the digital twin model according to the deviation to obtain a target conversion matrix; S144: Take the target conversion matrix as the initial conversion matrix, return and start performing from S110 again; S145: Determine a current digital twin model as a simulation control model.

[0070] By using the technical scheme of S110-S145, the application realizes optimization of a control model of a robot in a virtual simulation environment, without the need to actually manufacture a robot entity and without the need to bear high physical manufacturing costs and maintenance costs, so that the overall working cost in the control model optimization process can be reduced to a certain extent, and the application has significant economic efficiency and practicality. Since the application does not need to rely on a real robot entity in the process of optimizing the control model of the robot, the problem of interruption of model optimization caused by hardware failure or maintenance demand of the robot is avoided, so that the model optimization period is effectively shortened. Therefore, the application not only improves the rationality of the robot posture action, but also improves the learning efficiency of robot behavior learning.

[0071] The following is an embodiment of the device of the present application, which can be used to execute the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0072] Figure 7 The structure of the simulation model construction device provided by the embodiment of the present application is shown in the structural schematic diagram of the simulation model construction device. Figure 7 As shown, the simulation model construction device 400 is applied to a robot simulation system, and includes: A data acquisition module 410 is configured to acquire first action data of a first object, the first object being a physical entity in a real world; A parameter generation module 420 is configured to obtain driving parameters of a second object according to the first action data and a digital twin model, the second object being a digital twin of the first object; An action driving module 430 is configured to drive and control the second object to move in a virtual simulation environment based on the driving parameters, and obtain second action data of the second object; A model adjustment module 440 is configured to adjust the digital twin model according to the first action data and the second action data, and obtain a simulation control model of the robot.

[0073] In a possible implementation manner, the first object has a plurality of movable joints, and the data acquisition module 410 is specifically configured to acquire the first action data of the first object through a wearable device; the wearable device includes a virtual reality head-mounted display device, a posture sensor device, and a data glove device; the first action data includes position information, speed information, and movable joint information of the first object moving in the real world, and the movable joint information includes angle information and force information of the movable joint.

[0074] In a possible implementation manner, the parameter generation module 420 includes: A preprocessing unit is configured to pre-process the first action data through the digital twin model to obtain pre-processed data, the pre-processing including at least one of data verification, data filtering, and data format conversion; A data conversion unit is configured to convert the pre-processed data from a first coordinate system in which the first object is located to a second coordinate system in which the second object is located based on an initial conversion matrix in the digital twin model to obtain conversion data, the initial conversion matrix being a coordinate transformation matrix between the first coordinate system and the second coordinate system; A data generation unit is configured to generate the driving parameters according to the conversion data.

[0075] In a possible implementation, the data generation unit is specifically configured to: obtain a target action trajectory of the second object according to position coordinates of the first object at different time points in the conversion data, the target action trajectory including a target action direction; obtain a target posture of the second object at different time points according to angles of each movable joint of the first object at different time points in the conversion data; obtain a target action strength of the second object at different time points according to strengths generated by each movable joint of the first object at different time points in the conversion data; determine a target action speed of the second object at different time points according to displacements between position coordinates of the first object at adjacent time points in the conversion data and corresponding time intervals; and obtain the driving parameter according to the target action trajectory, the target posture, the target action strength, and the target action speed.

[0076] In a possible implementation, the model adjustment module 440 includes: The calculation unit is configured to determine a deviation between the first action data and the second action data. The adjustment unit is configured to adjust an initial conversion matrix in the digital twin model according to the deviation, to obtain the simulation control model.

[0077] In a possible implementation, the adjustment unit includes: The judgment subunit is configured to determine whether the deviation is located in a preset deviation interval. The determination subunit is configured to, in a case where the deviation is located in the preset deviation interval, determine the current digital twin model as the simulation control model.

[0078] In a possible implementation, the adjustment unit further includes: The correction subunit is configured to, in a case where the deviation is located outside the preset deviation interval, adjust the initial conversion matrix in the digital twin model according to the deviation, to obtain a target conversion matrix. The loop subunit is configured to: take the target conversion matrix as the initial conversion matrix; obtain the first action data of the first object; pre-process the first action data by using the digital twin model, to obtain pre-processed data; convert the pre-processed data from a first coordinate system in which the first object is located to a second coordinate system in which the second object is located based on the initial conversion matrix, to obtain conversion data; generate the driving parameter according to the conversion data; drive the second object to move in the virtual simulation environment based on the driving parameter, to obtain the second action data of the second object; and determine the deviation between the first action data and the second action data, until the deviation is located in the preset deviation interval, to obtain the simulation control model.

[0079] In a possible implementation, the correction subunit is specifically configured to construct a bias function based on the bias, unknowns in the bias function include the target rotation matrix and the target translation vector; the position coordinates in the first motion data and the second motion data are substituted into the bias function, and the bias function is solved to obtain the target rotation matrix and the target translation vector; the target rotation matrix is taken as the initial rotation matrix, and the target translation vector is taken as the initial translation vector to obtain the target conversion matrix.

[0080] In a possible implementation, the calculation unit is specifically configured to determine, according to the same movable joint corresponding to the first object and the second object, a difference between a first position coordinate and a second position coordinate of the same movable joint at different moments, to obtain a plurality of differences; the first position coordinate is a position coordinate of the same movable joint in the first motion data, and the second position coordinate is a position coordinate of the same movable joint in the second motion data; and determine an average value of the plurality of differences to obtain the bias.

[0081] It should be noted that the simulation model construction apparatus provided in the above embodiments is only used as an example for the division of the functional modules, and in actual applications, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the simulation model construction apparatus and the simulation model construction method provided in the above embodiments belong to the same concept, so for details not disclosed in the device embodiments of the present application, please refer to the above-mentioned simulation model construction method embodiments of the present application, which will not be described here.

[0082] The serial numbers of the above embodiments of the present application are only for description, and do not represent the advantages or disadvantages of the embodiments.

[0083] Figure 8 A structure schematic diagram of an electronic device provided in an embodiment of the present application is shown, as shown in the figure, the electronic device 500 includes a memory 501 and a processor 502, wherein the memory 501 stores executable program code 5011, and the processor 502 is configured to call and execute the executable program code 5011 to execute a simulation model construction method. Figure 8

[0084] The embodiment can divide the functional modules of the electronic device according to the above-mentioned method examples, for example, each functional module can be corresponding, or two or more functions can be integrated in one processing module, and the integrated module can be realized in the form of hardware. It should be noted that the division of the modules in the embodiment is illustrative, and is only a logical function division, and another division mode can be used in actual implementation.

[0085] ​In the case of adopting respective functional modules corresponding to respective functions, the electronic device can include a data acquisition module, a parameter generation module, an action driving module, a model construction module, and the like. It should be noted that all relevant content of each step involved in the above method embodiments can be referred to the function description of the corresponding functional module, and will not be repeated here.

[0086] The electronic device provided in the embodiment is used to execute the simulation model construction method described above, and thus can achieve the same effects as the implementation method described above.

[0087] In the case of adopting an integrated unit, the electronic device can include a processing module and a storage module. The processing module can be used to control and manage the actions of the electronic device. The storage module can be used to support the electronic device to execute related program codes and data, and the like.

[0088] The processing module can be a processor or a controller, which can implement or execute various exemplary logical blocks, modules, and circuits described in combination with the disclosure of the present application. The processor can also be a combination of computing functions, such as one or more microprocessor combinations, combinations of digital signal processing (DSP) and microprocessors, and the like. The storage module can be a memory.

[0089] The embodiment also provides a computer readable storage medium, which stores computer program codes. When the computer program codes run on a computer, the computer is caused to execute the related method steps described above to implement the simulation model construction method in the above embodiment.

[0090] The embodiment also provides a computer program product, which, when running on a computer, causes the computer to execute the related steps described above to implement the simulation model construction method in the above embodiment.

[0091] In addition, the electronic device provided by the embodiment of the present application can be a chip, an assembly, or a module. The electronic device can include a processor and a memory connected thereto. The memory is used to store instructions. When the electronic device is running, the processor can invoke and execute the instructions to cause the chip to execute the simulation model construction method in the above embodiment.

[0092] The electronic device, the computer readable storage medium, the computer program product, or the chip provided by the embodiment of the present application are all used to execute the corresponding simulation model construction method provided above, and thus the beneficial effects that can be achieved are referred to the beneficial effects of the corresponding simulation model construction method provided above, which will not be repeated here.

[0093] Through the description of the above embodiments, those skilled in the art can understand that, for the convenience and brevity of description, only the division of the above functional modules is exemplified, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0094] In the embodiments provided in the present application, it should be understood that the disclosed device and method can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the modules or units is only a logical function division, and in actual implementation, there can be another division manner, for example, a plurality of units or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed ones can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0095] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for constructing a simulation model, characterized in that, The method, applied to a robot simulation system, includes: Obtain the first action data of the first object, where the first object is a physical entity in the real world; Based on the first action data and the digital twin model, the driving parameters of the second object are obtained, and the second object is the digital twin of the first object; Based on the driving parameters, the second object is driven and controlled to move in the virtual simulation environment to obtain the second motion data of the second object; Based on the first action data and the second action data, the digital twin model is adjusted to obtain the simulation control model of the robot.

2. The method according to claim 1, wherein the first object has a plurality of movable joints, characterized in that, The acquisition of the first action data of the first object includes: The first action data of the first object is obtained through a wearable device; The wearable devices include virtual reality head-mounted displays, posture sensor devices, and data glove devices; The first motion data includes the position information, speed information, and movable joint information of the first object in the real world, and the movable joint information includes the angle information and force information of the movable joint.

3. The method according to claim 2, characterized in that, The process of obtaining the driving parameters of the second object based on the first action data and the digital twin model includes: The first action data is preprocessed using the digital twin model to obtain preprocessed data, wherein the preprocessing includes at least one of data verification, data filtering, and data format conversion. Based on the initial transformation matrix in the digital twin model, the preprocessed data is transformed from the first coordinate system where the first object is located to the second coordinate system where the second object is located to obtain transformed data. The initial transformation matrix is ​​the coordinate transformation matrix between the first coordinate system and the second coordinate system. The driving parameters are generated based on the converted data.

4. The method according to claim 3, characterized in that, The step of generating the driving parameters based on the conversion data includes: Based on the position coordinates of the first object at different times in the converted data, the target motion trajectory of the second object is obtained, and the target motion trajectory includes the target motion direction; Based on the angles of each movable joint of the first object at different times in the transformed data, the target posture of the second object at different times is obtained; Based on the force generated by each movable joint of the first object at different times in the converted data, the target motion force of the second object at different times is obtained; Based on the displacement between the position coordinates of the first object at adjacent time points and the corresponding time interval in the converted data, the target motion speed of the second object at different time points is determined; The driving parameters are obtained based on the target motion trajectory, the target posture, the target motion force, and the target motion speed.

5. The method according to claim 3, characterized in that, The step of adjusting the digital twin model based on the first action data and the second action data to obtain the robot's simulation control model includes: Determine the deviation between the first action data and the second action data; The initial transformation matrix in the digital twin model is adjusted according to the deviation to obtain the simulation control model.

6. The method according to claim 5, characterized in that, The step of adjusting the initial transformation matrix in the digital twin model according to the deviation to obtain the simulation control model includes: If the deviation is within a preset deviation range, the current digital twin model is determined as the simulation control model.

7. The method according to claim 5, characterized in that, The step of adjusting the initial transformation matrix in the digital twin model according to the deviation to obtain the simulation control model includes: If the deviation is outside the preset deviation range, the initial transformation matrix in the digital twin model is adjusted according to the deviation to obtain the target transformation matrix; Use the target transformation matrix as the initial transformation matrix; Obtain the first action data of the first object; The first action data is preprocessed using the digital twin model to obtain preprocessed data; Based on the initial transformation matrix, the preprocessed data is transformed from the first coordinate system where the first object is located to the second coordinate system where the second object is located, to obtain the transformed data; The driving parameters are generated based on the conversion data; Based on the driving parameters, the second object is driven and controlled to move in the virtual simulation environment to obtain the second motion data of the second object; The deviation between the first action data and the second action data is determined until the deviation is within the preset deviation range, thus obtaining the simulation control model.

8. The method according to claim 7, characterized in that, The initial transformation matrix includes an initial rotation matrix and an initial translation vector. Adjusting the initial transformation matrix in the digital twin model according to the deviation to obtain the target transformation matrix includes: The deviation function is constructed based on the deviation, wherein the unknowns in the deviation function include the target rotation matrix and the target translation vector; Substitute the position coordinates from the first motion data and the second motion data into the deviation function, and solve the deviation function to obtain the target rotation matrix and the target translation vector; The target transformation matrix is ​​obtained by using the target rotation matrix as the initial rotation matrix and the target translation vector as the initial translation vector.

9. The method according to claim 5, characterized in that, Determining the deviation between the first action data and the second action data includes: Based on the same movable joint corresponding to the first object and the second object, determine the difference between the first position coordinate and the second position coordinate of the same movable joint at different times, and obtain multiple differences; Wherein, the first position coordinate is the position coordinate of the same movable joint in the first motion data, and the second position coordinate is the position coordinate of the same movable joint in the second motion data; The average of the multiple differences is determined to obtain the deviation.

10. A simulation model construction device, characterized in that, The simulation model building device, used in robot simulation systems, includes: The data acquisition module is used to acquire the first action data of the first object, which is a physical entity in the real world; The parameter generation module is used to obtain the driving parameters of the second object based on the first action data and the digital twin model, wherein the second object is the digital twin of the first object; The motion driving module is used to drive and control the second object to move in the virtual simulation environment based on the driving parameters, and obtain the second motion data of the second object; The model adjustment module is used to adjust the digital twin model based on the first action data and the second action data to obtain the simulation control model of the robot.

11. An electronic device, characterized in that, The electronic device includes: Memory, used to store executable program code; A processor is configured to call and run the executable program code from the memory, causing the electronic device to perform the simulation model construction method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed, implements the simulation model construction method as described in any one of claims 1 to 9.

Images