Network training method and device, electronic equipment and computer readable storage medium

Abstract

The application provides a network training method and device, electronic equipment and computer readable storage medium; the method comprises: obtaining the initial state of the body data of a mechanical arm in a simulation environment and an interactive object, calling a parameter generation network to perform mapping processing on the body data to obtain the collision grid parameter of the interactive object, the friction coefficient and the stiffness coefficient of the interactive object in the process of the mechanical arm performing an interactive task; determining the first pose data of each operation point of the mechanical arm and the interactive object in the interaction process based on the collision grid parameter, the friction coefficient and the stiffness coefficient; determining a target loss gradient based on each first pose data and a preset second pose data of each reference point; and adjusting the network parameter of the parameter generation network by using the target loss gradient. Through the application, the optimization efficiency of physical parameters in a simulation environment can be improved.

Description

Technical Field

[0001] This application relates to the field of robotics technology, and in particular to a network training method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] With the rapid development of artificial intelligence technology, directly collecting training data for robots in the real physical world faces numerous severe challenges. These include high data collection costs, low efficiency, and security risks associated with real-world data collection, such as potential damage to robot equipment or environmental disturbances. Therefore, it is not feasible to rely solely on real-world data to train robots to perform complex tasks. Consequently, simulation environments can be constructed, and large-scale data can be collected within these environments to train robots. By adjusting the physical parameters within the simulation environment, the gap between the simulation and the real world can be effectively narrowed, significantly enhancing the training effect of robots in simulation environments and increasing the success rate of transferring task execution strategies to the real world.

[0003] In related technologies, physical parameters in simulation environments rely on manual adjustment, resulting in low optimization efficiency of physical parameters in simulation environments. Summary of the Invention

[0004] This application provides a network training method, apparatus, electronic device, and computer-readable storage medium, which can improve the optimization efficiency of physical parameters in a simulation environment.

[0005] The technical solution of this application embodiment is implemented as follows: This application provides a network training method, the method comprising: The robot arm's ontological data and the initial state of the interactive object are obtained in a simulation environment, which includes the robot arm and the interactive object. Obtain the parameter generation network to be trained, and perform a preset number of training iterations on the parameter generation network to obtain the trained parameter generation network: For each round, during the process of the robotic arm performing the interactive task, the parameter generation network is invoked to map the body data and obtain the collision mesh parameters, friction coefficient and stiffness coefficient of the interactive object. Based on the collision mesh parameters, the friction coefficient, and the stiffness coefficient, the first pose data of the robotic arm and the interactive object at each operation point during the interaction process is determined; The target loss gradient is determined based on each of the first pose data and the second pose data of each preset reference point; The network parameters of the parameter generation network are adjusted using the target loss gradient, and the interactive object is restored to the initial state.

[0006] This application provides a network training device, including: The mapping processing module is used to acquire the body data of the robotic arm and the initial state of the interactive object in the simulation environment, the simulation environment including the robotic arm and the interactive object; acquire the parameter generation network to be trained, and perform a preset number of training rounds on the parameter generation network to obtain the trained parameter generation network; for each round, during the process of the robotic arm performing the interactive task, the parameter generation network is called to perform mapping processing on the body data to obtain the collision mesh parameters, friction coefficient and stiffness coefficient of the interactive object; The first determining module is used to determine the first pose data of each operation point of the robotic arm and the interactive object during the interaction process based on the collision mesh parameters, the friction coefficient and the stiffness coefficient; The second determining module is used to determine the target loss gradient based on each of the first pose data and the second pose data of each preset reference point; The parameter adjustment module is used to adjust the network parameters of the parameter generation network using the target loss gradient and restore the interactive object to the initial state.

[0007] This application provides an electronic device, the electronic device comprising: Memory is used to store executable instructions or computer programs. The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the network training method provided in the embodiments of this application.

[0008] This application provides a computer-readable storage medium storing computer-executable instructions or computer programs, which, when executed by a processor, implement the network training method provided in this application.

[0009] This application provides a computer program product, including computer-executable instructions or a computer program, which, when executed by a processor, implements the network training method provided in this application.

[0010] The embodiments of this application have the following beneficial effects: By applying the embodiments of this application, the body data of the robotic arm and the initial state of the interactive object in the simulation environment are obtained, the parameter generation network to be trained is obtained, and the parameter generation network is trained for a preset number of rounds to obtain the trained parameter generation network. For each round, during the process of the robotic arm performing the interactive task, the parameter generation network is called to map the body data to obtain the collision mesh parameters, friction coefficient, and stiffness coefficient of the interactive object. A mapping relationship from the robotic arm's own state data to the physical parameters of the interactive object is established, realizing the automated generation of key physical parameters in the simulation environment and significantly reducing the dependence on manual adjustment. Secondly, based on the collision mesh parameters, friction coefficient, and stiffness coefficient, the first pose data of each operation point of the robotic arm and the interactive object in the interaction process is determined. Then, based on each first pose data and the second pose data of each preset reference point, the target loss gradient is determined. The network parameters of the parameter generation network are then adjusted using the target loss gradient, and the interactive object is restored to its initial state. In this way, the trained parameter generation network learns the mapping strategy from the state of the robotic arm to the optimal simulation parameters. As a result, in subsequent inference applications, the trained parameter generation network can be directly called to quickly generate physical parameters suitable for the robotic arm to perform tasks, avoiding manual parameter tuning and improving the optimization efficiency of physical parameters in the simulation environment. Attached Figure Description

[0011] Figure 1 This is a schematic diagram illustrating the application mode of the network training method provided in the embodiments of this application; Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application; Figure 3 This is a first flowchart illustrating the network training method provided in this application embodiment; Figure 4 This is a second flowchart illustrating the network training method provided in the embodiments of this application; Figure 5 This is a schematic diagram of the third process of the network training method provided in the embodiments of this application; Figure 6 This is a schematic diagram of the process for evaluating the trained parameter generation network provided in an embodiment of this application; Figure 7 This is a schematic diagram of the parameter generation network training process provided in the embodiments of this application; Figure 8 This is a schematic diagram of the parameter generation network inference process structure provided in the embodiments of this application; Figure 9 This is a schematic diagram of the process for training and evaluating the neural network of the physical parameter generator provided in the embodiments of this application; Figure 10This is a schematic diagram of the simulation environment for the robotic arm folding clothing task provided in the embodiments of this application; Figure 11 This is a schematic diagram of the neural network for the physical parameter generator provided in an embodiment of this application; Figure 12 This is a schematic diagram of the collision mesh of clothing provided in an embodiment of this application; Figure 13 This is a schematic diagram of the key operational points when the robotic arm folds clothing, as provided in the embodiments of this application; Figure 14 This is a schematic diagram of the gradient calculation graph provided in an embodiment of this application; Figure 15 This is a schematic diagram of the folding clothes task flow after physical parameter optimization provided in the embodiments of this application.

[0012] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0014] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0015] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0016] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0017] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0018] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for descriptive purposes only and is not intended to limit the scope of this application.

[0019] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0020] 1) Seven-axis robotic arm: It is a multi-degree-of-freedom mechanical device composed of multiple links connected by joints. It is a seven-axis redundant degree-of-freedom robotic arm with seven rotary joints.

[0021] 2) Parameter generation network: It is a fully connected neural network consisting of an input layer, two hidden layers and three output layers, used to map the current body data of the robotic arm to the relevant physical parameters in the simulation environment.

[0022] This application provides a network training method, apparatus, electronic device, and computer-readable storage medium, which can improve the optimization efficiency of physical parameters in a simulation environment.

[0023] The following describes exemplary applications of the electronic devices provided in the embodiments of this application. These electronic devices can be implemented as various types of terminals such as laptops, tablets, desktop computers, set-top boxes, smartphones, smart speakers, smartwatches, smart TVs, and in-vehicle terminals, or as servers. The following will describe exemplary applications when the device is implemented as a server.

[0024] See Figure 1 , Figure 1 This is a schematic diagram illustrating the application mode of the network training method provided in the embodiments of this application, for example. Figure 1 The system involves server 200, network 300, and terminal 400. Terminal 400 connects to server 200 through network 300, which can be a wide area network (WAN), a local area network (LAN), or a combination of both.

[0025] During the training of the parameter generation network, the terminal 400 sends the robot arm's ontology data and the initial state of the interactive object in the simulation environment to the server 200. The server 200 obtains the robot arm's ontology data and the initial state of the interactive object in the simulation environment, obtains the parameter generation network to be trained, and performs a preset number of training rounds on the parameter generation network to obtain the trained parameter generation network. For each round, during the robot arm's interactive task, the parameter generation network is invoked to map the ontology data, obtaining the collision mesh parameters, friction coefficient, and stiffness coefficient of the interactive object. Based on the collision mesh parameters, friction coefficient, and stiffness coefficient, the first pose data of each operation point of the robot arm and the interactive object during the interaction process is determined. Based on each first pose data and the second pose data of each preset reference point, the target loss gradient is determined. The network parameters of the parameter generation network are adjusted using the target loss gradient, and the interactive object is restored to its initial state. The terminal 400 is configured with a simulation environment for the robot arm to perform tasks, and the simulation environment includes the robot arm and the interactive object. For example, in a target tracking scenario, server 200 optimizes the physical parameters in a simulation environment used by the robotic arm to perform object tracking tasks based on a trained parameter generation network. Alternatively, in a human-computer interaction scenario, server 200 optimizes the physical parameters in a simulation environment used by the robotic arm to perform tasks such as folding clothes and packaging goods based on a trained parameter generation network.

[0026] See Figure 2 , Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may be a terminal or a server. Figure 2 The illustrated electronic device includes at least one processor 410, a memory 450, and at least one network interface 420. The various components of the electronic device are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 2 The general designated all buses as Bus System 440.

[0027] Processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0028] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices physically located away from the processor 410.

[0029] The memory 450 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory.

[0030] In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0031] Operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, for implementing various basic business functions and handling hardware-based tasks.

[0032] The network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including Bluetooth, WiFi, and Universal Serial Bus (USB).

[0033] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2 A network training device 455 stored in memory 450 is shown. It can be software in the form of programs and plug-ins, including the following software modules: mapping processing module 4551, first determination module 4552, second determination module 4553, and parameter adjustment module 4554. These modules are logical and can therefore be arbitrarily combined or further split according to the functions they implement. The functions of each module will be described below.

[0034] The network training method provided in this application will be described in conjunction with exemplary applications and implementations of the server devices provided in the embodiments of this application.

[0035] The following describes the network training method provided in the embodiments of this application. For example, to facilitate understanding of the network training method provided in the embodiments of this application, this application uses a scenario for optimizing physical parameters in a simulation environment as an example for explanation.

[0036] As mentioned above, the electronic device implementing the network training method of this application embodiment can be a terminal, a server, or a combination of both. The following explanation uses an electronic device as a server as an example to illustrate the network training method provided in this application embodiment. See also... Figure 3 , Figure 3 This is a first flowchart illustrating the network training method provided in this application embodiment, which will be combined with... Figure 3 The steps shown are explained.

[0037] In step 301, the body data of the robotic arm and the initial state of the interactive objects in the simulation environment are obtained.

[0038] Here, the robotic arm can be a seven-axis robotic arm. A seven-axis robotic arm is a multi-degree-of-freedom mechanical device composed of multiple links connected by joints; it is a seven-axis redundant degree-of-freedom robotic arm with seven rotational joints. The simulation environment includes the robotic arm and the interactive object. The simulation environment is used to simulate the physical characteristics and kinematic behavior of the robotic arm performing interactive tasks and is built based on an open-source simulation engine (Nvidia Newton). The simulation engine instantiates the robotic arm and the interactive object based on the imported Unified Robot Description Format (URDF) and the model description file of the interactive object, and initializes parameters such as the robotic arm control stiffness, the position of the robotic arm base, and the contact distance between the robotic arm and the interactive object. This simulation environment has differentiability, meaning that the functional relationship between physical quantities (such as position and velocity) in the simulation environment and input parameters (such as joint angles) is differentiable. For example, when the interactive task performed by the robotic arm in the simulation environment is folding clothes, the interactive object is the clothes, and the simulation environment includes the robotic arm, the robotic arm operating platform, and the foldable clothes on the operating platform. The initial state of the interactive object represents the original response state of the interactive object to external forces.

[0039] In step 302, the parameter generation network to be trained is obtained.

[0040] Here, the parameter generation network is a fully connected neural network consisting of an input layer, two hidden layers, and three output layers, used to map the current ontological data of the robotic arm to relevant physical parameters in the simulation environment.

[0041] After obtaining the parameter generation network to be trained, the parameter generation network is trained for a preset number of rounds to obtain the trained parameter generation network.

[0042] In step 303, for each round, during the process of the robotic arm performing the interactive task, the parameter generation network is invoked to map the body data and obtain the collision mesh parameters, friction coefficient and stiffness coefficient of the interactive object.

[0043] The collision mesh parameters of the interactive object are a set of values ​​defined based on the cage-based deformation model for collision detection and deformation calculation in the simulation environment. These parameters include the maximum number of meshes and the maximum distance between cage corners, and are presented as a 2D vector. The friction coefficient is the kinetic friction coefficient characterizing the surface of the interactive object when it comes into contact with other objects; it is a floating-point number in the range [0, 1]. The stiffness coefficient of the interactive object is a set of physical parameters describing its resistance to deformation. Taking clothing as an example, the stiffness coefficient includes the weft stretch coefficient (weft_ke), warp stretch coefficient (warp_ke), shear stretch coefficient (shear_ke), and bending damping coefficient (bend_kd), and is presented as a 4D vector.

[0044] In some embodiments, the ontology data includes real-time joint angle vectors, end-effector pose vectors, and end-effector contact force vectors. The parameter generation network includes a first hidden layer, a second hidden layer, a first output layer, a second output layer, and a third output layer. The parameter generation network is invoked to map the ontology data, obtaining the collision mesh parameters, friction coefficient, and stiffness coefficient of the interactive object. This can be achieved through the following steps: concatenating the real-time joint angle vectors, end-effector pose vectors, and end-effector contact force vectors to obtain a concatenated vector; then, using the weight matrix and bias matrix in the first hidden layer, performing nonlinear mapping on the concatenated vector to obtain a first processing result; using the weight matrix and bias matrix in the second hidden layer, performing nonlinear mapping on the first processing result to obtain a second processing result; using the weight matrix and bias matrix in the first output layer, performing nonlinear mapping on the second processing result to obtain the collision mesh parameters; using the weight matrix and bias matrix in the second output layer, performing nonlinear mapping on the second processing result to obtain the friction coefficient; and using the weight matrix and bias matrix in the third output layer, performing nonlinear mapping on the second processing result to obtain the stiffness coefficient.

[0045] Here, the real-time joint angle vector is a set of values ​​used to describe the current state of each joint of the robotic arm. The dimension of the real-time joint angle vector is equal to the number of joints of the robotic arm (7 in this embodiment). Each component of the real-time joint angle vector corresponds to the rotation angle value of a joint, collectively and uniquely determining the posture of the robotic arm in the joint space at the current moment. The end effector pose vector is a 6-dimensional vector, including the three-dimensional coordinates of the end effector in the robotic arm base coordinate system and the posture angles of the three axes. The end effector contact force vector includes the three-dimensional force acting on the end effector in the robotic arm base coordinate system and is a 3-dimensional vector. The real-time joint angle vector, the end effector pose vector, and the end effector contact force vector are concatenated to obtain a 16-dimensional concatenated vector.

[0046] The real-time joint angle vector, end-effector pose vector, and end-effector contact force vector in the stitched vector are standardized to obtain the standardized stitched vector. The network dimension of the first hidden layer is 128, and the parameter matrix in the first hidden layer is represented as follows. The bias matrix in the first hidden layer is represented as Based on the parameter matrix and bias matrix in the first hidden layer, a linear operation is performed on the standardized concatenated vector to obtain a representation vector. The ReLU activation function is then called to perform a nonlinear mapping on this representation vector to obtain the first processing result. For example, the first processing result is determined by the following formula (1): (1) in, This indicates the first processing result. This represents a standardized concatenated vector.

[0047] The network dimension of the second hidden layer is 128, and the parameter matrix in the second hidden layer is represented as follows: The bias matrix in the second hidden layer is represented as follows: Based on the parameter matrix and bias matrix in the second hidden layer, a linear operation is performed on the first processing result to obtain a representation vector. The activation function (ReLU) is called to perform nonlinear mapping processing on the representation vector, that is, to perform the processing in the above formula (1) on the representation vector to obtain the second processing result.

[0048] The network dimension of the first output layer is 2, and the parameter matrix in the first output layer is represented as follows: The bias matrix in the first output layer is represented as follows: Based on the parameter matrix and bias matrix in the first output layer, a linear operation is performed on the second processing result to obtain a representation vector. The normalization function (Softmax) is then called to normalize this representation vector, yielding the collision mesh parameters. For example, the normalization function is determined using the following formula (2): (2) in, This represents the result of normalization. Represents the normalization function. This represents the characterization vector.

[0049] The network dimension of the second output layer is 1, and the parameter matrix in the second output layer is represented as follows: The bias matrix in the second output layer is represented as follows: Based on the parameter matrix and bias matrix in the second output layer, a linear operation is performed on the second processing result to obtain a representation vector. The normalization function (Softmax) is called to normalize the representation vector, that is, to perform the processing in the above formula (2) on the representation vector to obtain the friction coefficient.

[0050] The network dimension of the third output layer is 4, and the parameter matrix in the third output layer is represented as follows: The bias matrix in the third output layer is represented as follows: Based on the parameter matrix and bias matrix in the third output layer, a linear operation is performed on the second processing result to obtain a representation vector. The normalization function (Softmax) is called to normalize the representation vector, that is, to perform the processing in the above formula (2) on the representation vector to obtain the stiffness coefficient.

[0051] In this embodiment, the first hidden layer, second hidden layer, first output layer, second output layer, and third output layer in the parameter generation network are respectively invoked to map the ontology data of the robotic arm in the simulation environment, thereby obtaining the collision mesh parameters, friction coefficient, and stiffness coefficient of the interactive object. This allows for the extraction and construction of deep features related to the physical parameter prediction task from the ontology data of the robotic arm layer by layer. The use of three independent output layers decouples different physical parameter generation tasks, enabling the focus on learning the mapping relationship between different physical parameters and shared deep features, thereby improving the accuracy of various physical parameter predictions.

[0052] Continue to refer to Figure 3 In step 304, based on the collision mesh parameters, friction coefficient, and stiffness coefficient, the first pose data of each operation point of the robotic arm and the interactive object during the interaction process is determined.

[0053] Here, collision mesh parameters, friction coefficient, and stiffness coefficient are physical parameters that can be modified in the simulation environment. Based on the collision mesh parameters, friction coefficient, and stiffness coefficient obtained by mapping the robot arm body data through a parameter generation network, the state of the interactive object in the simulation environment is updated. The first pose data is the pose data of each operation point when the robot arm performs an interactive operation on the state-updated interactive object.

[0054] In some embodiments, determining the first pose data of each operation point between the robotic arm and the interactive object during the interaction process based on collision mesh parameters, friction coefficient, and stiffness coefficient can be achieved through the following steps: updating the state of the interactive object based on collision mesh parameters, friction coefficient, and stiffness coefficient to obtain the updated interactive object; and acquiring the first pose data of each operation point when the robotic arm performs the interaction operation on the updated interactive object during the execution of the interaction task.

[0055] Here, in the simulation environment, the geometric model used for collision detection is reconstructed or adjusted based on the collision mesh parameters, the dynamic properties of the surface of the interactive object are adjusted based on the friction coefficient, and the constraint relationship between the nodes in the model that constitutes the interactive object is adjusted based on the stiffness coefficient, thereby changing the overall flexibility and anti-deformation ability of the interactive object, updating the state of the interactive object, and obtaining the updated interactive object. At this time, the response mode of the updated interactive object to external forces (such as contact forces) has been redefined.

[0056] After the state update of the interactive object is completed, the simulation environment enters the forward evolution phase. Following a preset control strategy or trajectory, the robotic arm is controlled to perform an interactive task (such as folding clothing) on ​​the updated interactive object. During the task execution, or after reaching a specific stage, a set of operation points on the updated interactive object will be identified and measured. These operation points are the positions where the robotic arm needs to perform operations to complete the task (e.g., corners or cuffs of clothing). The position and orientation angle of each operation point in three-dimensional space are combined to form the first pose data.

[0057] In this embodiment, the state of the interactive object is updated based on the collision mesh parameters, friction coefficient, and stiffness coefficient to obtain the updated interactive object. During the process of the robotic arm performing the interactive task, the first pose data of each operation point when the robotic arm performs the interactive operation on the updated interactive object is obtained. The dynamic response of the interactive object can be determined based on the currently generated physical parameters. The first pose data serves as the actual task execution trajectory in the simulation environment under the generated physical parameter configuration, providing a calculation basis for subsequent comparison with the reference pose data.

[0058] Continue to refer to Figure 3 In step 305, the target loss gradient is determined based on each first pose data and the second pose data of each preset reference point.

[0059] Here, the first pose data includes the first position and the first pose angle, and the second pose data is the pose data of each reference point preset in the model description file of the interactive object, which includes the reference position and the reference pose angle. For example, each reference point is represented by the following formula (3): (3) in, Represents the set of reference points. Indicate each reference point, ( ) represents the reference position of each reference point, ( () represents the reference attitude angle of each reference point.

[0060] In some embodiments, see Figure 4 , Figure 4This is a schematic diagram of the second process of the network training method provided in the embodiments of this application. Figure 3 Step 305 shown can be achieved through... Figure 4 Steps 3051 to 3054 are implemented, and will be explained in detail below.

[0061] In step 3051, for each operation point, the position error between the first position of the operation point and the reference position of the corresponding reference point is determined, and the attitude angle error between the first attitude angle of the operation point and the reference attitude angle of the corresponding reference point is determined.

[0062] Here, each operation point corresponds to a reference point. The first position is the 3D position of the operation point on the interactive object. For each operation point, the Euclidean distance is calculated between the first position of the operation point and the reference position of the corresponding reference point to obtain the position error. For example, the position error is expressed as... ,in, Indicates the first position of the operation point. This indicates the reference position of the corresponding reference point.

[0063] The first attitude angle is the three-dimensional attitude angle of the operation point on the interactive object. The three-dimensional attitude angle represents the radians around the three coordinate axes in the coordinate system of the robot arm base. For each operation point, the angle difference between the first attitude angle of the operation point and the reference attitude angle of the corresponding reference point is calculated to obtain the attitude angle error. For example, the first attitude angle is expressed as ( The second attitude angle is expressed as ( The attitude angle error is expressed as .

[0064] In step 3052, the position error and attitude angle error are fused to obtain the first loss value of the operation point.

[0065] Here, based on the first weighting coefficient corresponding to the position error and the second weighting coefficient corresponding to the attitude angle error, the position error and attitude angle error are weighted and summed to achieve fusion processing, resulting in a loss value corresponding to an operation point, i.e., the first loss value. For example, the first loss value is represented as... ,in, This represents the first weighting coefficient. This represents the second weighting coefficient.

[0066] In step 3053, the multiple first loss values ​​are summed and averaged to obtain the target loss value.

[0067] Here, the target loss value is obtained by summing and averaging multiple first loss values ​​based on the number of operation points. For example, the target loss value is determined using the following formula (4): (4) in, Indicates the target loss value. Indicates the number of operation points. Indicates the first position of the operation point, ( () represents the first attitude angle of the operation point. Indicates the reference position of the reference point, ( () represents the reference attitude angle of the reference point. This represents the first weighting coefficient. This represents the second weighting coefficient. The first and second weighting coefficients are used to measure the alignment position error and attitude angle error.

[0068] In step 3054, backpropagation is performed on the target loss value to obtain the set of partial derivatives of the target loss function at the network parameters, and the set of partial derivatives is determined as the target loss gradient.

[0069] Here, a gradient computation graph is constructed based on the network parameters of the parameter generation network. Backpropagation is performed on the target loss value based on the gradient computation graph to obtain the partial derivatives of the target loss function at each network parameter. The partial derivatives are then combined to form the target loss gradient.

[0070] In this embodiment, position error and attitude angle error are fused to obtain a first loss value for each operation point. Multiple first loss values ​​are summed and averaged to obtain a target loss value. Then, backpropagation is performed on the target loss value to obtain a set of partial derivatives of the target loss function at the network parameters. The set of partial derivatives is determined as the target loss gradient. By summing and averaging the loss values ​​of multiple operation points, a quantitative evaluation of the robot arm's task execution effect is achieved. This provides a unified standard for evaluating the quality of physical parameters generated by the parameter generation network. The target loss gradient accurately reveals the degree of influence of each network parameter on the task execution error and points out the most effective parameter adjustment direction to reduce the error.

[0071] In some embodiments, see Figure 5 , Figure 5 This is a schematic diagram of the third process of the network training method provided in the embodiments of this application. Figure 4 In step 3054 shown, "perform backpropagation on the target loss value to obtain the set of partial derivatives of the target loss function at the network parameters," which can be achieved through... Figure 5 Steps 30541 to 30543 are implemented, and will be explained in detail below.

[0072] In step 30541, a first objective parameter with a gradient and a second objective parameter without a gradient are determined from the network parameters of the parameter generation network.

[0073] Here, the network parameters of the parameter generation network include real-time joint angles (joint_q), real-time joint angular velocities (joint_qd), initial joint angles (joint_q_start), initial joint angular velocities (joint_qd_start), link poses (body_q), link velocities (body_qd), joint types (joint_type), joint axes (joint_axis), parent / child joints (joint_parent / joint_child), parent / child joint transformation matrices (joint_X_p / joint_X_c), joint degrees of freedom (joint_dof_dim), the centroid positions of each link (body_com), and link output information (body_out). The real-time joint angles, real-time joint angular velocities, link poses, link velocities, and link output information are determined to be differentiable parameters, i.e., the first objective parameters with gradients. The remaining network parameters are determined to be non-differentiable parameters, i.e., the second objective parameters without gradients.

[0074] In step 30542, a gradient calculation graph is constructed based on the first target parameter, the second target parameter, and the preset differential operator.

[0075] Here, the first node corresponding to the first objective parameter and the second node corresponding to the second objective parameter in the gradient computation graph are interconnected. The preset differential operator is the gradient function operator, including the differential operator of multiplication (kernel0) and the differential operator of exponentiation / root extraction (kernel1).

[0076] In some embodiments, the gradient computation graph is constructed based on the first target parameter, the second target parameter, and the preset differential operator, which can be achieved through the following steps: constructing a first node based on the first target parameter and constructing a second node based on the second target parameter; constructing a directed acyclic graph based on the first node, the second node, and the preset differential operator, and determining the directed acyclic graph as the gradient computation graph.

[0077] Here, a data structure for a first node corresponding to the first target parameter is created. This data structure for the first node is configured with a storage unit for storing the current value of the first target parameter, and an identifier for enabling gradient calculation is set. This identifier is used to indicate that this node is a gradient node that needs to calculate and store its corresponding gradient value during gradient backpropagation.

[0078] Create a data structure for the second node corresponding to the second objective parameter. This data structure for the second node is configured with a storage unit for storing the value of the second objective parameter, and a flag is set to disable gradient calculation. This flag is used to indicate that the node is a constant node during gradient backpropagation and does not participate in the calculation and updating of gradient values.

[0079] The first and second nodes are defined as input nodes in the gradient computation graph. Preset differential operators are instantiated as computation nodes in the gradient computation graph. Directed connections are established between input nodes and computation nodes, resulting in a graph structure composed of nodes and directed edges, i.e., a directed acyclic graph (DAG). This DAG is then defined as the gradient computation graph. This gradient computation graph contains no closed loops, represents the data flow and computation order, and clarifies the dependencies between operations: from one input node to one computation node, or from the output of one computation node to the input of another. See the example for reference. Figure 14 , Figure 14 This is a schematic diagram of the gradient calculation graph provided in an embodiment of this application. For example... Figure 14 As shown, the nodes corresponding to the first target parameters "real-time joint angle", "real-time joint angular velocity", "link pose", "link velocity" and "link output information" are the first nodes, and the nodes corresponding to the remaining second target parameters are the second nodes.

[0080] In this embodiment, a first node is constructed based on a first target parameter with a gradient, and a second node is constructed based on a second target parameter without a gradient. A directed acyclic graph is constructed based on the first node, the second node, and a preset differential operator, and the directed acyclic graph is determined as the gradient calculation graph. This allows the gradient calculation graph to clearly and completely record the dependency path of the target loss value on each first target parameter, and to automatically and accurately solve the partial derivatives of the target loss function at each trainable parameter (first target parameter). This structurates the gradient solution process and provides the necessary data structure foundation for the subsequent automatic gradient calculation using the backpropagation algorithm.

[0081] Continue to refer to Figure 5 In step 30543, in the gradient calculation graph, the partial derivatives of the target loss function at each first target parameter are calculated layer by layer based on the target loss value, and a set of partial derivatives is constructed based on each partial derivative.

[0082] Here, the output node representing the final calculation result is located in the gradient computation graph. This output node stores the calculation result of the target loss function, i.e., the target loss value. Using this output node as the starting point of backpropagation, gradient calculation and propagation are performed layer by layer towards the input nodes (i.e., the first and second nodes) and the computation nodes (i.e., the differential operators) along the reverse direction of the connection relationship in the gradient computation graph. For any computation node in the gradient computation graph, the partial derivatives of its output gradient with respect to each of the first target parameters at its input are calculated using the preset differential rules of that computation node, and the partial derivatives are combined into a set of partial derivatives.

[0083] In this embodiment, a gradient computation graph is constructed based on a first target parameter with a gradient, a second target parameter without a gradient, and a preset differential operator in the network parameters. In the gradient computation graph, the partial derivatives of the target loss function at each first target parameter are calculated layer by layer based on the target loss value. This enables the target loss value to be propagated backward layer by layer in the gradient computation graph, which can automatically and accurately solve the partial derivatives of the target loss function with respect to each trainable parameter (first target parameter). The partial derivatives are then combined to form the target loss gradient, providing a definite direction and magnitude basis for subsequent optimization algorithms such as gradient descent to update the network parameters.

[0084] Continue to refer to Figure 3 In step 306, the network parameters of the network are generated by adjusting the parameters using the target loss gradient, and the interaction object is restored to its initial state.

[0085] Here, the network parameters of the parameter generation network are adjusted based on the target loss gradient to achieve a comprehensive update of the network parameters, resulting in the trained parameter generation network at the current time step. For example, the updated network parameters of the parameter generation network are determined using the following formula (5): (5) in, express The updated parameters at each time point generate the network parameters of the network. express The parameters at time points generate the network parameters of the network. This represents the gradient of the target loss.

[0086] The interactive object is restored to its initial state so that the above training process can be repeated based on the ontology data configured by the robotic arm in the simulation environment if the current training round has not reached the preset round.

[0087] In step 307, it is determined whether the preset number of rounds has been reached.

[0088] Here, the preset number of rounds is the number of training rounds set before training the parameter generation network. For example, the preset number of rounds is 10. When the current training round is 10, it is determined that the preset number of rounds has been reached; when the current training round is 5, it is determined that the preset number of rounds has not been reached.

[0089] If the preset number of rounds has not been reached, proceed to step 303 to continue training the parameter generation network; if the preset number of rounds has been reached, proceed to step 308.

[0090] In step 308, the trained parameter generation network is obtained.

[0091] Based on steps 301 to 308 above, the simulation is driven by the physical parameters (collision mesh, friction coefficient, stiffness coefficient) output by the parameter generation network in each training round, and the deviation between the simulation result (first pose data) and the reference standard (second pose data) is calculated to form a loss function for training the parameter generation network. This allows the parameter generation network to learn the mapping strategy from the state of the robotic arm to the optimal simulation parameters. In subsequent inference applications, the trained parameter generation network can be directly called to quickly generate physical parameters suitable for the robotic arm to perform tasks, avoiding manual parameter tuning and improving the optimization efficiency of physical parameters in the simulation environment. This provides key technical support for establishing a high-fidelity, highly task-adaptive simulation environment.

[0092] In some embodiments, the trained parameter generation network needs to be evaluated before it is deployed. See also Figure 6 , Figure 6 This is a schematic flowchart illustrating the evaluation of a trained parameter generation network provided in an embodiment of this application. The evaluation process can be carried out through... Figure 6 Steps 310 to 312 are implemented, and will be explained in detail below.

[0093] In step 310, during the process of the robotic arm performing the interactive task, the trained parameter generation network is invoked to map the ontology data and obtain the target collision mesh parameters, target friction coefficient and target stiffness coefficient of the interactive object.

[0094] Here, the real-time joint angle vector, end-effector pose vector, and end-effector contact force vector from the configured ontology data are concatenated to obtain a concatenated vector. The trained parameter generation network includes a first target hidden layer, a second target hidden layer, a first target output layer, a second target output layer, and a third target output layer. The weight and bias matrices from the first target hidden layer are used to perform nonlinear mapping on the concatenated vector to obtain the first target processing result. The weight and bias matrices from the second target hidden layer are used to perform nonlinear mapping on the first target processing result to obtain the second target processing result. The weight and bias matrices from the first target output layer are used to perform nonlinear mapping on the second target processing result to obtain the target collision mesh parameters. The weight and bias matrices from the second target output layer are used to perform nonlinear mapping on the second target processing result to obtain the target friction coefficient; the weight and bias matrices from the third target output layer are used to perform nonlinear mapping on the second target processing result to obtain the target stiffness coefficient. That is, the process of determining the target collision mesh parameters, the target friction coefficient, and the target stiffness coefficient in step 310 is the same as the process of determining the collision mesh parameters, friction coefficient, and stiffness coefficient in step 303 above, and will not be repeated here.

[0095] In step 311, based on the target collision mesh parameters, the target friction coefficient, and the target stiffness coefficient, the third pose data of each operation point of the robotic arm and the interactive object during the interaction process is determined.

[0096] Here, the state of the interactive object is updated based on the target collision mesh parameters, target friction coefficient, and target stiffness coefficient to obtain the updated interactive object. During the execution of the interactive task by the robotic arm, the pose data of each operation point when the robotic arm performs the interactive operation on the updated interactive object is acquired, which is the third pose data. The process of determining the third pose data in step 311 is the same as the process of determining the first pose data in step 304 above, and will not be described again here.

[0097] In step 312, the parameter generation effect is evaluated based on each third pose data and the second pose data of each reference point, and the evaluation result is obtained.

[0098] Here, the target position error and target pose angle error are determined based on each third pose data and each second pose data of a reference point. The parameter generation effect of the trained parameter generation network is evaluated by combining the target position error and target pose angle error.

[0099] In this embodiment, the trained parameter generation network is first invoked to map the body data of the robotic arm in the simulation environment, obtaining the target collision mesh parameters, target friction coefficient, and target stiffness coefficient of the interactive object. Then, based on the target collision mesh parameters, target friction coefficient, and target stiffness coefficient, the third pose data of each operation point of the robotic arm and the interactive object during the interaction process is determined. Finally, the parameter generation effect is evaluated based on each third pose data and the second pose data of each reference point. This objectively and quantitatively evaluates the parameter optimization capability of the trained parameter generation network, more realistically reflecting its comprehensive performance in optimizing physical parameters. This provides reliable technical support for confirming whether it meets the preset performance requirements and is suitable for subsequent practical deployment.

[0100] In some embodiments, the parameter generation effect is evaluated based on each third pose data and each second pose data of a reference point to obtain the evaluation result. This can be achieved through the following steps: determining a second loss value based on each third pose data and each second pose data; determining the evaluation result as failing when the second loss value is greater than or equal to a preset threshold; acquiring the body data of the robotic arm in the simulation environment again, and performing a preset number of training rounds on the trained parameter generation network based on the body data.

[0101] Here, the third pose data includes the target position and target attitude angle, and the second pose data includes the reference position and reference attitude angle. For each operation point, the target position error between the target position of the operation point and the reference position of the corresponding reference point is determined, and the target attitude angle error between the target attitude angle of the operation point and the reference attitude angle of the corresponding reference point is determined. The target position error and the target attitude angle error are fused to obtain the loss value corresponding to one operation point. The loss values ​​corresponding to multiple operation points are summed and averaged to obtain the second loss value.

[0102] The preset threshold is used to evaluate the effect of parameter generation. When the second loss value is greater than or equal to the preset threshold, it indicates that the robot arm's performance in performing the task under the generated physical parameters has not achieved the preset effect. At this time, the evaluation result is determined to be unsuccessful. Then, the ontology data of the robot arm in the simulation environment is obtained again, and the trained parameter generation network is trained for a preset number of rounds based on the ontology data. The trained parameter generation network is obtained again, and the trained parameter generation network is evaluated again until a parameter generation network that passes the evaluation is obtained.

[0103] In this embodiment, a second loss value is determined based on the third pose data of each operation point of the robotic arm and the interactive object during the interaction process and the second pose data of each preset reference point. When the second loss value is greater than or equal to a preset threshold, the ontology data of the robotic arm in the simulation environment is acquired again, and the trained parameter generation network is trained for a preset number of rounds based on the ontology data. When the performance of the parameter generation network does not meet the preset standard, a new round of data acquisition and network retraining process can be automatically triggered, thereby ensuring the accuracy and robustness of the parameter generation network in optimizing physical parameters.

[0104] In some embodiments, when the second loss value is less than a preset threshold, the evaluation result is determined to be passed. At this time, the training of the parameter generation network can be stopped, and the parameter generation network that has passed the evaluation is determined as the final trained parameter generation network.

[0105] Example, reference Figure 7 , Figure 7This is a schematic diagram of the parameter generation network training process provided in this application embodiment. The process involves acquiring the ontology data of the robotic arm and the initial state of the interactive object in a simulation environment, which includes the robotic arm and the interactive object; acquiring the parameter generation network 320 to be trained and performing a preset number of training rounds to obtain the trained parameter generation network; for each round, during the robotic arm's interactive task, calling the parameter generation network 320 to map the ontology data 321 to obtain the collision mesh parameters, friction coefficient, and stiffness coefficient of the interactive object; based on the collision mesh parameters, friction coefficient, and stiffness coefficient, determining the first pose data of each operation point between the robotic arm and the interactive object during the interaction process; determining the target loss gradient based on each first pose data and the second pose data of each preset reference point; adjusting the network parameters of the parameter generation network using the target loss gradient, and restoring the interactive object to its initial state.

[0106] In some embodiments, the network training method provided in this application can be applied to scenarios such as target tracking and human-computer interaction. The method involves acquiring the ontology data of the robotic arm in a simulation environment and the initial state of the interactive object, obtaining a parameter generation network to be trained, and performing a preset number of training rounds on the parameter generation network to obtain the trained parameter generation network. For each round, during the robotic arm's interactive task, the parameter generation network is invoked to map the ontology data, obtaining the collision mesh parameters, friction coefficient, and stiffness coefficient of the interactive object. This establishes a mapping relationship from the robotic arm's own state data to the physical parameters of the interactive object, achieving automated generation of key physical parameters in the simulation environment and significantly reducing reliance on manual adjustment. Next, based on the collision mesh parameters, friction coefficient, and stiffness coefficient, the first pose data of each operation point during the interaction between the robotic arm and the interactive object is determined. Then, based on each first pose data and the second pose data of each preset reference point, the target loss gradient is determined. Finally, the network parameters of the parameter generation network are adjusted using the target loss gradient, and the interactive object is restored to its initial state. In this way, the trained parameter generation network learns the mapping strategy from the state of the robotic arm to the optimal simulation parameters. As a result, in subsequent inference applications, the trained parameter generation network can be directly called to quickly generate physical parameters suitable for the robotic arm to perform tasks, avoiding manual parameter tuning and improving the optimization efficiency of physical parameters in the simulation environment.

[0107] The joint planning method provided in the embodiments of this application is described below. See also: Figure 8 , Figure 8This is a schematic diagram of the parameter generation network inference process provided in this embodiment of the application. During the process of the robotic arm performing the interactive task, the trained parameter generation network 330 is invoked to map the real-time body data 331 of the robotic arm to obtain the predicted collision mesh parameters, predicted friction coefficient, and predicted stiffness coefficient of the interactive object.

[0108] The following will describe an exemplary application of the network training method provided in the embodiments of this application in optimizing the physical parameters of a simulation environment.

[0109] With the rapid development of artificial intelligence technology, especially the successful application of reinforcement learning in the field of robot control, it has become possible to train robots to complete complex real-world tasks using large amounts of data. However, directly collecting training data for robots in the real physical world faces many serious challenges, such as high data collection costs, low collection efficiency, and safety risks associated with data collection in the real world, which may lead to damage to robot equipment or destruction of the surrounding environment. Therefore, it is not possible to rely entirely on real-world data to train robots to perform complex tasks. Thus, it is possible to construct simulation environments and use them to collect large-scale data to train robots.

[0110] In simulation environments, the setting of physical parameters (such as the coefficient of friction between objects, elastic modulus, collision mesh fineness, damping coefficient, etc.) requires manual adjustment. Engineers or researchers typically need to determine a set of physical parameters through repeated trials and adjustments based on the specific requirements of the task and their own domain knowledge and experience. This method of adjusting parameters based on task characteristics and human experience has the following significant problems and drawbacks: 1) Poor generalization: Manual parameter tuning is often optimized for specific tasks or scenarios. When the task objectives, robot types or environmental objects change, the originally set physical parameters may no longer be applicable, requiring a tedious parameter tuning process. This makes it difficult for the simulation environment to adapt to diverse task requirements, and its generalization ability is greatly limited. 2) High cost of manual parameter tuning: The optimization process of physical parameters requires domain experts to invest a lot of time and energy. For complex simulation scenarios, there are a large number of physical parameters involved, and there are complex coupling relationships between the parameters. The process of manually exploring the optimal parameter combination is extremely time-consuming and labor-intensive, which significantly increases the development and use cost of the simulation environment.

[0111] 3) High difficulty in parameter tuning and difficulty in ensuring accuracy: The relationship between physical parameters and simulation results is often nonlinear and lacks a clear analytical expression. Manual tuning makes it difficult to systematically find the globally optimal or near-optimal parameter combination, resulting in a "reality gap" between the simulation environment and the real world. That is, robot strategies that perform well in the simulation environment will experience a significant performance drop when transferred to the real world.

[0112] Furthermore, related technologies utilize black-box optimization techniques such as Bayesian optimization and genetic algorithms to search for optimal physical parameters. However, these methods typically require extensive simulation experiments to evaluate the effects of different parameter combinations, resulting in high computational costs and slow convergence. Moreover, these methods often involve direct optimization for specific parameter sets, making it difficult to develop a general parameter generation mechanism that can quickly adapt to new tasks.

[0113] This application proposes a network training method to address the problems existing in related technologies, which includes the following improvements compared to related technologies: A neural network was constructed as a physical parameter generator in the simulation environment (the parameter generation network in the above embodiment). A task-related loss function was constructed, and then the gradient of the loss function was used to directly optimize the physical parameter generator to generate optimal physical parameters, thereby improving the robot's performance in completing tasks. This physical parameter generator can autonomously explore the optimal physical parameters without any human assistance. Furthermore, due to the use of differentiable gradient training, the output can quickly converge to the optimal parameters or their neighborhood. It plays a crucial role in realizing a more realistic simulation environment conducive to robot tasks and in achieving efficient robot training, thus possessing broad application prospects and high application value.

[0114] In this embodiment, the gradient of the task-related loss function is used to directly train the neural network and then used to generate physical parameters. Since the differentiable gradient is used for direct training, the convergence of the neural network can be completed in a few iterations, that is, the optimal physical parameters can be found in a short time. The solution process is simple and the time cost of parameter optimization is greatly reduced. This solves the problems of complex and slow solution and high time cost of traditional physical parameter tuning methods.

[0115] In this embodiment, a neural network is used to jointly optimize multiple physical parameters. Multiple nonlinear mapping layers can map the optimization space of physical parameters to a continuous high-dimensional space. The optimal solution can be found quickly through gradient descent. Compared with the inefficient method of manually determining physical parameters and continuously experimenting and comparing in traditional methods, the optimal physical parameters can be found efficiently, solving the problem of difficulty in optimization or getting trapped in local suboptimal conditions in traditional methods.

[0116] In this embodiment, by utilizing neural networks and differentiable gradients to adaptively optimize physical parameters, the input and output of the neural network and the scenario-specific loss function can be fine-tuned according to the different scenarios in which the robot completes the task. After a small amount of training, the parameters can be converged to the optimal parameters for that scenario. In contrast, traditional methods require repeated trial and error to optimize based on different scenarios and human experience. This solves the problem of insufficient generalization of traditional methods.

[0117] The training and evaluation processes of the physical parameter generator neural network (the parameter generation network in the above embodiments) are described below. First, a robot operation task of folding clothes is defined, and a relevant simulation environment is built. This simulation environment consists of a seven-axis robotic arm with a fixed base, a robotic arm operating platform, and a foldable clothing model on the platform. A double-hidden-layer fully connected neural network is designed as the physical parameter generator. Then, the current body data of the robotic arm (real-time joint angles, end-effector spatial position and attitude, and end-effector 3D contact force) is input into the physical parameter generator, which outputs estimated values ​​of the physical parameters to be optimized (collision mesh parameters of the interactive objects, friction coefficient, and stiffness coefficient of the interactive objects). Next, a loss function is designed to train the physical parameter generator. Since the loss function is strongly correlated with the task scenario, in the folding clothes task, the loss function is defined as the deviation between the correct trajectory and the example trajectory during key stages (such as when clamping and releasing the clothes). Then, the real-time loss is calculated using the loss function, and the loss gradient is directly backpropagated through the computational graph connecting the physical quantities in the differentiable simulation environment to update the physical parameter generator neural network. Finally, after a small number of training steps, the effect of parameter generation is verified by evaluating the real-time loss.

[0118] Example, reference Figure 9 , Figure 9 This is a schematic flowchart illustrating the training and evaluation of a physical parameter generator neural network provided in an embodiment of this application. The electronic device implementing the network training and evaluation method of this embodiment can be a server, combined with... Figure 9 The steps shown illustrate the training and evaluation process of the physical parameter generator neural network provided in the embodiments of this application.

[0119] In step 901, a simulation environment for a robotic arm to fold clothing is constructed.

[0120] Here, a simulation environment for a robotic arm folding clothing is built based on the open-source differentiable simulation engine (Nvidia Newton). First, the robot description file (Unified Robot DescriptionFormat, URDF) of the seven-axis robotic arm and the clothing model description file to be manipulated are imported into the simulation engine. The simulation engine initializes parameters such as the robotic arm's control stiffness, the position of the robotic arm's base, and the contact distance with the clothing. The collision mesh parameters, friction coefficient, and stiffness coefficient of the clothing are modifiable physical parameters (i.e., the output parameters of the physical parameter generator). See the example for reference. Figure 10 , Figure 10 This is a schematic diagram of the simulation environment for the robotic arm folding clothing task provided in this application embodiment. The simulation environment 1001 includes a robotic arm 1002, an operating table 1003, and clothing 1004.

[0121] Based on this, an interaction logic between the clothing folding task and the simulation environment was constructed. During the training phase, the simulation environment was reset before each training round, the robotic arm was reset to its initial posture, and the clothing's center of mass posture was reset to a fixed initial position. The simulation environment acquires the current physical parameters and updates the clothing state, then acquires the pose of the robotic arm at the key operation points of the clothing to calculate the loss function and update the physical parameter generator neural network. The original parameters are updated using the physical parameters output by the physical parameter generator neural network, and then the next training round begins.

[0122] During the evaluation phase, the physical parameters output by the fixed physical parameter generator are used. Then, during the folding process of the robotic arm, the arm folds the clothing according to a fixed procedure. The predicted points for the robotic arm to grip and release the clothing are determined based on the physical parameters. The simulation environment provides reference key points for gripping and releasing the clothing model (the reference key points are provided by the clothing model file and do not contain any physical parameter information). The effectiveness of the physical parameter optimization is evaluated based on the error between the predicted points and the reference key points.

[0123] In step 902, a physical parameter generator based on a neural network is constructed.

[0124] Here, a fully connected neural network with two hidden layers is used as the physics parameter generator. For example, refer to [reference needed]. Figure 11 , Figure 11 This is a schematic diagram of the physical parameter generator neural network provided in this application embodiment. The first hidden layer network 1101 and the second hidden layer network 1102 both have a dimension of 128. The total input vector of the physical parameter generator neural network includes the real-time joint angle of the current robotic arm, the six-dimensional spatial pose of the end effector, and the three-dimensional contact force of the end effector (the ontology data in the above embodiment). The real-time joint angle is a 7-dimensional vector. The six-dimensional spatial pose of the end effector includes the three-dimensional coordinates of the end effector in the coordinate system of the robotic arm base and the pose angles of the three axes (in radians: rx, ry, rz). Therefore, the six-dimensional spatial pose of the end effector is a 6-dimensional vector. The three-dimensional contact force of the end effector (i.e., fx, fy, fz) is the three-dimensional force on the end effector in the coordinate system of the robotic arm base, which is a 3-dimensional vector. The three input vectors are concatenated in sequence into a 16-dimensional vector to obtain the total input vector.

[0125] The physics parameter generator neural network generates collision mesh parameters, friction coefficients, and object stiffness coefficients in a multi-head output manner. The collision mesh for clothing uses a cage-based deformation model; see reference for an example. Figure 12 , Figure 12This is a schematic diagram of the collision mesh of clothing provided in an embodiment of this application. The collision mesh parameters are a 2D vector, including the maximum number of meshes and the maximum distance between cage corners. The friction coefficient refers to the kinetic friction coefficient of the clothing, which is a floating-point number in the interval [0, 1]. The stiffness coefficient of the object is a 4D vector, including the tensile coefficient of the clothing along the weft direction (weft_ke), the tensile coefficient of the clothing along the warp direction (warp_ke), the tensile coefficient of the clothing along the shear direction (shear_ke), and the bending damping coefficient (bend_kd).

[0126] The forward pass of the physical parameter generator neural network is as follows: First, the total input vector is block-normalized. For example, the real-time joint angles of the robotic arm need to be normalized according to the feasible range of joint angles. The normalized 16-dimensional total input vector is then input into two hidden layer networks in sequence, where the nonlinear mapping function is the activation function (ReLU). For example, the mapped representation vector (the first processing result in the above embodiment) is determined by the following formula (1): (1) in, Represents the mapped representation vector. This represents the total input vector (the standardized concatenated vector in the above embodiments).

[0127] The output of the second hidden layer is connected to the three output layers. Each output layer has its own weight matrix and bias matrix. After linear mapping of the weight matrix and bias matrix in each output layer, the normalized mapping result is obtained through nonlinear mapping processing using a normalization function (Softmax). The dimensions of the three output layers are 2D (collision mesh parameters), 1D (friction coefficient), and 4D (object stiffness coefficient), respectively.

[0128] For example, the normalization function is determined by the following formula (2): (2) in, This represents the normalized mapping result. Represents the normalization function. This represents the characterization vector.

[0129] The normalized mapping results are post-processed according to the corresponding parameter types. By means of pruning, each physical parameter is constrained to a reasonable range. For example, if the maximum number of grids in the collision mesh parameter is an integer, the mapping result will be rounded to the nearest integer.

[0130] In step 903, a loss function based on the pose error of key points is constructed.

[0131] Here, a loss function based on the pose error of key points in the clothing folding task is constructed. The set of key points for the robotic arm to grip and release the clothing (the first pose data in the above embodiment) is determined based on the generated physical parameters. For example, refer to... Figure 13 , Figure 13 This is a schematic diagram illustrating the key operational points when the robotic arm folds clothing, as provided in this embodiment of the application. The set of key operational points includes the key operational points when the robotic arm grips the clothing, and the key operational points when the robotic arm lowers the clothing.

[0132] From the set of reference key points provided in the simulation environment (the set of key points composed of clothing in the standard reference example, the second pose data in the above embodiment), the reference key point corresponding to each operation key point in the operation key point set is determined. For example, the set of reference key points is determined by the following formula (3): (3) in, Represents the set of reference key points. Indicate each reference key point, ( ) represents the reference position of each reference key point, ( ) represents the reference attitude angle of each reference key point.

[0133] The loss function is a weighted average of the position and attitude angle errors of each operational key point and reference key point. For example, the loss function (the target loss value in the above embodiment) is determined by the following formula (4): (4) in, Represents the loss function. Indicates the number of key operation points. Indicates the location of the key operation point, ( () represents the attitude angle of the key operation point. Indicates the reference position of the reference key point, ( () represents the reference attitude angle of the reference key point. This represents the position error coefficient. This represents the attitude angle error coefficient, used to calibrate the position error and attitude angle error.

[0134] In step 904, the differentiable gradient of the loss function is calculated and the physical parameter generator neural network is trained.

[0135] Here, when training the physical parameter generator neural network, the physical parameter generator neural network is updated based on the gradient backpropagation (BP) algorithm. For example, the network parameters of the physical parameter generator neural network are updated by the following formula (5): (5) in, express The network parameters of the physical parameter generator neural network at time step. express The network parameters of the physical parameter generator neural network at time step. This represents the gradient of the loss function (the target loss gradient in the above embodiment).

[0136] In a simulation environment of differentiable dynamics, the gradient of the loss function can be automatically differentiated using a differentiable variable manager. A gradient computation graph is then built around the physical quantities related to the loss function (the network parameters in the above embodiment), thus obtaining the automatically differentiated gradients of these physical quantities, including the gradient of the parameter generator. The differentiable variable manager requires a computation graph to achieve automatic differentiation. A gradient computation graph is a graph-structured data structure that describes data relationships. Parameters that are computationally related are connected in the computation graph, and the differentiated gradient can be transferred between these connected parameters.

[0137] Example, reference Figure 14 , Figure 14 This is a schematic diagram of the gradient calculation graph provided in the embodiments of this application. The gradient calculation graph 1401 includes information related to joint angles such as real-time joint angle (joint_q), real-time joint angular velocity (joint_qd), initial joint angle (joint_q_start), and initial joint angular velocity (joint_qd_start), as well as different differential operators kernel0 and kernel1. kernel0 represents the differential operator for multiplication, while kernel1 represents the differential operator for exponentiation / root extraction.

[0138] While the differentiable simulation environment evolves forward, the gradient of the loss function is calculated by the engine and stored in the gradient calculation graph. This gradient is obtained based on the engine's analytical solver, which is much faster and more accurate than the gradient obtained by numerical difference methods in the non-differentiable environment.

[0139] In step 905, the trained physical parameter generator neural network is evaluated.

[0140] Here, after the physical parameter generator neural network has been trained, the convergence of the loss function is evaluated, and the performance of the robotic arm performing the clothing-folding task under optimized physical parameters is visualized. For example, see reference. Figure 15 , Figure 15This is a schematic diagram of the folding clothes task process after physical parameter optimization provided in the embodiments of this application. After an average of 12 rounds of training, the positional deviation can be converged to 0.3cm, and the average attitude angle deviation is 1.3 degrees. This indicates that the generated physical parameters and the reference parameters obtained through a large number of manual trials have a similar gain effect on completing the folding clothes task. Furthermore, after 10 repeated experiments, the average training optimization time is only 6.7s, and the time to generate physical parameters is 0.023s. In terms of parameter tuning efficiency, parameter tuning cost, and optimization difficulty, this method is superior to traditional methods.

[0141] In the aforementioned scenario of optimizing physical parameters in the simulation environment, by constructing a neural network for generating physical parameters based on differentiable dynamics, the automated and intelligent tuning of physical parameters in the simulation environment is achieved. This significantly reduces the cost and difficulty of parameter tuning, greatly improves the efficiency and generalization ability of parameter optimization, and enables rapid convergence to the optimal parameters. This effectively narrows the gap between simulation and reality, thereby significantly enhancing the training effect of the robot in the simulation environment and improving the success rate of transferring its task execution strategy to the real world. This provides key technical support for achieving more efficient and reliable intelligent robot training.

[0142] The following description continues to illustrate the exemplary structure of the network training device 455 provided in the embodiments of this application as a software module. In some embodiments, such as... Figure 2 As shown, the software modules stored in the network training device 455 in the memory 450 may include: a mapping processing module 4551, used to acquire the body data of the robotic arm and the initial state of the interactive object in the simulation environment, the simulation environment including the robotic arm and the interactive object; acquire the parameter generation network to be trained, and perform a preset number of training rounds on the parameter generation network to obtain the trained parameter generation network; for each round, during the process of the robotic arm performing the interactive task, the parameter generation network is called to perform mapping processing on the body data to obtain the collision mesh parameters, friction coefficient and stiffness coefficient of the interactive object; a first determination module 4552, used to determine the first pose data of each operation point of the robotic arm and the interactive object in the interaction process based on the collision mesh parameters, friction coefficient and stiffness coefficient; a second determination module 4553, used to determine the target loss gradient based on each first pose data and the preset second pose data of each reference point; and a parameter adjustment module 4554, used to adjust the network parameters of the parameter generation network using the target loss gradient and restore the interactive object to the initial state.

[0143] In some embodiments, the ontology data includes real-time joint angle vectors, end-effector pose vectors, and end-effector contact force vectors. The parameter generation network includes a first hidden layer, a second hidden layer, a first output layer, a second output layer, and a third output layer. The mapping processing module 4551 is further configured to concatenate the real-time joint angle vectors, end-effector pose vectors, and end-effector contact force vectors to obtain a concatenated vector, and call the weight matrix and bias matrix in the first hidden layer to perform nonlinear mapping processing on the concatenated vector to obtain a first processing result; call the weight matrix and bias matrix in the second hidden layer to perform nonlinear mapping processing on the first processing result to obtain a second processing result; call the weight matrix and bias matrix in the first output layer to perform nonlinear mapping processing on the second processing result to obtain collision mesh parameters; call the weight matrix and bias matrix in the second output layer to perform nonlinear mapping processing on the second processing result to obtain the friction coefficient; and call the weight matrix and bias matrix in the third output layer to perform nonlinear mapping processing on the second processing result to obtain the stiffness coefficient.

[0144] In some embodiments, the first determining module 4552 is further configured to update the state of the interactive object based on the collision mesh parameters, friction coefficient and stiffness coefficient, to obtain the updated interactive object; and during the process of the robotic arm performing the interactive task, to obtain the first pose data of each operation point when the robotic arm performs the interactive operation on the updated interactive object.

[0145] In some embodiments, the first pose data includes a first position and a first pose angle, and the second pose data includes a reference position and a reference pose angle. The second determining module 4553 is further configured to, for each operation point, determine the position error between the first position of the operation point and the reference position of the corresponding reference point, and determine the pose angle error between the first pose angle of the operation point and the reference pose angle of the corresponding reference point; perform fusion processing on the position error and the pose angle error to obtain the first loss value of the operation point; sum and average multiple first loss values ​​to obtain the target loss value; perform backpropagation on the target loss value to obtain the set of partial derivatives of the target loss function at the network parameters, and determine the set of partial derivatives as the target loss gradient.

[0146] In some embodiments, the second determining module 4553 is further configured to determine, from the network parameters of the parameter generation network, a first target parameter with a gradient and a second target parameter without a gradient; construct a gradient computation graph based on the first target parameter, the second target parameter and a preset differential operator, wherein the first node corresponding to the first target parameter and the second node corresponding to the second target parameter in the gradient computation graph are interconnected; in the gradient computation graph, the partial derivatives of the target loss function at each first target parameter are calculated layer by layer based on the target loss value, and a set of partial derivatives is constructed based on each partial derivative.

[0147] In some embodiments, the second determining module 4553 is further configured to construct a first node based on a first target parameter, and construct a second node based on a second target parameter; construct a directed acyclic graph based on the first node, the second node and a preset differential operator, and determine the directed acyclic graph as a gradient calculation graph.

[0148] In some embodiments, the parameter adjustment module 4554 is further configured to, after obtaining the trained parameter generation network, during the process of the robotic arm performing the interaction task, call the trained parameter generation network to perform mapping processing on the ontology data to obtain the target collision mesh parameters, target friction coefficient, and target stiffness coefficient of the interactive object; based on the target collision mesh parameters, target friction coefficient, and target stiffness coefficient, determine the third pose data of each operation point of the robotic arm and the interactive object during the interaction process; and evaluate the parameter generation effect based on each third pose data and the second pose data of each reference point to obtain the evaluation result.

[0149] In some embodiments, the parameter adjustment module 4554 is further configured to determine a second loss value based on each third pose data and each second pose data; when the second loss value is greater than or equal to a preset threshold, determine that the evaluation result is unsuccessful; acquire the body data of the robotic arm in the simulation environment again, and perform a preset number of training rounds on the trained parameter generation network based on the body data.

[0150] This application provides a computer program product, which includes computer-executable instructions or a computer program stored in a computer-readable storage medium. The processor of an electronic device reads the computer-executable instructions or computer program from the computer-readable storage medium and executes the computer-executable instructions or computer program, causing the electronic device to perform the network training method provided in this application.

[0151] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the network training method provided in this application. For example, ... Figure 3 The network training method is shown.

[0152] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0153] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0154] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0155] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0156] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A network training method, characterized in that, The method includes: The robot arm's ontological data and the initial state of the interactive object are obtained in a simulation environment, which includes the robot arm and the interactive object. Obtain the parameter generation network to be trained, and perform a preset number of training iterations on the parameter generation network to obtain the trained parameter generation network: For each round, during the process of the robotic arm performing the interactive task, the parameter generation network is invoked to map the body data and obtain the collision mesh parameters, friction coefficient and stiffness coefficient of the interactive object. Based on the collision mesh parameters, the friction coefficient, and the stiffness coefficient, the first pose data of the robotic arm and the interactive object at each operation point during the interaction process is determined; The target loss gradient is determined based on each of the first pose data and the second pose data of each preset reference point; The network parameters of the parameter generation network are adjusted using the target loss gradient, and the interactive object is restored to the initial state.

2. The method according to claim 1, characterized in that, The ontology data includes real-time joint angle vectors, end-effector pose vectors, and end-effector contact force vectors. The parameter generation network includes a first hidden layer, a second hidden layer, a first output layer, a second output layer, and a third output layer. The process of calling the parameter generation network to map the ontology data yields the collision mesh parameters, friction coefficient, and stiffness coefficient of the interactive object, including: The real-time joint angle vector, the end-effector pose vector, and the end-effector contact force vector are concatenated to obtain a concatenated vector. The weight matrix and bias matrix in the first hidden layer are then used to perform nonlinear mapping processing on the concatenated vector to obtain a first processing result. The weight matrix and bias matrix in the second hidden layer are used to perform a non-linear mapping process on the first processing result to obtain the second processing result. The weight matrix and bias matrix in the first output layer are called to perform nonlinear mapping on the second processing result to obtain the collision mesh parameters; The weight matrix and bias matrix in the second output layer are called to perform nonlinear mapping on the second processing result to obtain the friction coefficient. The weight matrix and bias matrix in the third output layer are called to perform nonlinear mapping on the second processing result to obtain the stiffness coefficient.

3. The method according to claim 1, characterized in that, The determination of the first pose data of each operation point between the robotic arm and the interactive object during the interaction process, based on the collision mesh parameters, the friction coefficient, and the stiffness coefficient, includes: Based on the collision mesh parameters, the friction coefficient, and the stiffness coefficient, the state of the interactive object is updated to obtain the updated interactive object; During the process of the robotic arm performing the interactive task, the first pose data of each operation point is obtained when the robotic arm performs the interactive operation on the updated interactive object.

4. The method according to claim 3, characterized in that, The first pose data includes a first position and a first pose angle, and the second pose data includes a reference position and a reference pose angle. The step of determining the target loss gradient based on each of the first pose data and the second pose data for each preset reference point includes: For each of the operation points, the positional error between the first position of the operation point and the reference position of the corresponding reference point is determined, and the attitude angle error between the first attitude angle of the operation point and the reference attitude angle of the corresponding reference point is determined. The position error and the attitude angle error are fused to obtain the first loss value of the operation point; The target loss value is obtained by summing and averaging multiple first loss values. Backpropagation is performed on the target loss value to obtain the set of partial derivatives of the target loss function at the network parameters, and the set of partial derivatives is determined as the target loss gradient.

5. The method according to claim 4, characterized in that, The backpropagation of the target loss value yields a set of partial derivatives of the target loss function at the network parameters, including: From the network parameters of the parameter generation network, determine the first objective parameter that has a gradient and the second objective parameter that does not have a gradient; A gradient computation graph is constructed based on the first target parameter, the second target parameter, and a preset differential operator. The first node corresponding to the first target parameter and the second node corresponding to the second target parameter in the gradient computation graph are interconnected. In the gradient calculation graph, the partial derivatives of the target loss function at each of the first target parameters are calculated layer by layer based on the target loss value, and the set of partial derivatives is constructed based on each of the partial derivatives.

6. The method according to claim 5, characterized in that, The construction of the gradient calculation graph based on the first target parameter, the second target parameter, and the preset differential operator includes: A first node is constructed based on the first target parameter, and a second node is constructed based on the second target parameter; A directed acyclic graph is constructed based on the first node, the second node, and the preset differential operator, and the directed acyclic graph is determined as the gradient calculation graph.

7. The method according to any one of claims 1 to 6, characterized in that, After obtaining the trained parameter generation network, the method further includes: During the process of the robotic arm performing the interactive task, the trained parameter generation network is invoked to map the ontology data and obtain the target collision mesh parameters, target friction coefficient and target stiffness coefficient of the interactive object. Based on the target collision mesh parameters, the target friction coefficient, and the target stiffness coefficient, the third pose data of the robotic arm and the interactive object at each operation point during the interaction process is determined; The parameter generation effect is evaluated based on each of the third pose data and the second pose data of each of the reference points, and the evaluation result is obtained.

8. The method according to claim 7, characterized in that, The parameter generation effect evaluation based on each of the third pose data and the second pose data of each of the reference points is used to obtain the evaluation result, including: A second loss value is determined based on each of the third pose data and each of the second pose data; When the second loss value is greater than or equal to a preset threshold, the evaluation result is determined to be unsuccessful. The robot arm's ontological data in the simulation environment is acquired again, and the trained parameter generation network is trained for the preset number of rounds based on the ontological data.

9. A network training device, characterized in that, The device includes: The mapping processing module is used to acquire the body data of the robotic arm and the initial state of the interactive object in the simulation environment, the simulation environment including the robotic arm and the interactive object; acquire the parameter generation network to be trained, and perform a preset number of training rounds on the parameter generation network to obtain the trained parameter generation network; for each round, during the process of the robotic arm performing the interactive task, the parameter generation network is called to perform mapping processing on the body data to obtain the collision mesh parameters, friction coefficient and stiffness coefficient of the interactive object; The first determining module is used to determine the first pose data of each operation point of the robotic arm and the interactive object during the interaction process based on the collision mesh parameters, the friction coefficient and the stiffness coefficient; The second determining module is used to determine the target loss gradient based on each of the first pose data and the second pose data of each preset reference point; The parameter adjustment module is used to adjust the network parameters of the parameter generation network using the target loss gradient and restore the interactive object to the initial state.

10. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the network training method according to any one of claims 1 to 8.

11. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the network training method according to any one of claims 1 to 8.

Images