Joint angle prediction model training method and device, equipment and storage medium

Abstract

This application discloses a training method, apparatus, device, and storage medium for a joint angle prediction model, relating to the field of computer technology. The method includes: acquiring training sample data, which includes skeletal parameters, sample wrist joint pose sequences, and arm angle labels for the sample wrist joint pose sequences; obtaining arm angle feature information based on the sample wrist joint pose sequences using a selective state-space network of the joint angle prediction model to be trained; obtaining elbow position prediction information based on the arm angle feature information and skeletal parameters using a differentiable kinematic layer of the joint angle prediction model to be trained; and iteratively updating and training the joint angle prediction model to be trained based on a preset objective function, the training sample data, the arm angle feature information, and the elbow position prediction information to obtain a trained joint angle prediction model.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to the fields of robot control technology, data processing technology, etc., and particularly to a training method, apparatus, device, and storage medium for a joint angle prediction model. Background Technology

[0002] In recent years, teleoperated robots based on virtual reality (VR) have been widely used. The key issue of this technology is to obtain faster, more accurate and more human-like robot arm joint angles through end-joint pose.

[0003] Currently, methods for predicting joint angles in robotic arms include traditional numerical iterative prediction, data-driven model prediction, and time-series modeling prediction. However, these methods lack modeling of rigid body kinematics and physical constraints, making them prone to structural distortion and trajectory jitter. They also struggle to guarantee the absolute accuracy of the end effector, exhibiting centimeter-level steady-state errors. Furthermore, they cannot effectively select solutions that conform to human movement habits, resulting in overly mechanical robotic arm movements and a lack of modeling for trajectory smoothness. Summary of the Invention

[0004] This application provides a training method, apparatus, device, and storage medium for a joint angle prediction model, which can solve the problem of poor prediction performance and accuracy of robot joint angle prediction models. The technical solution is as follows: Firstly, a training method for a joint angle prediction model is provided, the method comprising: Acquire training sample data, which includes skeletal parameters, sample wrist joint pose sequences, and arm angle labels of the sample wrist joint pose sequences; Based on the sample wrist joint pose sequence, arm angle feature information is obtained using the selective state space network of the joint angle prediction model to be trained. Based on the arm angle feature information and skeletal parameters, the elbow position prediction information is obtained by utilizing the differentiable kinematic layer of the joint angle prediction model to be trained. Based on the preset objective function, the training sample data, arm angle feature information, and elbow position prediction information, the joint angle prediction model to be trained is iteratively updated and trained to obtain the trained joint angle prediction model.

[0005] In one possible implementation, obtaining arm angle feature information based on the sample wrist joint pose sequence using the selective state-space network of the joint angle prediction model to be trained includes: Based on the sample wrist joint pose sequence, high-dimensional feature information is obtained using the input network of the joint angle prediction model to be trained. Based on the high-dimensional feature information, arm angle feature information is obtained by utilizing the selective state-space network of the joint angle prediction model to be trained.

[0006] In one possible implementation, the input network includes an input layer and a one-dimensional convolutional layer. The step of obtaining high-dimensional feature information based on the sample wrist joint pose sequence using the input network of the joint angle prediction model to be trained includes: Based on the sample wrist joint pose sequence, high-dimensional latent feature information is obtained using the input layer; The high-dimensional latent feature information is smoothed using the one-dimensional convolutional layer to obtain high-dimensional feature information.

[0007] In one possible implementation, the selective state-space network includes a backbone network and a decoding network. The step of obtaining arm angle feature information based on the high-dimensional feature information using the selective state-space network of the joint angle prediction model to be trained includes: Based on the high-dimensional feature information, global temporal deep feature information is obtained using the backbone network; Based on the global temporal depth feature information, the arm angle feature information is obtained using the decoding network.

[0008] In one possible implementation, the backbone network includes multiple selective state space layers, and the step of obtaining global temporal deep feature information based on the high-dimensional feature information using the backbone network includes: Multiple selective state space layers are used to perform feature extraction and residual connection processing on the high-dimensional feature information; Based on the results of feature extraction and residual connection processing, global temporal depth feature information is obtained.

[0009] In one possible implementation, the preset objective function includes a position constraint objective function, an arm angle constraint objective function, and a smoothness constraint objective function. The step of iteratively updating and training the joint angle prediction model to be trained based on the preset objective function, the training sample data, arm angle feature information, and elbow position prediction information to obtain a trained joint angle prediction model includes: Based on the position constraint objective function, training sample data, and elbow position prediction information, the position constraint loss value is determined. Based on the arm angle constraint objective function, training sample data, and arm angle feature information, determine the arm angle constraint loss value; Based on the smoothing constraint objective function and arm angle feature information, the smoothing constraint loss value is determined; Based on the position constraint loss value, arm angle constraint loss value, and smoothness constraint loss value, the joint angle prediction model to be trained is iteratively updated and trained until the preset training termination condition is met, and the trained joint angle prediction model is obtained.

[0010] Secondly, a method for determining the joint angles of a robot arm is provided, the method comprising: Acquire the end-effector pose data to be processed; The end pose data to be processed is input into the joint angle prediction model trained by the method described above, so as to obtain arm angle feature information by utilizing the selective state space network of the joint angle prediction model. Based on the arm angle feature information, the elbow position prediction information is obtained by using the differentiable kinematic layer of the joint angle prediction model. The elbow position prediction information is numerically optimized to determine the robot arm joint angle corresponding to the end pose data to be processed.

[0011] Thirdly, a training device for a joint angle prediction model is provided, the device comprising: The first acquisition unit is used to acquire training sample data, which includes skeletal parameters, sample wrist joint pose sequences, and arm angle labels of the sample wrist joint pose sequences. The first obtaining unit is used to obtain arm angle feature information based on the sample wrist joint pose sequence and using the selective state space network of the joint angle prediction model to be trained. The second obtaining unit is used to obtain elbow position prediction information based on the arm angle feature information and skeletal parameters, using the differentiable kinematic layer of the joint angle prediction model to be trained. The first training unit is used to iteratively update and train the joint angle prediction model to be trained based on a preset objective function, the training sample data, arm angle feature information, and elbow position prediction information, so as to obtain a trained joint angle prediction model.

[0012] Fourthly, a device for determining the joint angle of a robot arm is provided, the device comprising: The second acquisition unit is used to acquire the end-effector pose data to be processed. The third obtaining unit is used to input the end pose data to be processed into the joint angle prediction model trained by the device described above, so as to obtain arm angle feature information by utilizing the selective state space network of the joint angle prediction model. The fourth obtaining unit is used to obtain elbow position prediction information based on the arm angle feature information and using the differentiable kinematic layer of the joint angle prediction model. The first optimization unit is used to perform numerical optimization processing on the elbow position prediction information to determine the robot arm joint angle corresponding to the end pose data to be processed.

[0013] Fifthly, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement the aspects described above and any possible implementation thereof.

[0014] Sixthly, an electronic device is provided, comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the methods described above and any possible implementations.

[0015] In a seventh aspect, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the aspects and any possible implementations described above.

[0016] The beneficial effects of the technical solution provided in this application include at least the following: As can be seen from the above technical solution, on the one hand, the embodiments of this application can obtain training sample data, which includes skeletal parameters, sample wrist joint pose sequences, and arm angle labels of the sample wrist joint pose sequences. Then, based on the sample wrist joint pose sequences, arm angle feature information can be obtained using the selective state space network of the joint angle prediction model to be trained. Based on the arm angle feature information and skeletal parameters, elbow position prediction information can be obtained using the differentiable kinematics layer of the joint angle prediction model to be trained. Based on the preset objective function, the training sample data, arm angle feature information, and elbow position prediction information, the joint angle prediction model to be trained is iteratively updated and trained to obtain a trained joint angle prediction model. Since the selective state space network of the trained joint angle prediction model can achieve global temporal modeling with linear complexity, and the human kinematic priors are embedded into the model network through the differentiable kinematics layer, the physical consistency of the model prediction results is guaranteed. This can improve the learning ability of the trained joint angle prediction model on wrist joint pose sequences and the prediction performance of joint angles, thereby optimizing the training effect of the joint angle prediction model.

[0017] As can be seen from the above technical solution, on the other hand, the embodiments of this application can obtain the end-effector pose data to be processed, input the end-effector pose data to be processed into the joint angle prediction model, and use the selective state space network of the joint angle prediction model to obtain arm angle feature information. Based on the arm angle feature information, the elbow position prediction information is obtained using the differentiable kinematic layer of the joint angle prediction model. The elbow position prediction information is numerically optimized to determine the robot arm joint angle corresponding to the end-effector pose data to be processed. Since the end-effector pose data can be processed by using the joint angle prediction model, the accuracy and real-time performance of the predicted robot arm joint angle are improved, the movement error of the wrist and elbow is reduced, and smoother and more natural humanoid robot upper limb movements can be generated, thereby ensuring the reliability of robot upper limb movements.

[0018] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating a training method for a joint angle prediction model provided in one embodiment of this application. Figure 2 This is a flowchart illustrating a method for determining the joint angle of a robot arm according to another embodiment of this application; Figure 3 This is a flowchart illustrating a method for determining the joint angle of a robot arm according to another embodiment of this application; Figure 4 This is a structural block diagram of a training device for a joint angle prediction model provided in another embodiment of this application; Figure 5 This is a structural block diagram of a device for determining the joint angle of a robot arm provided in another embodiment of this application. Detailed Implementation

[0021] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application, including various details to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0022] Obviously, the described embodiments are only some, not all, of the embodiments in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0023] It should be noted that the hardware observation equipment terminal involved in the embodiments of this application may include, but is not limited to, equipment in meteorological observation stations such as receivers, integrated processors, remote control devices, and base measurement boxes.

[0024] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0025] Please refer to Figure 1 This document illustrates a flowchart of a training method for a joint angle prediction model provided in one embodiment of this application. The training method for this joint angle prediction model may specifically include: Step 101: Obtain training sample data, which includes skeletal parameters, sample wrist joint pose sequences, and arm angle labels of the sample wrist joint pose sequences.

[0026] Step 102: Based on the sample wrist joint pose sequence, obtain arm angle feature information using the selective state space network of the joint angle prediction model to be trained.

[0027] Step 103: Based on the arm angle feature information and skeletal parameters, obtain elbow position prediction information using the differentiable kinematic layer of the joint angle prediction model to be trained.

[0028] Step 104: Based on the preset objective function, the training sample data, arm angle feature information, and elbow position prediction information, iteratively update and train the joint angle prediction model to be trained to obtain the trained joint angle prediction model.

[0029] It should be noted that the training sample data can be data collected according to a preset sliding window.

[0030] It should be noted that the sample wrist joint pose sequence can be an end-effector pose sequence. Skeletal parameters may include fixed shoulder joint coordinates, fixed wrist joint coordinates, upper arm length, forearm length, and fixed trunk reference vector, etc. The arm angle can be a rotation angle.

[0031] It should be noted that the sample wrist joint pose sequence can be determined based on the six-dimensional wrist joint pose sequence. The six-dimensional wrist joint pose can include position information (x, y, z) and attitude information (roll, pitch, yaw).

[0032] It should be noted that the selective state space network can be a Mamba network.

[0033] It should be noted that the joint angle prediction model to be trained may include a selective state-space network and a differentiable kinematics layer. Here, the differentiable kinematics layer can be a physical anchor point in the prediction model. It may not contain trainable parameters and can be part of the computational graph of the joint angle prediction model to be trained. It converts the rotation angle output by the selective state-space network into physically reasonable three-dimensional position coordinates of the elbow, and backpropagates the gradient of the spatial error to the network parameters of the model through its differentiable geometric operations, thereby enabling the model's network to internalize rigid body kinematics priors during iterative training.

[0034] It should be noted that some or all of the execution entities in steps 101 to 104 may be applications located on the local terminal, or may be functional units such as plug-ins or software development kits (SDKs) set in applications located on the local terminal, or may be processing engines located on network-side servers, or may be distributed systems located on the network side, such as processing engines or distributed systems in network-side model training platforms, etc. This embodiment does not impose any special limitations on these.

[0035] It is understood that the application can be a native program installed on the local terminal, or it can be a web application of a browser on the local terminal. This embodiment does not limit this.

[0036] In this way, global temporal modeling with linear complexity can be achieved through the selective state-space network of the trained joint angle prediction model. Furthermore, by embedding human kinematic priors into the model network through a differentiable kinematic layer, the physical consistency of the model's prediction results is ensured. This improves the trained joint angle prediction model's ability to learn wrist joint pose sequences and its performance in predicting joint angles, thereby optimizing the training effect of the joint angle prediction model.

[0037] Optionally, in one possible implementation of this embodiment, in step 102, firstly, high-dimensional feature information can be obtained based on the sample wrist joint pose sequence using the input network of the joint angle prediction model to be trained. Secondly, arm angle feature information can be obtained based on the high-dimensional feature information using the selective state space network of the joint angle prediction model to be trained.

[0038] In this implementation, the input network may include an input layer and a one-dimensional convolutional layer.

[0039] In a specific implementation of this method, firstly, high-dimensional latent feature information can be obtained using the input layer based on the sample wrist joint pose sequence. Secondly, the high-dimensional latent feature information can be smoothed using the one-dimensional convolutional layer to obtain high-dimensional feature information.

[0040] Here, the sample wrist joint pose sequence can be represented as a nine-dimensional wrist joint pose sequence. This avoids the singularity problems of Euler angles and quaternions.

[0041] In this specific implementation, one-dimensional convolutional layers can employ a copy padding strategy to avoid the physical continuity of feature information being disrupted by boundary jumps.

[0042] In this implementation, the selective state space network may include a backbone network and a decoding network.

[0043] In another specific implementation of this approach, firstly, global temporal depth feature information can be obtained using the backbone network based on the high-dimensional feature information. Secondly, arm angle feature information can be obtained using the decoding network based on the global temporal depth feature information.

[0044] In this implementation, the backbone network includes multiple selective state space layers.

[0045] One specific implementation involves first using multiple selective state space layers to perform feature extraction and residual connection processing on the high-dimensional feature information. Secondly, based on the results of the feature extraction and residual connection processing, global temporal deep feature information can be obtained.

[0046] Preferably, the number of selective state space layers can be four.

[0047] In this implementation, the decoding network may include a multilayer sensing mechanism output head layer and an L2 normalization layer.

[0048] Another specific implementation involves inputting global temporal depth feature information into the decoding network of a selective state space network, using a multilayer perceptron mechanism to output the head layer, obtaining two-dimensional feature vector information, and then using an L2 normalization layer to normalize the two-dimensional feature vector information to obtain arm angle feature information.

[0049] Here, the arm angle feature information can be a feature representation of the arm angle, and the arm angle feature information can be... arm angle It can be calculated based on arm angle feature information.

[0050] In this way, more accurate and effective arm angle feature information can be obtained by using the input network and selective state space network of the joint angle prediction model to be trained, based on the sample wrist joint pose sequence, thereby further improving the learning ability of the joint angle prediction model to be trained.

[0051] Optionally, in one possible implementation of this embodiment, in step 103, the arm angle feature information and skeletal parameters are input into the differentiable kinematics layer of the joint angle prediction model to be trained, and the elbow position prediction information is calculated using a preset position algorithm.

[0052] In this implementation, the skeletal parameters may include fixed position coordinates of the shoulder joint, fixed position coordinates of the wrist joint, upper arm length, forearm length, and a fixed reference vector of the torso. The fixed position coordinates of the shoulder joint may include the actual three-dimensional position coordinates of the shoulder joint. The fixed position coordinates of the wrist joint may include the actual three-dimensional position coordinates of the wrist joint.

[0053] In a specific implementation of this method, firstly, the projected distance can be calculated using a preset distance algorithm based on the fixed coordinates of the shoulder joint, the fixed coordinates of the wrist joint, the upper arm length, and the forearm length. Secondly, the orbital radius can be calculated using a preset radius algorithm based on the fixed coordinates of the shoulder joint and the projected distance. Thirdly, a local orthogonal basis for constructing the orbital plane can be determined using a preset orthogonal basis algorithm based on a fixed reference vector of the torso. Finally, the elbow position prediction information is calculated using a preset position algorithm based on the arm angle feature information, the fixed coordinates of the shoulder joint and the fixed coordinates of the wrist joint, the projected distance, the orbital radius, and the local orthogonal basis.

[0054] Here, the elbow position prediction information can include the three-dimensional position information of the elbow.

[0055] In this way, by utilizing differentiable kinematic layers, based on the predicted arm angle features, shoulder-wrist axis, upper arm / forearm length and other skeletal parameters, a more accurate and effective 3D elbow position can be obtained, thereby ensuring the physical consistency of the learned feature information.

[0056] Optionally, in one possible implementation of this embodiment, the preset objective function may include a position constraint objective function, an arm angle constraint objective function, and a smoothing constraint objective function. In step 104, firstly, a position constraint loss value can be determined based on the position constraint objective function, training sample data, and elbow position prediction information. Secondly, an arm angle constraint loss value can be determined based on the arm angle constraint objective function, training sample data, and arm angle feature information. Thirdly, a smoothing constraint loss value can be determined based on the smoothing constraint objective function and arm angle feature information. Finally, the joint angle prediction model to be trained can be iteratively updated and trained based on the position constraint loss value, arm angle constraint loss value, and smoothing constraint loss value until a preset training termination condition is met, resulting in a trained joint angle prediction model.

[0057] In a specific implementation of this method, firstly, the total loss value can be calculated using a preset objective function based on the position constraint loss value, arm angle constraint loss value, and smoothing constraint loss value. Secondly, based on the total loss value, the training sample data, arm angle feature information, and elbow position prediction information, the joint angle prediction model to be trained can be iteratively updated and trained until a preset training termination condition is met, resulting in a trained joint angle prediction model.

[0058] In this implementation, the preset training termination condition may include the total loss value satisfying a preset loss condition.

[0059] In this way, the model can be iteratively updated and trained using the joint loss of physical information such as position constraint objective function, arm angle constraint objective function and smoothness constraint objective function. This achieves the combination of spatial constraints and second-order smoothness constraints, thereby improving the physical rationality and hardware safety of the robot arm trajectory.

[0060] It should be noted that the specific implementation process provided in this embodiment can be combined with various specific implementation processes provided in the aforementioned implementation methods to implement the training method of the joint angle prediction model in this embodiment. Detailed descriptions can be found in the relevant content of the aforementioned implementation methods, and will not be repeated here.

[0061] Figure 2 This is a flowchart illustrating a method for determining the joint angle of a robot arm according to another embodiment of this application, as shown below. Figure 2 As shown.

[0062] Step 201: Obtain the end-effector pose data to be processed.

[0063] Step 202: Input the end pose data to be processed into the joint angle prediction model to obtain arm angle feature information by utilizing the selective state space network of the joint angle prediction model.

[0064] Step 203: Based on the arm angle feature information, obtain elbow position prediction information using the differentiable kinematic layer of the joint angle prediction model.

[0065] Step 204: Perform numerical optimization processing on the elbow position prediction information to determine the robot arm joint angle corresponding to the end pose data to be processed.

[0066] It should be noted that the end-effector pose data can include the end-effector pose data output by the VR controller. The end-effector pose data can be an end-effector pose sequence. The end-effector pose data can include the target pose of the robot wrist joint.

[0067] It should be noted that the execution entities of steps 201 to 204 may be part or all of the applications located on the local terminal, or may be functional units such as plug-ins or software development kits (SDKs) set in the applications located on the local terminal, or may be the processing engine located on the network-side server, or may be the distributed system located on the network side, such as the processing engine or distributed system in the robot arm joint angle determination platform on the network side. This embodiment does not impose any special limitations on this.

[0068] It is understood that the application can be a native program installed on the local terminal, or it can be a web application of a browser on the local terminal. This embodiment does not limit this.

[0069] Optionally, in one possible implementation of this embodiment, in step 204, a numerical optimization algorithm is used to perform numerical optimization processing on the elbow position prediction information to determine the robot arm joint angle corresponding to the end pose data to be processed.

[0070] Optionally, in one possible implementation of this embodiment, the joint angle prediction model may further include an input network. The end-effector pose data is input into the input network, and high-dimensional feature information corresponding to the end-effector pose data is output.

[0071] Here, the end-effector pose data can be six-dimensional end-effector pose data, or the end-effector pose data can be preprocessed to obtain nine-dimensional end-effector pose data.

[0072] Optionally, in one possible implementation of this embodiment, the robot arm joint angles may include seven degrees of freedom. The seven degrees of freedom may include three shoulder joints, one elbow joint, and three wrist joints.

[0073] In this way, by acquiring the end-effector pose data to be processed, and inputting the end-effector pose data into the joint angle prediction model, the arm angle feature information can be obtained by utilizing the selective state-space network of the joint angle prediction model. Based on the arm angle feature information, the elbow position prediction information can be obtained by utilizing the differentiable kinematic layer of the joint angle prediction model. The elbow position prediction information is then numerically optimized to determine the robot arm joint angle corresponding to the end-effector pose data to be processed. Since the end-effector pose data can be processed by using the joint angle prediction model, the accuracy and real-time performance of the predicted robot arm joint angles are improved, the motion errors of the wrist and elbow are reduced, and smoother and more natural human-like robot upper limb movements can be generated, thereby ensuring the reliability of robot upper limb movements.

[0074] To better understand the methods of the embodiments of this application, the methods of the embodiments of this application will be described below in conjunction with the accompanying drawings and specific application scenarios.

[0075] Figure 3 This is a flowchart illustrating a method for determining the joint angle of a robot arm according to another embodiment of this application, as shown below. Figure 3 As shown. In this embodiment, in this application scenario, the preset joint angle prediction model may include an input network, a selective state-space network, and a differentiable kinematics layer. The training method for this joint angle prediction model may include: Step 301: Obtain training sample data, which includes skeletal parameters, sample wrist joint pose sequences, and arm angle labels of the sample wrist joint pose sequences.

[0076] Step 302: Input the training sample data into the joint angle prediction model to be trained. The joint angle prediction model to be trained includes an input network, a selective state space network, and a differentiable kinematic layer.

[0077] Step 303: Based on the sample wrist joint pose sequence, use the input network of the joint angle prediction model to be trained to obtain high-dimensional feature information.

[0078] In this embodiment, the sample wrist joint pose sequence can be a continuous nine-dimensional wrist joint pose representation.

[0079] Here, the initial six-dimensional wrist joint pose sequence can be obtained, and the six-dimensional wrist joint pose sequence can be transformed to obtain the sample wrist joint pose sequence.

[0080] Preferably, the six-dimensional wrist joint pose sequence can be transformed based on a preset rotation matrix to obtain a transformed matrix. Then, based on the six-dimensional wrist joint pose sequence and the first two columns of the transformed matrix, a sample wrist joint pose sequence can be obtained.

[0081] In this embodiment, the input network may include an input layer and a one-dimensional convolutional layer.

[0082] Here, the sample wrist joint pose sequence is passed through the input layer to obtain the high-dimensional latent feature information corresponding to the sample wrist joint pose sequence, and then the high-dimensional latent feature information is passed through a one-dimensional convolutional layer to obtain high-dimensional feature information.

[0083] Step 304: Based on high-dimensional feature information, obtain global temporal deep feature information using the backbone network of the selective state space network.

[0084] In this embodiment, the backbone network of the selective state space network can be the Mamba backbone network. The backbone network of the selective state space network can include multiple selective state space layers.

[0085] Preferably, the number of selective state space layers can be four.

[0086] In this embodiment, multiple selective state space layers are used to perform feature extraction and residual connection processing on high-dimensional feature information. Based on the results of feature extraction and residual connection processing, global temporal depth feature information is obtained.

[0087] Preferably, firstly, the high-dimensional feature information is processed through a first selective state space layer to obtain the result of residual connection processing and first intermediate feature information. Based on the result of residual connection processing and the first intermediate feature information, first fused feature information is obtained. Secondly, the first fused feature information is processed through a second selective state space layer to obtain the result of residual connection processing and second intermediate feature information. Based on the result of residual connection processing and the second intermediate feature information, second fused feature information is obtained. Thirdly, the second fused feature information is processed through a third selective state space layer to obtain the result of residual connection processing and third intermediate feature information. Based on the result of residual connection processing and the third intermediate feature information, third fused feature information is obtained. Finally, the third fused feature information is processed through a fourth selective state space layer to obtain the result of residual connection processing and fourth intermediate feature information. Based on the result of residual connection processing and the fourth intermediate feature information, global temporal depth feature information is obtained.

[0088] Here, global temporal deep feature information can include high-dimensional feature information and global temporal feature information. High-dimensional feature information can be the original local temporal features.

[0089] Understandably, the result of residual connection processing in each selective state space layer can be the input feature information of that layer. The superposition of residuals in each of multiple selective state space layers ensures that the global temporal deep feature information always retains its memory and information of shallower features. Thus, performing residual connection processing on the features input to the Mamba backbone network yields a fused and enhanced feature of the original features and the global temporal features extracted by Mamba. This preserves low-level information while incorporating high-level semantics, and ensures smooth gradient backpropagation.

[0090] Step 305: Based on global temporal depth feature information, obtain arm angle feature information using the decoding network of a selective state space network.

[0091] In this embodiment, the decoding network may include a multilayer sensing mechanism output head layer and an L2 normalization layer.

[0092] Firstly, the global temporal depth feature information is input into the decoding network of the selective state space network. After passing through the multilayer perception mechanism, the head layer is output to obtain the intermediate feature information corresponding to the global temporal depth feature information. Then, the intermediate feature information corresponding to the global temporal depth feature information is passed through the L2 normalization layer to obtain the arm angle feature information.

[0093] In this embodiment, the intermediate feature information corresponding to the global temporal depth feature information can be two-dimensional feature vector information, i.e., the original two-dimensional vector. The arm angle feature information can be the feature representation of the arm angle.

[0094] For example, the arm angle feature information can be Based on global temporal depth feature information, the arm angle feature information is obtained by using the decoding network of the selective state space network, which can be specifically expressed as formula (1): (1) in, It can be arm angle feature information, The arm angle, or swivel angle, predicted by the network can be calculated using arm angle feature information. It can output the original two-dimensional vector of the head layer of the multilayer sensing mechanism, that is, the intermediate feature information. It can be a preset coefficient, or it can be a minimum value, such as... It can be used to prevent the denominator from being zero, thus avoiding numerical crashes during calculations.

[0095] Understandably, in a 7-DoF robotic arm, when the shoulder and wrist positions are fixed, the elbow can rotate freely on a circular ring; this rotation angle is the arm angle. If the network directly predicts a scalar angle, such as directly outputting 180° or -180°, it is prone to drastic numerical jumps at the angle boundaries, leading to training non-convergence. Therefore, here the multilayer perceptron outputs a two-dimensional vector from the head layer. Then, it is subjected to forced L2 normalization to accurately map it onto the unit circle, thereby obtaining continuous and valid [data / form]. .

[0096] Step 306: Based on arm angle feature information and skeletal parameters, obtain elbow position prediction information using the differentiable kinematic layer of the joint angle prediction model to be trained.

[0097] In this embodiment, skeletal parameters may include fixed position coordinates of the shoulder joint, fixed position coordinates of the wrist joint, upper arm length, forearm length, and fixed reference vector of the torso.

[0098] In this embodiment, based on arm angle feature information and skeletal parameters, the elbow position prediction information is calculated using the differentiable kinematic layer of the joint angle prediction model to be trained, as shown in formulas (2) to (5): First, based on the fixed position coordinates of the shoulder joint, the fixed position coordinates of the wrist joint, the upper arm length, and the forearm length, the projected distance d can be calculated using a preset distance algorithm, specifically expressed as formula (2): (2) in, It can be the length of the upper arm. It can be the length of the forearm. The coordinates of the shoulder position can be fixed. You can set the coordinates of the wrist position.

[0099] Understandable, It can be the projected distance along axis n from the shoulder to the center of the track. It can be the same as the wrist joint position information in the wrist joint pose sequence.

[0100] Secondly, based on the fixed position coordinates and projection distance of the shoulder joint, the radius of the three-dimensional orbital circle in which the elbow rotates can be calculated using a preset radius algorithm, i.e., the orbital radius. Specifically, this is expressed as formula (3): (3) in, It can be the length of the upper arm. It can be the projected distance along axis n from the shoulder to the center of the track. An adjustment coefficient can be preset, which can be a minimum value, such as... It can be used to prevent the denominator from being zero, thus avoiding numerical crashes during calculations.

[0101] Furthermore, based on a fixed reference vector of the torso, a pre-defined orthogonal basis algorithm can be used to determine the local orthogonal basis constructed from the orbital plane. , Specifically, this is expressed as formula (4): (4) in, A fixed reference vector can be used to establish a reference plane with a rotation angle of 0°. This can be the direction vector from the shoulder to the wrist, i.e., the unit direction axis. , It is a locally orthogonal basis.

[0102] Furthermore, based on arm angle feature information, fixed position coordinates of shoulder joint and wrist joint, projection distance, orbit radius, and local orthogonal basis, the elbow position prediction information is calculated using a preset position algorithm. Specifically, this is expressed as formula (5): (5) Among them, The wrist can be fixed at coordinates. It can be the projected distance along axis n from the shoulder to the center of the track. It can be a unit direction axis. The orbital radius can be used, and u and v can be local orthogonal bases. It can be the arm angle feature information.

[0103] Step 307: Based on the position constraint objective function, arm angle constraint objective function, smoothness constraint objective function, training sample data, arm angle feature information and elbow position prediction information, iteratively update the joint angle prediction model to be trained until the preset training termination condition is met, and obtain the trained joint angle prediction model.

[0104] In this embodiment, the preset objective function may include a position constraint objective function, an arm angle constraint objective function, and a smoothness constraint objective function.

[0105] Here, firstly, the position constraint loss value can be determined based on the position constraint objective function, training sample data, and elbow position prediction information. Secondly, the arm angle constraint loss value can be determined based on the arm angle constraint objective function, training sample data, and arm angle feature information. Thirdly, the smoothing constraint loss value can be determined based on the smoothing constraint objective function and arm angle feature information. Finally, based on the position constraint loss value, arm angle constraint loss value, and smoothing constraint loss value, the joint angle prediction model to be trained is iteratively updated and trained until a preset training termination condition is met, resulting in a trained joint angle prediction model.

[0106] Preferably, the preset objective function can be used to calculate the total loss value. Specifically, it can be expressed as formula (6): (6) in, These can be weighted hyperparameters, which can balance the relative importance of the three loss terms. For example, if the elbow position is considered more important, then... Increase the size. This can be the arm angle constraint loss value. It can be a position constraint loss value. It can be a smoothing constraint loss value, i.e., jerk, which can be used to monitor the temporal changes of the arm angle.

[0107] Furthermore, the objective function for position constraints can be specifically expressed as formula (7): (7) in, This can be the position constraint loss value, i.e. This can represent the root mean square error loss at the elbow position. It can be a normalization factor. Batch indexing is possible. It can be indexed by time step. Mask weights can be used. It can provide information for predicting elbow position. It can provide accurate elbow position information. The coefficients can be preset.

[0108] The objective function for arm angle constraint can be expressed as formula (8): (8) in, This can be the arm angle constraint loss value. It can be a normalization factor. Batch indexing is possible. It can be indexed by time step. Mask weights can be used. It can be used to predict the arm angle. This can be the true value of the arm angle, i.e., the arm angle label. Here, the objective function for arm angle constraints uses the L1 norm to calculate the mean absolute error between the predicted arm angle and the true arm angle.

[0109] The objective function for smoothing constraints can be expressed as formula (9): (9) in, The loss value can be a smoothing constraint. It can be a normalization factor. Batch indexing is possible. It can be indexed by time step. The time window size can be set. Mask weights can be used. It can be the first The sample in the The arm angle at frame time.

[0110] Understandably, the predefined objective function can be a function representing the joint loss of physical information. The arm angle constraint objective function can be used to align the predicted arm angle with the true value. The position constraint objective function can be the MSE loss calculated from the elbow's three-dimensional position coordinates. The smoothness constraint objective function can be a second-order difference constraint to suppress high-frequency jitter.

[0111] Step 308: Obtain the end-effector pose data to be processed.

[0112] In this embodiment, the end-effector pose data can be the end-effector pose sequence output by the VR controller.

[0113] Step 309: Input the end pose data to be processed into the joint angle prediction model to obtain arm angle feature information by utilizing the selective state space network of the joint angle prediction model.

[0114] Step 310: Based on the arm angle feature information, use the differentiable kinematic layer of the joint angle prediction model to obtain the elbow position prediction information.

[0115] Step 311: Use a numerical optimization algorithm to perform numerical optimization processing on the elbow position prediction information in order to determine the robot arm joint angle corresponding to the end pose data to be processed.

[0116] In this embodiment, the numerical optimization solution algorithm can be a numerical optimization solver.

[0117] Preferably, the joint angles of the robot arm can be calculated using numerical optimization algorithms based on the end-effector pose data and elbow position prediction information, specifically expressed as formula (10): (10) in, It can be used to adjust the joint angles of the robot arm. This can be the reaching error. It can be the positive kinematic mapping function of the end effector. It can be a manifold penalty. It can be the positive kinematic mapping function of the elbow joint. These can be weight hyperparameters. Candidate joint angles can be used, i.e., optimization variables. It can represent the joint configuration of a 7-DOF arm. The target six-dimensional pose that the end effector expects to achieve ( ∈SE(3)), It can predict information about the elbow position, i.e., the absolute three-dimensional elbow position.

[0118] Understandably, the arrival error is the deviation between the actual pose of the robotic arm's end effector and the target pose, used to ensure tracking accuracy in Cartesian space. The forward kinematics mapping function of the end effector represents the actual pose of the robotic arm's end effector calculated using forward kinematics at the current joint angle. The manifold penalty forces the solved pose to closely approximate the predicted anthropomorphic manifold space, i.e., the elbow position must be aligned, thus greatly reducing the dimensionality of blind search and effectively preventing severe jitter caused by singularities. The forward kinematics mapping function of the elbow joint represents the calculated actual 3D spatial position of the robotic arm's elbow at the current joint angle. The weight hyperparameter is used to balance the priority between end effector Cartesian tracking accuracy (first half of the formula) and anthropomorphic pose regularization (second half of the formula). Elbow position prediction information is input into the solver here as a powerful spatial prior and regularization term.

[0119] Here, the architecture for determining the joint angles of the robotic arm can be based on a humanoid inverse kinematics Mamba network (PiM-HL-IK) architecture.

[0120] Thus, based on the technical solution of this embodiment, the joint angle prediction model can be trained to process the end pose data to be processed, thereby improving the efficiency and accuracy of robot joint angle prediction. It can achieve a total inference latency of about 2.2ms, support high-frequency control of 400-500Hz, and enable the robot hand, i.e., the end effector error to reach the sub-millimeter level, and the elbow error to be less than 12mm.

[0121] Moreover, in the technical solution of this embodiment, the joint angle prediction model achieves the combination of spatial constraints and second-order smoothness constraints through the loss function of differentiable kinematics layer and physical information, thereby improving the physical rationality of the trajectory and hardware security, so that the robot's upper limb can generate smooth and natural human-like movements.

[0122] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0123] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0124] Figure 4 This paper shows a structural block diagram of a training device for a joint angle prediction model according to an embodiment of the present application, as follows: Figure 4 As shown. The training device 400 for the joint angle prediction model in this embodiment may include a first acquisition unit 401, a first acquisition unit 402, a second acquisition unit 403, and a first training unit 404. The first acquisition unit 401 is used to acquire training sample data, which includes skeletal parameters, sample wrist joint pose sequences, and arm angle labels of the sample wrist joint pose sequences. The first acquisition unit 402 is used to obtain arm angle feature information based on the sample wrist joint pose sequences using the selective state space network of the joint angle prediction model to be trained. The second acquisition unit 403 is used to obtain elbow position prediction information based on the arm angle feature information and skeletal parameters using the differentiable kinematic layer of the joint angle prediction model to be trained. The first training unit 404 is used to iteratively update and train the joint angle prediction model to be trained based on a preset objective function, the training sample data, the arm angle feature information, and the elbow position prediction information to obtain a trained joint angle prediction model.

[0125] Optionally, in one possible implementation of this embodiment, the first obtaining unit 402 is used to obtain high-dimensional feature information based on the sample wrist joint pose sequence using the input network of the joint angle prediction model to be trained; and to obtain arm angle feature information based on the high-dimensional feature information using the selective state space network of the joint angle prediction model to be trained.

[0126] Optionally, in one possible implementation of this embodiment, the input network includes an input layer and a one-dimensional convolutional layer. The first obtaining unit 402 is used to obtain high-dimensional latent feature information based on the sample wrist joint pose sequence using the input layer; and to smooth the high-dimensional latent feature information using the one-dimensional convolutional layer to obtain high-dimensional feature information.

[0127] Optionally, in one possible implementation of this embodiment, the selective state space network includes a backbone network and a decoding network. The first obtaining unit 402 is used to obtain global temporal depth feature information based on the high-dimensional feature information and using the backbone network; and to obtain arm angle feature information based on the global temporal depth feature information and using the decoding network.

[0128] Optionally, in one possible implementation of this embodiment, the backbone network includes multiple selective state space layers, and the first obtaining unit 402 is used to perform feature extraction and residual connection processing on the high-dimensional feature information using the multiple selective state space layers; and to obtain global temporal deep feature information based on the results of the feature extraction and residual connection processing.

[0129] Optionally, in one possible implementation of this embodiment, the preset objective function includes a position constraint objective function, an arm angle constraint objective function, and a smoothing constraint objective function. The first training unit 404 is used to determine a position constraint loss value based on the position constraint objective function, training sample data, and elbow position prediction information; determine an arm angle constraint loss value based on the arm angle constraint objective function, training sample data, and arm angle feature information; determine a smoothing constraint loss value based on the smoothing constraint objective function and arm angle feature information; and iteratively update and train the joint angle prediction model to be trained based on the position constraint loss value, arm angle constraint loss value, and smoothing constraint loss value until a preset training termination condition is met, thereby obtaining a trained joint angle prediction model.

[0130] In this embodiment, training sample data can be acquired by a first acquisition unit. The training sample data includes skeletal parameters, sample wrist joint pose sequences, and arm angle labels for the sample wrist joint pose sequences. The first acquisition unit, based on the sample wrist joint pose sequences, uses the selective state-space network of the joint angle prediction model to be trained to obtain arm angle feature information. The second acquisition unit, based on the arm angle feature information and skeletal parameters, uses the differentiable kinematic layer of the joint angle prediction model to be trained to obtain elbow position prediction information. Finally, the first training unit, based on a preset objective function and the number of training samples... Based on arm angle feature information and elbow position prediction information, the joint angle prediction model to be trained is iteratively updated and trained to obtain a trained joint angle prediction model. Since the selective state space network of the trained joint angle prediction model can achieve global temporal modeling with linear complexity, and human kinematic priors are embedded into the model network through a differentiable kinematic layer, the physical consistency of the model prediction results is guaranteed. This can improve the training ability of the trained joint angle prediction model to learn wrist joint pose sequences and predict joint angles, thereby optimizing the training effect of the joint angle prediction model.

[0131] Figure 5 This application provides a structural block diagram of a device for determining the joint angle of a robot arm according to one embodiment of the present application. Figure 5 As shown. The robot arm joint angle determination device 500 of this embodiment may include a second acquisition unit 501, a third acquisition unit 502, a fourth acquisition unit 503, and a first optimization unit 504. The second acquisition unit 501 is used to acquire end-effector pose data to be processed; the third acquisition unit 502 is used to input the end-effector pose data to be processed into a joint angle prediction model trained by a joint angle prediction model training device, so as to obtain arm angle feature information using the selective state space network of the joint angle prediction model; the fourth acquisition unit 503 is used to obtain elbow position prediction information based on the arm angle feature information and using the differentiable kinematic layer of the joint angle prediction model; the first optimization unit 504 is used to perform numerical optimization processing on the elbow position prediction information to determine the robot arm joint angle corresponding to the end-effector pose data to be processed.

[0132] Optionally, in one possible implementation of this embodiment, the first optimization unit 504 is used to perform numerical optimization processing on the elbow position prediction information using a numerical optimization solution algorithm to determine the robot arm joint angle corresponding to the end pose data to be processed.

[0133] Optionally, in one possible implementation of this embodiment, the joint angle prediction model may further include an input network. The end-effector pose data is input into the input network, and high-dimensional feature information corresponding to the end-effector pose data is output.

[0134] In this embodiment, the end-effector pose data to be processed can be acquired by the second acquisition unit. The third acquisition unit inputs the end-effector pose data to be processed into the joint angle prediction model trained by the device described above. The selective state space network of the joint angle prediction model is used to obtain arm angle feature information. The fourth acquisition unit obtains elbow position prediction information based on the arm angle feature information and the differentiable kinematic layer of the joint angle prediction model. The first optimization unit performs numerical optimization processing on the elbow position prediction information to determine the robot arm joint angle corresponding to the end-effector pose data to be processed. Since the end-effector pose data can be processed by the joint angle prediction model, the accuracy and real-time performance of the predicted robot arm joint angles are improved, the movement errors of the wrist and elbow are reduced, and smoother and more natural humanoid robot upper limb movements can be generated, thereby ensuring the reliability of robot upper limb movements.

[0135] The technical solution of this application involves the collection, storage, use, processing, transmission, provision, and disclosure of user personal information, such as user image and attribute data, which comply with relevant laws and regulations and do not violate public order and good morals.

[0136] According to embodiments of this application, this application also provides an electronic device, a readable storage medium, and a computer program product.

[0137] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this invention disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0138] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A training method for a joint angle prediction model, characterized in that, The method includes: Acquire training sample data, which includes skeletal parameters, sample wrist joint pose sequences, and arm angle labels of the sample wrist joint pose sequences; Based on the sample wrist joint pose sequence, arm angle feature information is obtained by using the selective state space network of the joint angle prediction model to be trained. Based on the arm angle feature information and skeletal parameters, the elbow position prediction information is obtained by utilizing the differentiable kinematic layer of the joint angle prediction model to be trained. Based on the preset objective function, the training sample data, arm angle feature information, and elbow position prediction information, the joint angle prediction model to be trained is iteratively updated and trained to obtain the trained joint angle prediction model.

2. The method of claim 1, wherein, The step of obtaining arm angle feature information based on the sample wrist joint pose sequence and using the selective state-space network of the joint angle prediction model to be trained includes: Based on the sample wrist joint pose sequence, high-dimensional feature information is obtained using the input network of the joint angle prediction model to be trained. Based on the high-dimensional feature information, arm angle feature information is obtained by utilizing the selective state-space network of the joint angle prediction model to be trained.

3. The method according to claim 2, characterized in that, The input network includes an input layer and a one-dimensional convolutional layer. Based on the sample wrist joint pose sequence, the input network of the joint angle prediction model to be trained obtains high-dimensional feature information, including: Based on the sample wrist joint pose sequence, high-dimensional latent feature information is obtained using the input layer; The high-dimensional latent feature information is smoothed using the one-dimensional convolutional layer to obtain high-dimensional feature information.

4. The method according to claim 2, characterized in that, The selective state-space network includes a backbone network and a decoding network. The process of obtaining arm angle feature information based on the high-dimensional feature information using the selective state-space network of the joint angle prediction model to be trained includes: Based on the high-dimensional feature information, global temporal deep feature information is obtained using the backbone network; Based on the global temporal depth feature information, the arm angle feature information is obtained using the decoding network.

5. The method according to claim 4, characterized in that, The backbone network includes multiple selective state space layers. The process of obtaining global temporal deep feature information based on the high-dimensional feature information using the backbone network includes: Multiple selective state space layers are used to perform feature extraction and residual connection processing on the high-dimensional feature information; Based on the results of feature extraction and residual connection processing, global temporal depth feature information is obtained.

6. The method according to claim 1, characterized in that, The preset objective functions include a position constraint objective function, an arm angle constraint objective function, and a smoothness constraint objective function. Based on the preset objective functions, the training sample data, arm angle feature information, and elbow position prediction information, the joint angle prediction model to be trained is iteratively updated and trained to obtain a trained joint angle prediction model, including: Based on the position constraint objective function, training sample data, and elbow position prediction information, the position constraint loss value is determined. Based on the arm angle constraint objective function, training sample data, and arm angle feature information, determine the arm angle constraint loss value; Based on the smoothing constraint objective function and arm angle feature information, the smoothing constraint loss value is determined; Based on the position constraint loss value, arm angle constraint loss value, and smoothness constraint loss value, the joint angle prediction model to be trained is iteratively updated and trained until the preset training termination condition is met, and the trained joint angle prediction model is obtained.

7. A method for determining the joint angles of a robot arm, characterized in that, The method includes: Acquire the end-effector pose data to be processed; The end pose data to be processed is input into the joint angle prediction model trained by the method according to any one of claims 1-6, so as to obtain arm angle feature information by utilizing the selective state space network of the joint angle prediction model. Based on the arm angle feature information, the elbow position prediction information is obtained by using the differentiable kinematic layer of the joint angle prediction model. The elbow position prediction information is numerically optimized to determine the robot arm joint angle corresponding to the end pose data to be processed.

8. A training device for a joint angle prediction model, characterized in that, The device includes: The first acquisition unit is used to acquire training sample data, which includes skeletal parameters, sample wrist joint pose sequences, and arm angle labels of the sample wrist joint pose sequences. The first obtaining unit is used to obtain arm angle feature information based on the sample wrist joint pose sequence and by using the selective state space network of the joint angle prediction model to be trained. The second obtaining unit is used to obtain elbow position prediction information based on the arm angle feature information and skeletal parameters, using the differentiable kinematic layer of the joint angle prediction model to be trained. The first training unit is used to iteratively update and train the joint angle prediction model to be trained based on a preset objective function, the training sample data, arm angle feature information, and elbow position prediction information, so as to obtain a trained joint angle prediction model.

9. A device for determining the joint angle of a robot arm, characterized in that, The device includes: The second acquisition unit is used to acquire the end-effector pose data to be processed. The third obtaining unit is used to input the end pose data to be processed into the joint angle prediction model trained by the device according to claim 8, so as to obtain arm angle feature information by utilizing the selective state space network of the joint angle prediction model. The fourth obtaining unit is used to obtain elbow position prediction information based on the arm angle feature information and using the differentiable kinematic layer of the joint angle prediction model. The first optimization unit is used to perform numerical optimization processing on the elbow position prediction information to determine the robot arm joint angle corresponding to the end pose data to be processed.

10. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1-7.

11. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-7.

12. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method according to any one of claims 1-7.

Images