Pose control method for mechanical arm with body visual capability

Abstract

The invention discloses a pose control method for a mechanical arm with a body vision capability. The pose control method comprises the steps of generating voxel grouping features, remarkably reducing sparse invalid calculation caused by blank voxels, reducing calculation overhead and improving control frequency and real-time performance. Semantic feature extraction and fusion are carried out, multi-scale visual semantic features with local detail and global semantic understanding are obtained, and the perception and decision support capability of a complex three-dimensional scene is enhanced; according to the prediction action based on time integration, error accumulation and time confusion caused by rolling prediction are effectively restrained, and the stability and robustness of mechanical arm control output are improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent control technology for industrial robots, specifically a method for controlling the posture of a robotic arm with embodied vision capabilities. Technical Background In recent years, embodied intelligence, as an emerging research direction in the field of artificial intelligence, has gradually attracted widespread attention from academia and industry. Embodied intelligence emphasizes that intelligent agents acquire cognitive and decision-making abilities through interaction with the physical environment, which differs from traditional artificial intelligence models based on pure computation or abstract reasoning. In particular, exploring a new vision-guided pose control method can effectively promote the further development of embodied intelligence in robotic arms, where embodied intelligence is widely applied. Imitation learning has had a significant impact on the development of robotic arms. Imitation learning generally involves three steps: experts collecting data, learning from the collected data, and autonomous operation using the learned model. Among these, the method of collecting robot data is a key factor.

[0002] In existing technologies, data acquisition methods are generally based on position information. However, torque and angular velocity information are not used during the training or operation of robotic arms, and the hardness of objects cannot be identified. This leads to certain limitations when facing some complex physical environments and highly dynamic scenarios.

[0003] To achieve visual perception in robotic arms, control outputs are generated based on 2D image input. However, due to inherent limitations of 2D images in spatial structure and camera perspective, the model struggles to correctly understand the current environment. Current solutions utilize depth images or point clouds generated by excitation radar; however, training and inference using models based on voxels or 3D point clouds is inefficient for robotic arms, especially when applying models like Transformer with O(n²) complexity for attention computation. Furthermore, existing 3D feature extraction methods for robotic arm operations use a multi-end-to-end approach, directly inputting 3D representations and outputting extracted 3D features. This coarse feature usage lacks distinction between high-level and low-level semantic features, preventing the model from balancing local details and overall semantics. This significantly reduces the robotic arm's control over its environment, leading to inaccurate output actions.

[0004] Traditionally, robotic arm models output only one action. In this case, it is easy to cause compound error problems, where a small error causes the model to enter an unknown state, leading to random behavior and resulting in greater discrepancies. At the same time, outputting only a single action makes it difficult to solve time-related obfuscation problems, such as pauses during demonstrations, which are difficult to model using Markov single-step strategies. Summary of the Invention

[0005] This invention addresses the shortcomings of existing vision-guided robotic arm pose control methods by proposing a robotic arm pose control method with embodied vision capabilities. This method solves the problems of insufficient utilization of three-dimensional spatial information, high computational overhead of three-dimensional sparse voxels leading to insufficient real-time performance, and difficulty in fully integrating multi-source state information such as robotic arm joint angles, angular velocities, and torques, which makes it difficult to achieve precise operations.

[0006] To achieve the above objectives, the present invention provides a method for controlling the pose of a robotic arm with embodied vision capabilities, comprising the following steps: S1 Voxel Grouping Feature Generation: The depth map of the environment is acquired using a depth camera and converted into a 3D voxel mesh. Since the proportion of blank voxels in the high-resolution voxel space is relatively high, direct voxel-by-voxel modeling would result in a large amount of invalid computation. The 3D voxel mesh is input into a 3D convolutional layer, and voxel grouping and local feature aggregation are achieved through stride convolution to obtain voxel grouping features F. This significantly reduces the participation of blank voxels in the computation and reduces the overall computational complexity, thereby meeting the real-time requirements of robotic arm control. S2 Semantic Feature Extraction and Fusion: The voxel grouping features F obtained in step S1 are hierarchically encoded by Transformer to obtain low-level semantic features and high-level semantic features, which are then fused to form a multi-scale semantic fusion feature Z that has both local detail and global semantic understanding capabilities, for subsequent action prediction and control decision-making. S3 Predictive Action Based on Temporal Integration: At discrete execution time t, the multi-scale semantic fusion feature Z obtained in step S2... t Input Prediction Action Transformer Model The predicted action is the sequence of actions (action blocks) at multiple future time steps. To avoid time confusion and error superposition caused by rolling prediction, the overlapping prediction results are integrated and weighted to obtain the final execution action, thereby improving the stability of the control output.

[0007] Further, the specific process of step S1) is as follows: [The text abruptly ends here, so the translation stops.] ) three-dimensional voxel mesh Input to a 3D convolutional layer with a kernel size of 5×5×5 and a stride of (5,5,5) for downsampling and local feature aggregation to obtain voxel grouping features F with a size of 20×20×20.

[0008] The voxel grouping feature F The 3D convolution calculation process satisfies the following: Where W(u,v,w) represents the weight parameters of the 3D convolution kernel, b represents the bias term, and i,j,k represent the indices of the voxel grouping features in the three spatial dimensions, respectively.

[0009] Further, step S2) specifically involves: dividing the voxel grouping feature F from step S1 into m equal layers along the depth direction, and flattening the two-dimensional planar feature corresponding to each layer slice group to form a sequence of length 400. The sequences at each level are then input into a low-level semantic Transformer encoding network with shared parameters. Obtain local low-level semantic features : in The sequence length is 400, and the channel dimension is... ; The Transformer coding network performs multiple layers of attention coding units as follows: in, ; is a learnable linear projection matrix, and d is the dimension of the attention feature.

[0010] Step S2) further includes: concatenating the local low-level semantic features of each level along the sequence dimension to obtain aggregated low-level semantic features. And input L into the high-level semantic Transformer network Obtain global high-level semantic features , Attention execution unit and Consistency; finally, L and H are residually fused and concatenated along the channel dimension to output multi-scale semantic fusion features. , is represented as: .

[0011] Further, step S3) specifically involves: at discrete execution time t, the multi-scale semantic fusion features obtained in step S2 are processed... A Transformer model that predicts actions from inputs and style latent variables trained using all collected data. This yields the sequence of actions (action blocks) for the next k steps: ,in This represents the predicted action at time t to time t+j. Due to rolling prediction, the action at the same time t may be overwritten by predictions from multiple historical times. The final action is obtained by using a time-integrated weighted average of the overlapping predicted actions. This is to suppress composite errors and mitigate temporal confusion by fusing overlapping predictions.

[0012] The training and use of style latent variables s: During the training phase, based on the collected demonstration data, the distribution parameters (mean) of style latent variables s are output through the style encoding network. With variance ), and set the distribution to a diagonal Gaussian distribution 𝒩( During training, s is sampled from this distribution and input along with visual semantic features into the Transformer model for predicting actions. To enhance the ability to represent different demonstration styles; during the inference phase, in order to obtain stable and consistent control output, the style latent variable is fixed as a zero vector (i.e., s=0), so that action blocks can be output without sampling.

[0013] Predicting Action Transformer Model This is a conditional sequence-to-sequence action block prediction network, which uses the multi-scale semantic fusion features output in step S2. As input (optionally, the style latent variable s can be injected as a conditional variable after embedding), first, through feature projection and positional encoding, ... The sequence representation is converted into a unified dimension. Then, the Transformer unit, which includes multiple layers of self-attention and feedforward transformation, performs global dependency modeling and context aggregation on the sequence representation to obtain the control-related latent space representation. Finally, the action vector sequence A for the next k discrete moments is obtained by parallel regression through the action regression output head. The action vector is used to represent the joint space, joint angular velocity, and joint torque of the robotic arm.

[0014] Furthermore, the time integration adopts a weighted average form: in, This represents the predicted action at time t from time ti.

[0015] Furthermore, it also includes training the Transformer model f(⋅) for predicting actions based on the collected data; the collected data includes the joint angles, joint angular velocities, joint torques, and joint torques of the master arm and slave arm.

[0016] The joint angles, angular velocities, torques, and corresponding depth maps of the master and slave arms are input into the Transformer model f(⋅) for predicting motion. The model outputs motion blocks and control-related information to drive the slave arm to execute. The bilateral control system satisfies the following position tracking and force feedback consistency objectives: Objective 1: Objective Two: ; in, Indicates joint angle, Indicates joint torque / torque; subscript l Indicates the main boom system, subscript f This indicates a slave arm system, thereby enabling intuitive two-way control of the control target.

[0017] A computer-readable storage medium is also provided, storing a computer program that, when executed by a processor, implements the method of any one of claims 1 to 9.

[0018] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention constructs a three-dimensional voxel representation through a depth map and uses large stride three-dimensional convolution to perform voxel grouping downsampling and local aggregation, which significantly reduces sparse and invalid computation caused by blank voxels, reduces computational overhead, and improves control frequency and real-time performance.

[0019] 2. This invention uses a hierarchical Transformer to extract low-level semantics and fuse high-level semantics, thereby obtaining multi-scale visual semantic features that combine local details with global semantic understanding, enhancing the perception and decision support capabilities for complex 3D scenes.

[0020] 3. This invention effectively suppresses the error accumulation and time confusion caused by rolling prediction by using parallel prediction of action blocks combined with a time integration fusion strategy, thereby improving the stability and robustness of the robotic arm control output.

[0021] 4. This invention, by introducing bilateral control to enhance data acquisition and consistency objectives, enables the model to comprehensively utilize joint angles, angular velocities, torque / force, and visual information, achieving more intuitive master-slave operation and force feedback consistency, thereby improving refined operation capabilities and system stability. Attached Figure Description

[0022] Figure 1 This is a model structure diagram of a robotic arm pose control method with embodied vision capability according to the present invention.

[0023] Figure 2 This is a flowchart illustrating the pose control method for a robotic arm with embodied vision capabilities according to the present invention.

[0024] Figure 3This is a schematic diagram illustrating the training and reasoning stages of a robotic arm pose control method with visual embodied intelligence capabilities according to the present invention. Detailed Implementation

[0025] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0026] like Figure 1 As shown, the present invention provides a robotic arm pose control method with visual embodied intelligence capabilities, which mainly includes: depth camera acquisition and 3D voxel representation, deep 3D voxel convolution downsampling (voxel grouping feature generation), hierarchical 3D visual semantic feature extraction and fusion, action block prediction output based on temporal integration, and a bilateral control-enhanced model learning and inference process. The specific process is as follows: Figure 2 As shown: S1 Voxel Grouping Feature Generation: The depth camera acquires an environmental depth map, which is then converted into a 3D voxel mesh representation V; the 3D voxel mesh V is a high-resolution voxel space, for example, with a size of... Voxel values ​​can be used to represent occupancy information, distance information, or task-related voxel features (which can be selected by those skilled in the art according to the specific implementation).

[0027] To reduce the number of blank voxels involved in the computation and lower the computational complexity per voxel, a 3D voxel grid V is input into a 3D convolutional layer for downsampling and local feature aggregation. Specifically, the kernel size of the 3D convolutional layer is... With a step size of (5,5,5), the 3D voxel mesh is compressed to a size of The voxel grouping feature F, its transformation relationship is expressed as: The three-dimensional convolution calculation process satisfies the following: Where W(u,v,w) represents the weight parameters of the 3D convolution kernel, b represents the bias term, and i,j,k represent the indices of the output voxel grouping features in the three spatial dimensions, respectively.

[0028] S2 Semantic Feature Extraction and Fusion: The voxel grouping features F obtained in step S1 are hierarchically encoded using Transformer to obtain low-level and high-level semantic features, which are then fused to form multi-scale semantic fusion features Z, which are used for subsequent action prediction.

[0029] Specifically, the voxel grouping feature F is divided into M=20 layer slice groups along the depth direction. For each layer slice group, the corresponding two-dimensional planar feature ( Flattening the sequence yields a sequence representation of length 400, denoted as: Wherein, a sequence length of 400 corresponds to Spatial flattening, passageway dimension This represents the number of feature channels.

[0030] S21 Low-level semantic feature extraction: Flattening features at each level The inputs are fed into a low-level semantic Transformer encoding network with shared parameters. To obtain local low-level semantic features: in The Transformer coding network may contain multiple layers of attention coding units, and its attention calculation can be expressed as: in W Let be a learnable linear projection matrix, and d be the dimension of the attention feature.

[0031] S22 Advanced Semantic Fusion: Concatenates local low-level semantic features from various levels along the sequence dimension to obtain aggregated low-level semantic features. The L is then input into the high-level semantic Transformer network T_{high} to obtain global high-level semantic features: in This represents the channel dimension of high-level semantic features. Subsequently, L and H are residually fused and concatenated along the channel dimension to form multi-scale semantic fusion features: Z is used to represent 3D visual semantic information that combines local details with global semantics.

[0032] S3 Predictive Action Based on Temporal Integration: At discrete execution time t, the multi-scale semantic fusion features obtained in step S2 are used... (Optional, style latent variables s can be used as conditional inputs) Input predicts action Transformer model Output the action sequence (action block) for the next k steps: in This refers to "the predicted action at time t to time t+j". Since rolling prediction can cause actions at the same time to be overwritten by predictions from multiple historical times, this invention employs a time-integrated weighted average for overlapping predictions to obtain the final execution action. : By fusing overlapping predictions, composite errors can be suppressed and time confusion can be mitigated, thereby improving the stability of the control output.

[0033] Among them, the predictive action Transformer model For conditional sequence-to-sequence action block prediction networks: it uses As input (optionally, the style latent variable s can be injected as a conditional variable after embedding), first, through feature projection and positional encoding, ... The sequence representation is converted to a unified dimension. Then, a Transformer unit containing multiple layers of self-attention and feedforward transformation performs global dependency modeling and context aggregation on this sequence representation to obtain a control-related latent space representation. Finally, the action vector sequence A(t) for the next k discrete time steps is obtained through a one-time parallel regression by the action regression output head. The action vector is used to represent the joint space-related control quantities of the robotic arm, such as joint angle, joint angular velocity, joint torque (or a combination thereof).

[0034] During the training phase, based on the collected demonstration data, the distribution parameters (mean) of the style latent variable s are output through the style coding network. With variance ), and set the distribution to a diagonal Gaussian distribution. During training, s is sampled from this distribution and input along with visual semantic features into the Transformer model for predicting actions. To enhance the ability to represent different demonstration styles, during the inference phase, in order to obtain a stable and consistent control output, the style latent variables are fixed as a zero vector (i.e., s=0), so that action blocks can be output without sampling.

[0035] During the data acquisition phase, joint angles, joint angular velocities, and joint torques / torques of the master and slave arms are simultaneously acquired, along with depth maps from a depth camera. Based on this data, a Transformer model for predicting motion is then developed. Training is performed. The joint angles, joint angular velocities, joint torques / torques, and depth maps of the master arm and slave arm are input into the model, and action blocks and control information are output to drive the slave arm to execute.

[0036] Specifically, the control objectives of the entire bilateral control system include: Objective 1 (Location Tracking): ; Objective 2 (action-reaction relationship): .

[0037] in, Indicates joint angle, This indicates joint torque / torque; the subscript 'l' indicates the master arm system, and the subscript 'f' indicates the slave arm system. By satisfying objective one, position consistency / tracking relationship between the master and slave arms is achieved; by satisfying objective two, force feedback consistency is achieved, thus allowing the operator to perform intuitive two-way control of the control target.

[0038] like Figure 3 As shown, the training and inference process of the robotic arm pose control method with visual embodied intelligence capability of the present invention is illustrated: in the training phase, the joint information of the master arm / slave arm and the depth map demonstration data are used to train the model; in the inference phase, the slave arm inputs the joint angle, angular velocity, torque and corresponding depth map into the model, and the model outputs the subsequent joint control information and action sequence for the slave arm to execute.

[0039] The present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described method for controlling the pose of a robotic arm.

[0040] The contents not described in detail in this specification are existing technologies known to those skilled in the art.

[0041] Finally, it should be noted that the above specific embodiments are only used to illustrate the technical solution of this patent and not to limit it. Although this patent has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solution of this patent without departing from the spirit and scope of the technical solution of this patent, and all such modifications and substitutions should be covered within the scope of the claims of this patent.

Claims

1. A method for pose control of a robot arm with body vision capability, characterized by, The control method comprises the following steps: S1: generating a voxel grouping feature, using a depth camera to collect an environment depth map, converting the depth map into a three-dimensional voxel grid, inputting the three-dimensional voxel grid into a three-dimensional convolution layer, and realizing voxel grouping and local feature aggregation through step convolution to obtain a voxel grouping feature F; S2: extracting and fusing a semantic feature, performing hierarchical Transformer coding on the voxel grouping feature F obtained in step S1 to obtain a low-level semantic feature and a high-level semantic feature and fuse them to form a multi-scale semantic fusion feature Z; S3 Time-integrated predicted action: at the discrete execution time t, the multi-scale semantic fusion feature Z obtained by step S2 t Input the predicted action Transformer model Get the predicted action, and perform time-integrated weighted fusion on the overlapping predicted results to obtain the final execution action.

2. The method according to claim 1, wherein The specific process of the step S1) is: inputting a three-dimensional voxel grid with a size of to a three-dimensional convolution layer with a convolution kernel size of 5×5×5 and a step size of (5,5,5) to perform down-sampling and local feature aggregation to obtain a voxel grouping feature F with a size of 20×20×20. ​ 3.The method of claim 2, wherein The voxel grouping feature F Wherein, the three-dimensional convolution calculation process satisfies: Wherein, W(u,v,w) represents a weight parameter of the three-dimensional convolution kernel, b represents a bias term, and i, j, and k represent indexes of the voxel grouping feature in three spatial dimensions.

4. The method of claim 1, wherein the method further comprises: The specific process of the step S2) is: dividing the voxel grouping features F of the step S1 into m hierarchical slice groups in the depth direction, flattening the two-dimensional plane features corresponding to each hierarchical slice group to form a sequence with a length of 400 , and inputting the sequences of each hierarchy into a low-level semantic Transformer encoding network sharing parameters to obtain local low-level semantic features : wherein the sequence length is 400 and the channel dimension is ; The Transformer coding network executes a plurality of attention coding units as follows: wherein, ; is a learnable linear projection matrix, and d is the dimension of the attention feature.

5. The method according to claim 4, wherein Step S2) further includes: concatenating the local low-level semantic features at each level along the sequence dimension to obtain aggregated low-level semantic features. And input L into the high-level semantic Transformer network Obtain global high-level semantic features , Attention execution unit and Consistency; finally, L and H are residually fused and concatenated along the channel dimension to output multi-scale semantic fusion features. , is represented as: 。 6. The method of claim 1, wherein the method further comprises: The specific process of the step S3) is: at a discrete execution moment t, the multi-scale semantic fusion features obtained in the step S2 are input into the action prediction Transformer model trained by all the collected data and a style hidden variable s to obtain a future k-step action sequence: , wherein represents the predicted action at the moment t for the moment t+j; and a time integration weighted average is adopted for the overlapping predicted actions to obtain a final execution action .​​​ 7. The method according to claim 6, wherein The time integration adopts a weighted average form: wherein, represents a predicted action at time t at time (t-i). 8.The method of claim 1, wherein Further comprising training a predicted action Transformer model f(·) based on collected data; the collected data includes joint angles, joint angular velocities, joint torques, and joint torques of the master arm and the slave arm.

9. The method according to claim 8, wherein, Inputting the joint angles, joint angular velocities, joint torques, joint torques, and corresponding depth maps of the master arm and the slave arm into the predicted action Transformer model f(·) to output an action block and control related information to drive the slave arm to execute; The position tracking and force feedback consistency goals satisfied by the bilateral control system include: Goal one: Goal two: wherein, denotes joint angle, denotes joint torque; subscript l denotes master arm system, subscript f denotes slave arm system, thereby achieving intuitive bidirectional control over the control target.

10. A computer readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to realize the method of any one of claims 1 to 9.

Images