Method and device for controlling a robot arm, robot, storage medium and program product

Abstract

The method comprises the following steps: training an image prediction model by using a complete structure image and a structure defect image of a mechanical arm as an image data pair, so that the image prediction model has the capability of completing the image features of the structure defect image to generate a complete structure image, and the capability is migrated to a conventional data set (real robot self-sampling data), thereby improving the stability of the model, and the training data set used is a single image, not a multi-view image, thereby solving the problem that the related art is seriously dependent on complex multi-view image input; and training an action prediction model based on an image and action data pair, thereby improving the prediction stability and accuracy of the action prediction model on the basis of the second image prediction model providing more complete image information, and the mechanical arm is more accurately controlled.

Description

Technical Field

[0001] This disclosure relates to intelligent control technology for robots, and in particular to a control method, device, robot, storage medium, and program product for a robotic arm. Background Technology

[0002] The inverse dynamics model is a core mathematical model in robotics and control theory, used to solve the problem of "deriving the required force from the desired motion". Its core function is to calculate, given the robot's current state (including joint positions and velocities) and the desired acceleration, the precise torque or force required to be applied to each joint to achieve that motion. Essentially, this model constructs a mapping from "motion result" to "force input," and its value lies in serving as a crucial bridge between low-level actuators and high-level motion planning, enabling robots to achieve precise trajectory tracking, compliant force interaction control, and efficient energy utilization.

[0003] However, existing inverse dynamics models (IDMs) either rely on entire continuous video sequences or heavily depend on complex multi-view image inputs. Specifically, to extract robotic arm motion from video footage, some models require receiving entire continuous video sequences and extracting frame-to-frame motion. While these models can effectively extract robotic arm motion, they demand consecutive video sequences of the same task as input, placing excessive demands on the data. Other models can receive single-frame robotic arm images and output corresponding actions; however, the effectiveness of these methods is limited by the input image format. They only show good results when receiving complete multi-view robotic arm images and exhibit poor stability when processing data from outside the robotic arm frame. Summary of the Invention

[0004] To address the technical problems in related technologies, embodiments of this disclosure provide a control method, apparatus, robot, storage medium, and program product for a robotic arm.

[0005] According to a first aspect of the present disclosure, a method for controlling a robotic arm is provided, the method comprising:

[0006] A first image prediction model is trained using a first training dataset, which is a demonstration image data of robotic arm operation, including a first image data pair of a first target robotic arm. Each first image data pair includes a first structural defect image of any first target robotic arm and a corresponding first structural complete image. The first image prediction model is used to perform image feature completion processing on the structural defect image of any first target robotic arm.

[0007] The first image prediction model is optimized and adjusted using the second training dataset to obtain the second image prediction model. The second training dataset consists of real operation image data of the robotic arm, including second image data pairs of the second target robotic arm. Each second image data pair includes a second structural defect image and a second structural integrity image of any second target robotic arm.

[0008] A motion prediction model is trained based on the second image prediction model and the third training dataset. The third training dataset consists of the second structural defect image and the operation motion data of the second target robotic arm in the second image data pair. The motion prediction model is used to predict the operation motion of the robotic arm based on the structural defect image of any robotic arm.

[0009] In response to receiving an image of any third target robotic arm, the third target robotic arm is controlled by the operation action output by the motion prediction model.

[0010] As an optional embodiment, a first image prediction model is trained using a first training dataset, which is a demonstration image data of a robotic arm operation, including first image data pairs of a first target robotic arm. Each first image data pair includes a first structural defect image of any first target robotic arm and a corresponding first structurally complete image. The first image prediction model is used to perform image feature completion processing on any structural defect image of the first target robotic arm, including:

[0011] A preset image region of the first structural complete image of the first target robotic arm is occluded to obtain an image of the first structural defect.

[0012] First image features are extracted from the first complete image of the structure, and second image features are extracted from the first image of the structure with defects, wherein the first image features are image features with complete image information, and the second image features are image features with image defects;

[0013] The first image prediction model is trained based on the first image features and the second image features;

[0014] The first training dataset is a demonstration image data of robotic arm operation, including a first image data pair of a first target robotic arm. Each first image data pair includes a first structural defect image of any first target robotic arm and a corresponding first structural complete image. The first image prediction model is used to perform image feature completion processing on the structural defect image of any first target robotic arm.

[0015] As an optional embodiment, training the first image prediction model based on the first image features and the second image features includes:

[0016] The second image feature is input into the first image prediction model to perform feature completion processing on the second image feature through the first image prediction model to obtain the third image feature;

[0017] The first model loss function is determined based on the third image feature and the first image feature;

[0018] The network parameters of the first image prediction model are optimized and adjusted based on the first model loss function to complete the training of the first image prediction model.

[0019] As an optional embodiment, determining the first model loss function based on the third image features and the first image features includes:

[0020] Feature information of a preset image region is determined from the third image features and the first image features, respectively;

[0021] Calculate the image loss value between the feature information of the preset image region in the third image feature and the first image feature, and the image loss value is the first model loss function.

[0022] As an optional embodiment, the action prediction model is trained based on the second image prediction model and a third training dataset. The third training dataset consists of the second structural defect image and the operation action data of the second target robotic arm in the second image data pair. The action prediction model is used to predict the operation action of the robotic arm based on any structural defect image of the robotic arm, including:

[0023] The second structural defect image in the second image data pair is input into the second image prediction model so that the second structural defect image is processed by the second image prediction model to complete the image features and obtain the third structurally complete image.

[0024] The complete image of the third structure is input into the action prediction model to obtain the first predicted operation action data;

[0025] The network parameters of the motion prediction model are optimized and adjusted based on the first predicted operation data and the operation data of the second target robotic arm to complete the training of the motion prediction model.

[0026] As an optional embodiment, the step of optimizing and adjusting the network parameters of the motion prediction model based on the first predicted operation motion data and the operation motion data of the second target robotic arm to complete the training of the motion prediction model includes:

[0027] The second model loss function is determined based on the first predicted operation data and the operation data of the second target robotic arm.

[0028] The network parameters of the action prediction model are optimized and adjusted according to the loss function of the second model to complete the training of the action prediction model.

[0029] According to a second aspect of the present disclosure, a control device for a robotic arm is provided, the device comprising:

[0030] The first training module is used to train a first image prediction model using a first training dataset. The first training dataset is a demonstration image data of a robotic arm operation, including a first image data pair of a first target robotic arm. Each first image data pair includes a first structural defect image of any first target robotic arm and a corresponding first structural complete image. The first image prediction model is used to perform image feature completion processing on the structural defect image of any first target robotic arm.

[0031] The model optimization module is used to optimize and adjust the first image prediction model using the second training dataset to obtain the second image prediction model. The second training dataset is real operation image data of the robotic arm, including second image data pairs of the second target robotic arm. Each second image data pair includes a second structural defect image and a second structural complete image of any second target robotic arm.

[0032] The second training module is used to train an action prediction model based on the second image prediction model and the third training dataset. The third training dataset consists of the second structural defect image and the operation action data of the second target robotic arm in the second image data pair. The action prediction model is used to predict the operation action of the robotic arm based on the structural defect image of any robotic arm.

[0033] The motion control module is used to control the third target robotic arm in response to receiving an image of any third target robotic arm, by controlling the third target robotic arm through the operation action output by the motion prediction model.

[0034] As an optional embodiment, the first training module includes:

[0035] The image processing unit is used to perform occlusion processing on a preset image area of ​​the first structural complete image of the first target robotic arm to obtain the first structural defect image;

[0036] The feature extraction unit is used to extract a first image feature from the first structurally complete image and a second image feature from the first structurally defective image, wherein the first image feature is an image feature with complete image information and the second image feature is an image feature with image defects;

[0037] The first model training unit is used to train the first image prediction model based on the first image features and the second image features;

[0038] The first training dataset is a demonstration image data of robotic arm operation, including a first image data pair of a first target robotic arm. Each first image data pair includes a first structural defect image of any first target robotic arm and a corresponding first structural complete image. The first image prediction model is used to perform image feature completion processing on the structural defect image of any first target robotic arm.

[0039] As an optional embodiment, the first model training unit is further configured to:

[0040] The second image feature is input into the first image prediction model to perform feature completion processing on the second image feature through the first image prediction model to obtain the third image feature;

[0041] The first model loss function is determined based on the third image feature and the first image feature;

[0042] The network parameters of the first image prediction model are optimized and adjusted based on the first model loss function to complete the training of the first image prediction model.

[0043] As an optional embodiment, the first model training unit, used to optimize and adjust the network parameters of the first image prediction model based on the first model loss function to complete the training of the first image prediction model, can be implemented as follows:

[0044] Feature information of a preset image region is determined from the third image features and the first image features, respectively;

[0045] Calculate the image loss value between the feature information of the preset image region in the third image feature and the first image feature, and the image loss value is the first model loss function.

[0046] As an optional embodiment, the second training module includes:

[0047] The image completion unit is used to input the second structural defect image in the second image data pair into the second image prediction model, so as to perform image feature completion processing on the second structural defect image through the second image prediction model to obtain a third structurally complete image;

[0048] An action prediction unit is used to input the complete image of the third structure into the action prediction model to obtain first predicted operation action data;

[0049] The parameter tuning unit is used to optimize and adjust the network parameters of the motion prediction model based on the first predicted operation motion data and the operation motion data of the second target robotic arm, so as to complete the training of the motion prediction model.

[0050] As an optional embodiment, the parameter tuning unit is further configured to:

[0051] The second model loss function is determined based on the first predicted operation data and the operation data of the second target robotic arm.

[0052] The network parameters of the action prediction model are optimized and adjusted according to the loss function of the second model to complete the training of the action prediction model.

[0053] According to a third aspect of the present disclosure, a robot is provided, the robot including a robotic arm, the robotic arm being operated and controlled by the robotic arm control method described in the first aspect above.

[0054] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising:

[0055] Memory, used to store computer program products;

[0056] A processor is configured to execute a computer program product stored in the memory, and when the computer program product is executed, to implement the method described in the first aspect above.

[0057] According to a fifth aspect of the present disclosure, a computer-readable storage medium is provided having computer program instructions stored thereon, which, when executed by a processor, implement the method described in the first aspect.

[0058] According to a sixth aspect of the present disclosure, a computer program product is provided, including computer program instructions that, when executed by a processor, implement the method described in the first aspect above.

[0059] The technical solution provided in this disclosure uses image data pairs consisting of a structurally complete image and an image of structural defects of a robotic arm to train a first image prediction model. This enables the model to complete the image features of the defective image to generate a structurally complete image. The first image prediction model is then further optimized and adjusted using real robotic arm operation image data to obtain a second image prediction model. This solves the problem that conventional datasets (real robotic arm operation image data) are difficult to use for training image prediction models because they do not contain complete robotic arm information. Through two-stage training, the ability of the image prediction model to complete structural defective images is transferred to conventional datasets, improving the model's stability. The training dataset used consists entirely of single images, not multi-view images, which solves the problem of relying heavily on complex multi-view image inputs in related technologies. The structural defect images of the robotic arm can be completed using only single-view image input. At the same time, it further lays a data foundation for the prediction accuracy of the subsequent motion prediction model. The motion prediction model is then trained based on the second image prediction model and a dataset including the second structural defect images and the second target robotic arm's operation motion data from the second image data pairs. With the more complete image information provided by the second image prediction model, the prediction stability and accuracy of the motion prediction model are improved, thereby enabling more precise control of the robotic arm.

[0060] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0061] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0062] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein:

[0063] Figure 1 This is one of the schematic flowcharts of a robotic arm control method according to an embodiment of the present disclosure.

[0064] Figure 2 This is a second schematic flowchart illustrating a control method for a robotic arm according to one embodiment of the present disclosure.

[0065] Figure 3 This is a third schematic flowchart illustrating a robotic arm control method according to one embodiment of the present disclosure.

[0066] Figure 4 This is a schematic structural block diagram of a control device for a robotic arm according to one embodiment of the present disclosure.

[0067] Figure 5 This is a schematic structural block diagram of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0068] The current embodied intelligence (EI) industry faces limitations in both data and alignment: On the one hand, compared to the vast internet-scale corpora of vision or language, scalable, rich, and semantically aligned action data is extremely scarce. Data collection is expensive, time-consuming, and strongly coupled with specific hardware, leading to difficulties in cross-body transfer and multi-task generalization. Existing datasets are generally small in scale and highly task-specific. On the other hand, the three main data sources each have their inherent drawbacks: for example, simulation data is scalable but suffers from Sim2Real (from simulation to reality) and physical fidelity issues; while real robot data is practical but costly and easily constrained by task-specific distributions; internet videos contain physical or motion priors, but research on "how to directly extract actions from videos" remains insufficient. Furthermore, while VLA (Vision-Language-Action) improves multi-task learning through language conditionation, it requires large-scale task-conditional action data, further exacerbating data costs and body-specific differences.

[0069] To address these current industry challenges, recent solutions have focused on both the data and paradigm sides. On the data side, the concept of "task-agnostic action data" has been proposed. This approach decouples action data from specific tasks to lower the acquisition threshold and support cross-task reuse. A representative example is ATARA, which uses scripts / reinforcement learning to uniformly explore and automatically collect large-scale images and actions within a cubic workspace for subsequent model use. On the paradigm side, two main lines have emerged: one is the video generation + Inverse Dynamics Model (IDM) decoupling route, using visual generation to predict "near-future observations," and then using IDM to map frame-by-frame images to robot actions; SuSIE, UniPi, RoboDreamer, UniSim, and Gen2Act belong to this category. The other is the VLA multi-task strategy, which uses language conditionalization to achieve cross-task functionality, but is still constrained by the cost of task-conditional data and differences in form. At the specific model level, the industry has improved the accuracy and robustness of "image to action" through structural priors and parametric modifications. For example, arm decoupling is introduced to reduce cross-arm interference, orientation / geometric priors are injected into the decoder to improve the stability of high-precision regression, and a two-stage framework is used to supplement the semantic gaps of task-irrelevant data with video models, with IDM acting as a "robot adapter" to output precise actions. These technical approaches have demonstrated considerable gains in public tests and real-world playback, and verified the feasibility of linking with diffusion-based video generation to complete realistic manipulation tasks, thus providing empirical support for large-scale embodied learning with "low-cost data + high-precision execution".

[0070] However, existing inverse dynamics models (IDMs) either rely on a continuous video sequence or heavily depend on complex multi-view image inputs.

[0071] To enable those skilled in the art to clearly and accurately understand the technical solutions of the embodiments of this disclosure, the technical solutions of the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings.

[0072] Figure 1 This is one of the schematic flowcharts illustrating a robotic arm control method according to an embodiment of the present disclosure. Figure 1 As shown, a control method for a robotic arm according to an embodiment of this disclosure may include the following steps:

[0073] Step 101: Train a first image prediction model using a first training dataset, wherein the first training dataset is a demonstration image data of a robotic arm operation, including first image data pairs of a first target robotic arm. Each first image data pair includes a first structural defect image of any first target robotic arm and a corresponding first structural complete image. The first image prediction model is used to perform image feature completion processing on the first structural defect image of any first target robotic arm.

[0074] The first training dataset may include a large number of first image data pairs, which are demonstration image data of robotic arm operations. These may be from the ATARA dataset, which includes standard photographs (or standard images) of various robotic arms, where the robotic arm is always fully visible in the center of the frame. For example, the ATARA dataset contains a clear photograph showing a complete robotic arm reaching for a red block. In this embodiment of the disclosure, the robotic arm in any image from the ATARA dataset is defined as the first target robotic arm.

[0075] Any first image data pair of the first target robotic arm consists of two versions of the same image: an image version with a complete robotic arm structure and an image version with structural defects, i.e., a first image with structural defects and a first image with a complete structure. As an example, the first image with a complete structure can be constructed by randomly occluding the edges of the first image with a complete structure, so as to simulate the effect of the robotic arm partially appearing out of the frame.

[0076] As an optional embodiment, step 101 can be implemented as follows: occlusion processing is performed on a preset image region of the first structurally complete image of the first target robotic arm to obtain a first structurally flawed image; first image features are extracted from the first structurally complete image and second image features are extracted from the first structurally flawed image, wherein the first image features are image features with complete image information and the second image features are image features with image flaws; a first image prediction model is trained based on the first image features and the second image features. Furthermore, the second image features are input into the first image prediction model to perform feature completion processing on the second image features through the first image prediction model to obtain third image features; then, a first model loss function is determined based on the third image features and the first image features; and the network parameters of the first image prediction model are optimized and adjusted based on the first model loss function to complete the training of the first image prediction model.

[0077] The first image prediction model can utilize a self-attention mechanism. For example, firstly, it can leverage strong structural priors. Since the robotic arm is not a random collection of pixels, but a structured object composed of links, joints, and other components connected in a specific order and geometric relationship, the approximate position and orientation of the occluded part can be inferred based on a particular joint or structure of the robotic arm. The first image prediction model can then learn "structural prior knowledge" about the robotic arm from the first training dataset. Then, using the self-attention mechanism, context awareness and information aggregation are achieved. The incomplete feature map (i.e., the first structurally flawed image) received by the first image prediction model can be viewed as a grid composed of many "feature vectors," each corresponding to a local region of the image. The self-attention mechanism associates each feature vector (including visible and invisible regions) with all other feature vectors and calculates an "attention score." This score represents the importance of information from other regions during the completion process. Therefore, for a feature vector representing an occluded region with missing information, the attention mechanism will find that it is highly correlated with feature vectors of certain visible regions (e.g., a clear joint connection). Then, these highly correlated visible region feature information can be highly weighted and aggregated. Finally, by aggregating this strongly correlated (highly correlated) information from the visible context and combining it with learned structural priors, the first image prediction model can "generate" a feature representation for the missing region that best conforms to logic and physical laws. In this way, image feature completion can be achieved based on a single image or a single viewpoint image.

[0078] Specifically, see Figure 2 The process is shown. Figure 2 This is a second schematic flowchart illustrating a control method for a robotic arm according to one embodiment of the present disclosure. Figure 2The diagram illustrates the training process of the first image prediction model, which includes at least encoder 1, encoder 2, and a predictor. Encoder 1 receives a complete image A (corresponding to a first structurally complete image) and outputs E1 (corresponding to a first image feature). Encoder 2 receives an occluded image A1 (corresponding to a first structurally flawed image) and outputs E2 (corresponding to a second image feature, which is a fragmented feature map or an image feature of an image flaw). Then, E2 is transmitted to the predictor so that the predictor can complete the features of E2 and output the feature-complete E3 (corresponding to a third image feature). E3 is then trained to learn the distribution of E1 as much as possible. The loss functions of both are calculated and backpropagated to adjust the network parameters of the first image prediction model (encoder 1, encoder 2, and predictor), thus achieving the training of the first image prediction model.

[0079] Through the above process, based on the ATARA robotic arm dataset, the feature completion capability of training a predictor using existing complete robotic arm data is achieved. Since many robotic arm video datasets exhibit robotic arm out-of-frame (occlusion or image defects), in this embodiment, edge occlusion is applied to the complete robotic arm data to simulate the out-of-frame situation.

[0080] Step 102: Optimize and adjust the first image prediction model using the second training dataset to obtain the second image prediction model. The second training dataset consists of real operation image data of the robotic arm, including second image data pairs of the second target robotic arm. Each second image data pair includes a second structural defect image and a second structural complete image of any second target robotic arm.

[0081] The first image prediction model is trained on the ATARA dataset. However, the ATARA dataset may differ from the real-world operating environment of the robotic arm. To adapt its prediction capabilities to this environment, the first image prediction model gradually transfers its feature completion capabilities from the ATARA data to the self-collected data (real-world robotic arm operation images). In this step, the training approach of the first image prediction model from step 101 is continued. Since the self-collected data (real-world robotic arm operation images) rarely contains complete robotic arm data, this stage only slightly occludes some images (resulting in the second structural defect image). The robotic arm data with a complete structure corresponding to the second structural defect image in the self-collected data is then used as the second complete structural image. Subsequently, the first image prediction model is further fine-tuned and trained based on the second structural defect image and the second complete structural image to obtain the second image prediction model. This allows the first image prediction model to transfer its feature completion capabilities to the self-collected data.

[0082] In some embodiments, the fine-tuning process may refer to, for example, Figure 3The fine-tuning process of the first image prediction model shown (solid box) includes encoder 1, encoder 2, and predictor 1. Any image A is used as the initial input, and image A is occluded to obtain A1 (as described in the previous second structural defect image). Encoder 1 extracts image features E1 (incomplete feature maps or image features of image defects) from A1. E1 is used as the input to predictor 1, causing predictor 1 to perform feature completion processing to obtain E2. Simultaneously, image A is used as the input to encoder 2, and encoder 2 extracts image features E3 (image features of complete structure) from A. Predictor 1 is fine-tuned based on E2 and E3 to obtain predictor 2. It should be noted that in the examples of this disclosure, encoder 1 and encoder 2 can be the same encoder deployed, extracting different features for different input images. Alternatively, two encoders can be deployed to process different input images respectively.

[0083] The training process of the second image prediction model can be found in the training process of the first image prediction model. For the sake of brevity, it will not be described in detail here.

[0084] Step 103: Train the action prediction model based on the second image prediction model and the third training dataset. The third training dataset consists of the second structural defect image and the second target robotic arm's operation action data in the second image data pair. The action prediction model is used to predict the operation action of any robotic arm based on the structural defect image of any robotic arm.

[0085] The second image prediction model already has the ability to complete features of images with structural defects on the self-collected dataset. In order to further predict the robot arm's movements based on the completed images, in this step, the motion prediction model will be trained to predict the robot arm's operation movements based on the images. Optionally, if it is an image with structural defects, the second image prediction model will be used to complete the features of the image with structural defects to obtain the completed image, and then the motion prediction model will be used to predict the operation movements. In this way, visual features are mapped into action commands.

[0086] As one embodiment, the action prediction model can add a neural network (possibly a fully connected layer or another Transformer) to the second image prediction model. This allows it to leverage the image completion capabilities of the second image prediction model, and also utilize the neural network in conjunction with the output of the second image prediction model (i.e., the completed image, which is a high-dimensional feature vector containing complete information about the robotic arm's state) to learn and understand the current state and environment of the robotic arm. This high-dimensional information is then progressively condensed and mapped to the required action space to predict the robotic arm's actions, ultimately outputting a predicted action. This predicted action can be the rotation angle of each joint of the target robotic arm for each frame of the image or the current frame. As one embodiment, each target robotic arm can include 14 joints; thus, the predicted action consists of a 14-dimensional vector.

[0087] The operation data of the second target robotic arm can be operation data from the robotic arm's sensors. Through annotation and other operations, it can be made into a dataset with real action labels. In this step, this dataset can be used as test data to impose conditions on the action prediction model, making its predicted robotic arm operation more closely resemble the robotic arm's actual operation. In this embodiment of the disclosure, the second structural defect image and the operation data of the second target robotic arm in the second image data pair constitute an <image feature, real action> data pair. The massive data pair constitutes the training dataset of the action prediction model.

[0088] Optionally, this step can be specifically implemented as follows: inputting the second structural defect image from the second image data pair into the second image prediction model to perform image feature completion processing on the second structural defect image through the second image prediction model, thereby obtaining a third complete structural image; inputting the third complete structural image into the action prediction model to obtain first predicted operation action data; optimizing and adjusting the network parameters of the action prediction model based on the first predicted operation action and the operation action data of the second target robotic arm to complete the training of the action prediction model. Further, a second model loss function can be determined based on the first predicted operation action and the operation action data of the second target robotic arm, and the network parameters of the action prediction model can be optimized and adjusted according to the second model loss function to complete the training of the action prediction model.

[0089] For example, when a human operator controls a robotic arm to perform a task (such as picking up blocks) via a joystick, teach pendant, or even a VR device:

[0090] First, record video: The camera continuously captures footage of the robotic arm's movement and saves it frame by frame. (For example, an image of a second structural defect.)

[0091] Simultaneously, the system records the underlying control commands sent to the robotic arm at the exact moment each frame is captured. (This is the "Real Motion" label.)

[0092] Then, data pairing: After the data acquisition is completed, each frame of image is precisely paired with the corresponding robotic arm action command at that moment, forming tens of thousands of <image features, real action> data pairs.

[0093] For example, at time t, the camera captures the 100th frame of the image, showing the robotic arm moving to the right. At the same time t, the instruction sent to the robotic arm is [0.01, 0.0, 0.0, 0] (meaning: move 1 centimeter to the right, remain stationary in other dimensions, and do not grasp). Therefore, the <100th frame image, [0.01, 0.0, 0.0, 0]> constitutes a training sample.

[0094] In some embodiments, training the action prediction model may refer to, for example, Figure 3 The process is shown. Figure 3 This is a third schematic flowchart illustrating a control method for a robotic arm according to one embodiment of the present disclosure. Figure 3 The training process of the action prediction model is shown in the dashed box. The action prediction model includes at least a second image prediction model (encoder 2 and predictor 2) and a decoder. Based on encoder 2, predictor 2, and decoder, the data pair <image A, real action B> is taken as input. Image A is processed by encoder 2 and predictor 2 to obtain a completed image feature E4. E4 is input to the decoder so that the decoder predicts the robotic arm's predicted operation C in image A. At the same time, a loss function is calculated based on the real action B and the predicted operation C to adjust the decoder's network parameters, and the training is completed.

[0095] Specifically, a training image (from self-collected data) is input, and the image passes through an encoder -> predictor -> decoder, ultimately outputting a predicted action. The predicted action is compared with pre-prepared, paired real action labels (real action data) to determine the loss function (second model loss function, such as calculating the mean squared error, MSE Loss). The calculated error (loss) is used through backpropagation to update the parameters of the decoder, predictor, and encoder. In this way, the decoder is trained to make more accurate action predictions, the predictor is trained to complete the features that best help the decoder make correct predictions, and the encoder is trained to extract more essential visual features. Finally, through repeated iterations with a large amount of data, the entire system learns the ability to predict the correct action from (potentially incomplete) images. This improves the model's robustness, overcomes the problem of incomplete robotic arm structures in the input image, and makes the robot's visual control more stable and reliable, truly enabling it to work under non-ideal visual conditions. Through this embodiment, even when receiving a single-view robotic arm image input, or when the robotic arm structure itself in the input image is incomplete, it can stably predict and output high-quality actions.

[0096] Step 104: In response to receiving an image of any third target robotic arm, control the third target robotic arm with the operation action output by the motion prediction model.

[0097] The third target robotic arm can be any robotic arm in the real environment that needs to be controlled. After its image is acquired, it can be transmitted to the motion prediction model. The encoder in the motion prediction model extracts the initial image features (assuming they are incomplete feature maps or image features with image defects). Then, the predictor completes the image and transmits the completed image to the decoder to obtain the predicted operation action. Based on this operation action, the third target robotic arm can be controlled.

[0098] For example, in a real-world robotic arm environment, a camera captures a frame of image in real time, which is then fed into the aforementioned processing flow. The final motion prediction model can output a motion command (e.g., [0.1, 0.05, 0.0, 1]). This command is sent directly to the real robotic arm controller via a communication interface (e.g., ROS, Socket), and the robotic arm executes the motion. This demonstrates how to directly apply robotic arm image data with minimal inter-domain differences to the motion prediction model (i.e., execute the forward propagation process of encoder + predictor + decoder) to predict a relatively accurate motion, which can then be used to calculate accuracy and enable playback on a real device.

[0099] In summary, the technical solution provided by this disclosure uses image data pairs consisting of structurally complete images and images of structural defects of the robotic arm to train a first image prediction model. This enables the model to complete the image features of images with structural defects, thereby generating structurally complete images. The first image prediction model is then further optimized and adjusted using real robotic arm operation image data to obtain a second image prediction model. This solves the problem that conventional datasets (real robotic arm operation image data) are difficult to use for training image prediction models because they do not contain complete robotic arm information. Through two-stage training, the ability of the image prediction model to complete images with structural defects is transferred to conventional datasets, improving the model's stability. The training dataset used consists entirely of single images, not multi-view images, which solves the problem of relying heavily on complex multi-view image inputs in related technologies. The structural defect images of the robotic arm can be completed using only single-view image input. At the same time, it further lays a data foundation for the prediction accuracy of the subsequent motion prediction model. The motion prediction model is then trained based on the second image prediction model and a dataset including the second structural defect images and the second target robotic arm's operation data from the second image data pairs. With the more complete image information provided by the second image prediction model, the prediction stability and accuracy of the motion prediction model are improved, thereby enabling more precise control of the robotic arm.

[0100] Figure 4 This is a schematic structural block diagram of a control device for a robotic arm according to one embodiment of the present disclosure. Figure 4 As shown, a control device for a robotic arm according to an embodiment of this disclosure may include:

[0101] The first training module 401 is used to train a first image prediction model using a first training dataset. The first training dataset is a demonstration image data of a robotic arm operation, including a first image data pair of a first target robotic arm. Each first image data pair includes a first structural defect image of any first target robotic arm and a corresponding first structural complete image. The first image prediction model is used to perform image feature completion processing on the first structural defect image of any first target robotic arm.

[0102] Model optimization module 402 is used to optimize and adjust the first image prediction model using the second training dataset to obtain a second image prediction model. The second training dataset is real operation image data of the robotic arm, including second image data pairs of the second target robotic arm. Each second image data pair includes a second structural defect image and a second structural complete image of any second target robotic arm.

[0103] The second training module 403 is used to train an action prediction model based on the second image prediction model and the third training dataset. The third training dataset consists of the second structural defect image and the operation action data of the second target robotic arm in the second image data pair. The action prediction model is used to predict the operation action of the robotic arm based on the structural defect image of any robotic arm.

[0104] The motion control module 404 is used to control the third target robotic arm by means of the operation action output by the motion prediction model in response to receiving an image of any third target robotic arm.

[0105] As an optional embodiment, the first training module includes:

[0106] The image processing unit is used to perform occlusion processing on a preset image area of ​​the first structural complete image of the first target robotic arm to obtain the first structural defect image;

[0107] The feature extraction unit is used to extract a first image feature from the first structurally complete image and a second image feature from the first structurally defective image, wherein the first image feature is an image feature with complete image information and the second image feature is an image feature with image defects;

[0108] The first model training unit is used to train the first image prediction model based on the first image features and the second image features;

[0109] The first training dataset is a demonstration image data of robotic arm operation, including a first image data pair of a first target robotic arm. Each first image data pair includes a first structural defect image of any first target robotic arm and a corresponding first structural complete image. The first image prediction model is used to perform image feature completion processing on the structural defect image of any first target robotic arm.

[0110] As an optional embodiment, the first model training unit is further configured to:

[0111] The second image feature is input into the first image prediction model to perform feature completion processing on the second image feature through the first image prediction model to obtain the third image feature;

[0112] The first model loss function is determined based on the third image feature and the first image feature;

[0113] The network parameters of the first image prediction model are optimized and adjusted based on the first model loss function to complete the training of the first image prediction model.

[0114] As an optional embodiment, the first model training unit, used to optimize and adjust the network parameters of the first image prediction model based on the first model loss function to complete the training of the first image prediction model, can be implemented as follows:

[0115] Feature information of a preset image region is determined from the third image features and the first image features, respectively;

[0116] Calculate the image loss value between the feature information of the preset image region in the third image feature and the first image feature, and the image loss value is the first model loss function.

[0117] As an optional embodiment, the second training module includes:

[0118] The image completion unit is used to input the second structural defect image in the second image data pair into the second image prediction model, so as to perform image feature completion processing on the second structural defect image through the second image prediction model to obtain a third structurally complete image;

[0119] An action prediction unit is used to input the complete image of the third structure into the action prediction model to obtain first predicted operation action data;

[0120] The parameter tuning unit is used to optimize and adjust the network parameters of the motion prediction model based on the first predicted operation motion data and the operation motion data of the second target robotic arm, so as to complete the training of the motion prediction model.

[0121] As an optional embodiment, the parameter tuning unit is further configured to:

[0122] The second model loss function is determined based on the first predicted operation data and the operation data of the second target robotic arm.

[0123] The network parameters of the action prediction model are optimized and adjusted according to the loss function of the second model to complete the training of the action prediction model.

[0124] In summary, the technical solution provided by this disclosure uses image data pairs consisting of structurally complete images and images of structural defects of the robotic arm to train a first image prediction model. This enables the model to complete the image features of images with structural defects, thereby generating structurally complete images. The first image prediction model is then further optimized and adjusted using real robotic arm operation image data to obtain a second image prediction model. This solves the problem that conventional datasets (real robotic arm operation image data) are difficult to use for training image prediction models because they do not contain complete robotic arm information. Through two-stage training, the ability of the image prediction model to complete images with structural defects is transferred to conventional datasets, improving the model's stability. The training dataset used consists entirely of single images, not multi-view images, which solves the problem of relying heavily on complex multi-view image inputs in related technologies. The structural defect images of the robotic arm can be completed using only single-view image input. At the same time, it further lays a data foundation for the prediction accuracy of the subsequent motion prediction model. The motion prediction model is then trained based on the second image prediction model and a dataset including the second structural defect images and the second target robotic arm's operation data from the second image data pairs. With the more complete image information provided by the second image prediction model, the prediction stability and accuracy of the motion prediction model are improved, thereby enabling more precise control of the robotic arm.

[0125] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this disclosure. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0126] Below, for reference Figure 5 This describes an electronic device according to embodiments of the present disclosure. The electronic device may be either or both of a first device and a second device, or a standalone device independent of them, which may communicate with the first device and the second device to receive acquired input signals from them.

[0127] Figure 5 A block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0128] like Figure 5 As shown, the electronic device includes one or more processors and memory.

[0129] A processor can be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and can control other components in an electronic device to perform desired functions.

[0130] The memory can store one or more computer program products, and the memory can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program products can be stored on the computer-readable storage medium, and the processor can run the computer program products to implement the robotic arm control methods and / or other desired functions of the various embodiments of this disclosure described above.

[0131] In one example, the electronic device may also include input devices and output devices, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0132] In addition, the input device may also include, for example, a keyboard, a mouse, etc.

[0133] This output device can output various information to the outside, including determined distance information, direction information, etc. The output device may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0134] Of course, for the sake of simplicity, Figure 5 Only some of the components of the electronic device relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.

[0135] In addition to the methods and devices described above, embodiments of this disclosure may also be computer program products, including computer program instructions that, when executed by a processor, cause the processor to perform the steps in the control methods for a robotic arm according to various embodiments of this disclosure as described in the foregoing sections of this specification.

[0136] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0137] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the steps in the control methods for the robotic arm according to various embodiments of this disclosure as described in the foregoing portion of this specification.

[0138] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0139] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0140] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0141] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0142] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0143] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0144] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0145] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A control method for a robotic arm, characterized in that, The method includes: A first image prediction model is trained using a first training dataset, which is a demonstration image data of robotic arm operation, including a first image data pair of a first target robotic arm. Each first image data pair includes a first structural defect image of any first target robotic arm and a corresponding first structural complete image. The first image prediction model is used to perform image feature completion processing on the first structural defect image of any first target robotic arm. The first image prediction model is optimized and adjusted using the second training dataset to obtain the second image prediction model. The second training dataset consists of real operation image data of the robotic arm, including second image data pairs of the second target robotic arm. Each second image data pair includes a second structural defect image and a second structural integrity image of any second target robotic arm. A motion prediction model is trained based on the second image prediction model and the third training dataset. The third training dataset consists of the second structural defect image and the operation motion data of the second target robotic arm in the second image data pair. The motion prediction model is used to predict the operation motion of the robotic arm based on the structural defect image of any robotic arm. In response to receiving an image of any third target robotic arm, the third target robotic arm is controlled by the operation action output by the motion prediction model. The third target robotic arm is any robotic arm in the real environment that needs to be controlled. The step of training the first image prediction model using the first training dataset includes: occluding a preset image region of the first structurally complete image of the first target robotic arm to obtain a first structurally defective image; extracting a first image feature from the first structurally complete image and a second image feature from the first structurally defective image, wherein the first image feature is an image feature with complete image information and the second image feature is an image feature with image defects; and training the first image prediction model based on the first image feature and the second image feature. The step of training the action prediction model based on the second image prediction model and the third training dataset includes: inputting the second structural defect image from the second image data pair into the second image prediction model to perform image feature completion processing on the second structural defect image through the second image prediction model to obtain a third structurally complete image; inputting the third structurally complete image into the action prediction model to obtain first predicted operation action data; determining a second model loss function based on the first predicted operation action data and the operation action data of the second target robotic arm; and optimizing and adjusting the network parameters of the action prediction model according to the second model loss function to complete the training of the action prediction model.

2. The method according to claim 1, characterized in that, Training the first image prediction model based on the first image features and the second image features includes: The second image feature is input into the first image prediction model to perform feature completion processing on the second image feature through the first image prediction model to obtain the third image feature; The first model loss function is determined based on the third image feature and the first image feature; The network parameters of the first image prediction model are optimized and adjusted based on the first model loss function to complete the training of the first image prediction model.

3. The method according to claim 2, characterized in that, The step of determining the first model loss function based on the third image features and the first image features includes: Feature information of a preset image region is determined from the third image features and the first image features, respectively; Calculate the image loss value between the feature information of the preset image region in the third image feature and the first image feature, and the image loss value is the first model loss function.

4. A control device for a robotic arm, characterized in that, The device includes: The first training module is used to train a first image prediction model using a first training dataset. The first training dataset is a demonstration image data of a robotic arm operation, including a first image data pair of a first target robotic arm. Each first image data pair includes a first structural defect image of any first target robotic arm and a corresponding first structural complete image. The first image prediction model is used to perform image feature completion processing on the first structural defect image of any first target robotic arm. The model optimization module is used to optimize and adjust the first image prediction model using a second training dataset to obtain a second image prediction model. The second training dataset consists of real operation image data of the robotic arm, including second image data pairs of the second target robotic arm. Each second image data pair includes a second structural defect image and a second structurally complete image of the second target robotic arm. The second training module is used to train an action prediction model based on the second image prediction model and a third training dataset. The third training dataset consists of the second structural defect image in the second image data pair and the operation action data of the second target robotic arm. The action prediction model is used to predict the operation action of the robotic arm based on the structural defect image of any robotic arm. The motion control module is used to control the third target robotic arm in response to receiving an image of any third target robotic arm, by controlling the third target robotic arm through the operation action output by the motion prediction model. The third target robotic arm is any robotic arm in the real environment that needs to be controlled. The step of training the first image prediction model using the first training dataset includes: occluding a preset image region of the first structurally complete image of the first target robotic arm to obtain a first structurally defective image; extracting a first image feature from the first structurally complete image and a second image feature from the first structurally defective image, wherein the first image feature is an image feature with complete image information and the second image feature is an image feature with image defects; and training the first image prediction model based on the first image feature and the second image feature. The step of training the action prediction model based on the second image prediction model and the third training dataset includes: inputting the second structural defect image from the second image data pair into the second image prediction model to perform image feature completion processing on the second structural defect image through the second image prediction model to obtain a third structurally complete image; inputting the third structurally complete image into the action prediction model to obtain first predicted operation action data; determining a second model loss function based on the first predicted operation action data and the operation action data of the second target robotic arm; and optimizing and adjusting the network parameters of the action prediction model according to the second model loss function to complete the training of the action prediction model.

5. A robot, characterized in that, The robot includes a robotic arm, which is operated and controlled by the robotic arm control method described in any one of claims 1-3.

6. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-3.

7. A computer program product comprising computer program instructions, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-3.

Images