Method for training visual language action model and mechanical arm operating device

Abstract

The present disclosure provides a method for training a visual language action model and a robot operating device. The method comprises obtaining a multi-modal training dataset, the multi-modal training dataset comprising a plurality of first multi-modal training data and a plurality of second multi-modal training data, each of the first multi-modal training data and the second multi-modal training data comprising a robot operating video, text description information, and true joint angle information corresponding to each video frame. Based on the multi-modal training dataset, the first visual language action model to be trained is iteratively trained for multiple rounds until a preset end training condition is met, and a target visual language action model is obtained. In each round of iterative training, a strategy failure associated video frame is determined, and the frame weight of the strategy failure associated video frame in the next round of iterative training is improved. Thus, the adaptability of the target visual language action model obtained by training to complex and high-difficulty operation scenarios can be effectively improved, and the success rate of executing tasks can be improved.

Description

Technical Field

[0001] This disclosure relates to deep learning technology, and in particular to a training method for a visual language action model and a robotic arm operating device. Background Technology

[0002] Vision-Language-Action (VLA) models are the decision-making systems for robots, with their core function being to convert robot observations into actions. VLAs are a core technological component in Robot Imitation Learning (IL), and their core value lies in their cross-modal fusion architecture, deeply coupling the robot's environmental perception (visual modality, such as video of the operating scene), task semantic understanding (language modality, such as natural language commands), and action sequence generation (action modality, such as robot joint motion trajectories), providing end-to-end "perception-understanding-execution" capabilities for IL. However, current mainstream VLA training generally adopts an isolated paradigm of "generation as the endpoint," meaning the model only completes the unidirectional generation task of "visual-language input to action sequence output," without forming a feedback mechanism with downstream IL policy training. Ultimately, this leads to low success rates for robots driven by such VLAs in actual task execution. Summary of the Invention

[0003] To address the aforementioned technical problems, this disclosure provides a training method for a visual language action model and a robotic arm operating device.

[0004] One aspect of this disclosure provides a method for training a visual language action model, comprising: acquiring a multimodal training dataset, the multimodal training dataset including multiple first multimodal training data and multiple second multimodal training data, each of the first multimodal training data and each of the second multimodal training data respectively including a robotic arm operation video, text description information, and ground truth joint angle information of the robotic arm corresponding to each video frame in the robotic arm operation video, the text description information being used to describe the video content of the robotic arm operation video; performing multiple rounds of iterative training on a first visual language action model to be trained based on the multimodal training dataset until a preset training termination condition is met, and obtaining the target visual language action model from the first visual language action model to be trained. A visual language action model is defined, wherein, in each round of iterative training, based on the prediction policy inference results corresponding to each video frame in each of the robotic arm operation videos output by the first visual language action model to be trained, policy failure-associated video frames are determined, and the frame weight of the policy failure-associated video frames in the next round of iterative training is increased. The prediction policy inference result corresponding to each video frame includes a sequence of predicted joint angles of the robotic arm. The sequence of predicted joint angles includes the predicted joint angle information corresponding to the future video frames predicted based on the video frame, and the predicted joint angle information corresponding to the video frame. The policy failure-associated video frames represent the video frames that cause the first visual language action model to be trained to generate policy inference results that lead to task execution failure.

[0005] In another aspect of this disclosure, a robotic arm manipulation device is provided for use with a robot. The robotic arm manipulation device includes a robotic arm, a robotic arm control device, and the aforementioned target visual language action model. The target visual language action model is used to perform reasoning based on input operation commands and video of the robotic arm's working environment, and outputs a strategy reasoning result. The strategy reasoning result includes a sequence of joint angles of the robotic arm. The robotic arm control device controls the robotic arm to execute the strategy reasoning result.

[0006] In another aspect of this disclosure, a training apparatus for a visual language action model is provided, comprising: a first training data acquisition module, configured to acquire a multimodal training dataset, the multimodal training dataset including multiple first multimodal training data and multiple second multimodal training data, each of the first multimodal training data and each of the second multimodal training data respectively including a robotic arm operation video, text description information, and ground truth joint angle information of the robotic arm corresponding to each video frame in the robotic arm operation video, the text description information being used to describe the video content of the robotic arm operation video; and a first model training module, configured to perform multiple rounds of iterative training on a first visual language action model to be trained based on the multimodal training dataset until a preset end-of-training condition is met, and the model to be trained is then trained. A visual language action model is used to obtain a target visual language action model. In each round of iterative training, based on the prediction policy inference results corresponding to each video frame in the robotic arm operation video output by the first visual language action model to be trained, the policy failure associated video frames are determined, and the frame weight of the policy failure associated video frames in the next round of iterative training is increased. The prediction policy inference result corresponding to each video frame includes the predicted joint angle sequence of the robotic arm. The predicted joint angle sequence includes the predicted joint angle information corresponding to the future video frame predicted based on the video frame, and the predicted joint angle information corresponding to the video frame. The policy failure associated video frame represents the video frame that causes the first visual language action model to be trained to generate a policy inference result that leads to task execution failure.

[0007] In another aspect of this disclosure, an electronic device is provided, comprising: a memory for storing a computer program; and a processor for executing the computer program stored in the memory, wherein when the computer program is executed, it implements the method described above.

[0008] In another aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, characterized in that, when the computer program is executed by a processor, it implements the above-described method.

[0009] In another aspect of this disclosure, a computer program product is provided, including computer program instructions that, when executed by a processor, implement the method described above.

[0010] Based on the embodiments of this disclosure, a multimodal input integrating vision, language, and action is constructed by integrating robotic arm operation videos (visual information), text description information (semantic information), and ground truth joint angle information (action information) corresponding to each video frame through a multimodal training dataset. This allows the first visual language action model to be trained to capture environmental and operational details through robotic arm operation videos, clarify task semantics through text description information, and calibrate action prediction accuracy with reference to ground truth joint angle information. This significantly improves the clarity of semantic understanding of the task and the accuracy of action prediction of the trained target visual language action model. Meanwhile, by identifying and weighting the video frames associated with policy failures (i.e., key frames leading to task failure) in each iteration of training, the traditional isolated training paradigm of "generation is the end" is broken. This allows the first visual language action model to no longer learn indiscriminately from all video frames, but instead selectively strengthens the learning weights of video frames associated with policy failures. This shifts training resources towards key interaction stages (video frames associated with policy failures), effectively improving the adaptability of the trained target visual language action model to complex and challenging operation scenarios (such as folding clothes, cleaning a table, and throwing bottles), thus increasing the success rate of task execution. In addition, through an adaptive training strategy that increases the frame weights of video frames associated with policy failures, the first visual language action model can focus on high-value training samples (video frames associated with policy failures) for targeted optimization, accelerating the convergence speed of the first visual language action model on the core task difficulties, and greatly improving training efficiency and quality.

[0011] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0012] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0013] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein:

[0014] Figure 1 This is a flowchart illustrating a training method for a visual language action model provided in an exemplary embodiment of this disclosure.

[0015] Figure 2 This is a flowchart illustrating step S100 provided in an exemplary embodiment of this disclosure.

[0016] Figure 3 This is a flowchart illustrating step S104 provided in an exemplary embodiment of this disclosure.

[0017] Figure 4This is a flowchart illustrating step S104 provided in another exemplary embodiment of this disclosure.

[0018] Figure 5 This is a flowchart illustrating a method for training a visual language action model provided in another exemplary embodiment of this disclosure.

[0019] Figure 6 This is a flowchart illustrating a method for training a visual language action model provided in yet another exemplary embodiment of this disclosure.

[0020] Figure 7 This is a schematic diagram illustrating the application of the training method for visual language action models provided in this publication.

[0021] Figure 8 This is a schematic diagram of a robotic arm operating device provided in an exemplary embodiment of the present disclosure.

[0022] Figure 9 This is a schematic diagram of the structure of one embodiment of the training device for the visual language action model of this disclosure.

[0023] Figure 10 This is a schematic diagram of the structure of an application embodiment of the electronic device disclosed herein. Detailed Implementation

[0024] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0025] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0026] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0027] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0028] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.

[0029] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0030] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0031] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0032] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0033] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0034] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0035] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0036] Current training of visual-language action models generally adopts an isolated paradigm of "generation as endpoint," meaning the model only completes unidirectional generation from "visual-language input" to "action sequence output," without forming a closed-loop feedback mechanism with downstream IL policy training. This prevents the reverse transmission of feedback on the policy's effectiveness in real-world physical environments (such as task success / failure signals and quantified action deviation values) back to the model. This directly leads to two problems: firstly, the success rate of action sequences generated by such visual-language action models driving real robots to perform tasks is low; secondly, because the model cannot focus on high-value, difficult samples for targeted optimization, it consumes a large amount of redundant data for indiscriminate training, resulting in low overall training efficiency.

[0037] Figure 1 This is a flowchart illustrating a training method for a visual language action model provided in an exemplary embodiment of this disclosure. This embodiment can be applied to electronic devices, such as... Figure 1 As shown, it includes the following steps:

[0038] Step S100: Obtain the multimodal training dataset.

[0039] The multimodal training dataset includes multiple first multimodal training datasets and multiple second multimodal training datasets. Each first multimodal training dataset and each second multimodal training dataset includes: a video of the robotic arm operation, textual description information, and ground truth joint angle information of the robotic arm corresponding to each video frame in the video of the robotic arm operation.

[0040] The text description information is used to describe the video content of the robotic arm operation video. For example, the text description information includes target object attribute information and interaction relationships. The interaction relationship represents the process and goal of the robotic arm's strategy reasoning result for the target object. For instance, if the robotic arm operation video includes: the robotic arm picks up a green apple, the text description information could include: target object = apple, target object attribute information = green, interaction relationship = robotic arm picks up apple.

[0041] Robotic arm operation videos are dynamic image data recordings of a robotic arm performing a preset target task (such as grasping, assembling, folding, etc.) in a real or simulated environment. Robotic arm operation videos may include, for example,: visual information of the robotic arm's movements (such as joint motion, actuator operation, and timing logic), visual information of the target object directly acted upon by the robotic arm (such as the object's shape, attributes, and interaction with the robotic arm), and visual information of the background (visual information of the robotic arm's working environment, which may include static scenes, dynamic interference, and implicit attributes).

[0042] The ground truth joint angle information for each video frame refers to the actual angle values ​​of each joint of the robotic arm in the specific posture presented in that video frame. In one embodiment, the ground truth joint angle information for each video frame includes the angle values ​​of each joint on the robotic arm in that video frame. For example, assuming a dual-arm design, the ground truth joint angle information of the robotic arm can be 14-dimensional data, including the angle values ​​of the 7 joints on the left robotic arm and the angle values ​​of the 7 joints on the right robotic arm.

[0043] Step S110: Based on the multimodal training dataset, perform multiple rounds of iterative training on the first visual language action model to be trained until the preset end-of-training conditions are met, and obtain the target visual language action model from the first visual language action model to be trained.

[0044] In each round of iterative training, based on the prediction policy reasoning results corresponding to each video frame in each robotic arm operation video output by the first visual language action model to be trained, the video frames associated with policy failure are determined, and the frame weight of the video frames associated with policy failure in the next round of iterative training is increased.

[0045] The prediction policy inference result for each video frame includes: a sequence of predicted joint angles for the robotic arm. This sequence includes: predicted joint angle information for future video frames predicted based on the current video frame, and predicted joint angle information for the current video frame. For example, the predicted joint angle sequence for video frame i includes: predicted joint angle information for video frames i to i+n respectively. The predicted joint angle information includes the angle values ​​of each joint in the robotic arm. For example, the ground truth joint angle information for the robotic arm may include: predicted angle values ​​for the 7 joints on the left robotic arm and predicted angle values ​​for the 7 joints on the right robotic arm. The policy failure associated video frame represents the video frame that causes the first visual language action model to be trained to generate a policy inference result that leads to task execution failure. The task represents the embodied operation target object that the first visual language action model to be trained drives the robotic arm to perform. For example, the angle difference between each joint angle value in the predicted joint angle information corresponding to the i-th video frame and the corresponding joint angle value in the true joint angle information is calculated. Then, the average value of the above angle difference is calculated. When the average value exceeds the preset angle difference threshold, the video frame can be identified as a strategy failure associated video frame.

[0046] Frame weight is a numerical score assigned to each video frame in a video. It is used to quantify the importance and relevance of that video frame to the model's current decision task. Frame weight can help the model know which parts of the input video to focus on in order to better understand the environment, make inferences, and generate appropriate actions.

[0047] For example, a PiZero (Pi0) model can be used as a first-person visual language action model.

[0048] In one implementation, the first multimodal training data and the second multimodal training data can be input into the first visual language action model to be trained. The first visual language action model outputs the predicted joint angle sequence (predicted policy inference result) corresponding to each video frame. In each round of iterative training, based on the difference between the predicted joint angle sequence corresponding to each video frame and the ground truth joint angle information of the corresponding video frame, a preset loss function is used to determine the loss function value. Furthermore, based on the predicted policy inference result corresponding to each video frame, the policy failure-associated video frames are determined, and the frame weights of the policy failure-associated video frames are increased. This generates the frame weights of the policy failure-associated video frames to be used in the next round of iterative training, so that the policy failure-associated video frames are trained iteratively with these frame weights in the next round of iterative training. The preset loss function may include, but is not limited to, cross-entropy error function or mean squared error function. The process iteratively executes operations such as determining the loss function value, identifying video frames associated with policy failures, and increasing the frame weights of these frames. This iterative adjustment of the parameters of the first visual language action model to be trained continuously reduces the loss function value until it converges, satisfying the preset training termination condition. The trained first visual language action model is then used as the target visual language action model. Parameter optimizers such as Stochastic Gradient Descent (SGD), Adaptive Gradient Descent (Adagrad), Adaptive Moment Estimation (Adam), and Root Mean Square Prop (RMSprop) can be used to adjust the parameters of the first visual language action model. For example, the parameter optimizer can calculate the gradient of each parameter of the first visual language action model to be trained, and adjust these parameters along the direction of the gradient. The gradient represents the direction in which the loss function value decreases the most. This iterative process of determining the loss function value continues until it no longer decreases, completing the training of the first visual language action model and yielding the target visual language action model.

[0049] In this embodiment, a multimodal input integrating visual, language, and action information is constructed by integrating robotic arm operation videos (visual information), text description information (semantic information), and ground truth joint angle information (action information) corresponding to each video frame through a multimodal training dataset. This allows the first visual language action model to be trained to capture environmental and operational details through robotic arm operation videos, clarify task semantics through text description information, and calibrate action prediction accuracy with reference to ground truth joint angle information. This significantly improves the clarity of semantic understanding of the task and the accuracy of action prediction of the trained target visual language action model. Meanwhile, by identifying and weighting the video frames associated with policy failures (i.e., key frames leading to task failure) in each iteration of training, the traditional isolated training paradigm of "generation is the end" is broken. This allows the first visual language action model to no longer learn indiscriminately from all video frames, but instead selectively strengthens the learning weights of video frames associated with policy failures. This shifts training resources towards key interaction stages (video frames associated with policy failures), effectively improving the adaptability of the trained target visual language action model to complex and challenging operation scenarios (such as folding clothes, cleaning a table, and throwing bottles), thus increasing the success rate of task execution. In addition, through an adaptive training strategy that increases the frame weights of video frames associated with policy failures, the first visual language action model can focus on high-value training samples (video frames associated with policy failures) for targeted optimization, accelerating the convergence speed of the first visual language action model on the core task difficulties, and greatly improving training efficiency and quality.

[0050] In some alternative implementations, the textual description information in this disclosure may include: foreground information, background information, and interaction relationships of the robotic arm operation video.

[0051] Foreground information includes target object attribute information. Target object attribute information may include, for example, the target object's color, texture, material, and size. The interaction relationship (task semantics) represents the process and goal of the robotic arm's strategy reasoning for the target object. In other words, it can be understood as what the robotic arm does to the target object. For example, the interaction relationship may include: picking up an apple, folding clothes, cleaning a table, and discarding a bottle.

[0052] Background information can include information about environmental elements present around the robotic arm performing its task in the video of mechanical operation, affecting task perception and execution. Information about environmental elements can include, for example, their material, color, and texture. For instance, assuming clothing (the target object) is placed on a wooden table covered with a white tablecloth, the background information could include: white tablecloth, wooden table.

[0053] In this embodiment, multiple robotic arm operation videos are acquired based on different robotic arm perspectives. A robotic arm perspective refers to the camera perspective on the robotic arm. For example, the robotic arm perspective may include: a left perspective (the perspective of a camera mounted on the left robotic arm), a right perspective (the perspective of a camera mounted on the right robotic arm), and a middle perspective (the perspective of a camera mounted between the left and right robotic arms). The middle perspective of the robotic arm is a fixed perspective and does not move synchronously with the left / right robotic arms; that is, the camera mounted between the left and right robotic arms is a fixed camera and does not move synchronously with the left / right robotic arms. The left and right perspectives of the robotic arm are both moving perspectives and move synchronously with the corresponding robotic arms; that is, the camera mounted on the left robotic arm moves synchronously with the left robotic arm, and the camera mounted on the right robotic arm moves synchronously with the right robotic arm. Therefore, the multimodal training dataset may include robotic arm operation videos acquired based on the left perspective, robotic arm operation videos acquired based on the right perspective, and robotic arm operation videos acquired based on the middle perspective.

[0054] In this embodiment of the disclosure, the first multimodal training data corresponds to at least one second multimodal training data, and the corresponding first multimodal training data and second multimodal training data have different target object attribute information and / or background information.

[0055] In one implementation, each first multimodal training data may correspond to one or more second multimodal training data. In the corresponding first and second multimodal training data: the ground truth joint angle information is the same, the attributes and / or background information of the target object in the robotic arm operation video are different, the remaining information of the robotic arm operation video is the same, the attributes and / or background information of the target object in the text description information are different, and the remaining information of the text description information is the same.

[0056] In this embodiment, multimodal training data is constructed by collecting robotic arm operation videos from different robotic arm perspectives. This ensures that the first visual language action model to be trained can learn the object shape, material, and motion trajectory features from multiple perspectives, significantly improving the model's adaptability to the operational needs of the robotic arm from different observation angles in real-world scenarios and enhancing the policy generalization ability. Simultaneously, by differentiating the target object attributes and background information between the first and second multimodal training data, rich scene variables are provided for the first visual language action model to be trained, effectively enhancing the adaptability and task execution stability of the trained target visual language action model in complex and variable operational scenarios.

[0057] Figure 2 This is a flowchart illustrating step S100 provided in an exemplary embodiment of this disclosure. In some alternative embodiments, such as Figure 2As shown, step S100 may include the following steps:

[0058] Step S101: Obtain multiple first multimodal training data.

[0059] Among them, several of the robotic arm operation videos in the first multimodal training data were collected from different robotic arm perspectives.

[0060] In one implementation, multiple robotic arm operation videos can be acquired in the real working environment or simulation environment of the robotic arm, based on the left, right, and center viewpoints of the robotic arm. For each acquired robotic arm operation video, a visual language model can be used to generate textual description information for that video. For example, the Qwen2.5-V large model can be used. The ground truth joint angle information of the robotic arm corresponding to each video frame in the robotic arm operation video can be obtained through an angle encoder, or the ground truth joint angle information can be obtained by data fusion using multiple sensors. For example, by combining tilt sensors and accelerometers, the tilt angle and acceleration information of the robotic arm joints can be measured, and then the ground truth joint angle information can be obtained through data processing and algorithm calculation. Alternatively, an external Inertial Measurement Unit (IMU) can be used to obtain the ground truth joint angle information through a series of calculations, such as establishing a coordinate system and acquiring attitude data. A first multimodal training dataset is constructed from the robotic arm operation video, the corresponding textual description information, and the ground truth joint angle information corresponding to each video frame of the robotic arm operation video.

[0061] Step S102: For multiple first multimodal training data, adjust the target object attribute information and / or background information in the text description information of the first multimodal training data to obtain the text description information of the second multimodal training data.

[0062] Specifically, the text description information in the first multimodal training data can be used as the target object attribute information and / or background information as a shared text prompt. The shared text prompt can be modified to obtain the text description information of the second multimodal training data.

[0063] For example, in the text description information of the first multimodal training data: target object attribute information = red cotton T-shirt, and background information = white tablecloth. By adjusting the color in the target object attribute information to white, the text description information of the resulting second multimodal training data includes: target object attribute information = white cotton T-shirt, and background information = white tablecloth. Alternatively, the text description information of the first multimodal training data includes: target object attribute information = red apple, and background information = wooden table. By adjusting the material in the background information to plastic, the text description information of the resulting second multimodal training data includes: target object attribute information = red apple, and background information = plastic table.

[0064] Step S103: Based on the robotic arm operation video of the first multimodal training data and the text description information of the second multimodal training data, determine the robotic arm operation video of the second multimodal training data.

[0065] The robotic arm operation videos in the first multimodal training data and the robotic arm operation videos in the second multimodal training data have different target object attribute information and / or background information. The ground truth joint angle information from the first multimodal training data is used as the ground truth joint angle information in the second multimodal training data, thereby completing the construction of the second multimodal training data. The first multimodal training data corresponds to the second multimodal training data.

[0066] In this embodiment, the target object attribute information and / or background information (included in the text description information) in the first multimodal training data can be adjusted in different ways to obtain the text description information in multiple second multimodal training data. Then, based on the robotic arm operation video of the first multimodal training data and the text description information of the multiple second multimodal training data, multiple robotic arm operation videos of the second multimodal training data are generated, and the first multimodal training data corresponds to the multiple second multimodal training data.

[0067] In an optional implementation, in this embodiment of the disclosure, step S103 may include: extracting the depth map corresponding to each video frame in the robotic arm operation video of the first multimodal training data to obtain a depth map sequence, and then using a video generation model based on the depth map sequence and the text description information of the second multimodal training data to obtain the robotic arm operation video of the second multimodal training data.

[0068] For example, the depth map sequence may include depth maps corresponding to each video frame in the robotic arm operation video of the first multimodal training data. The depth map of each video frame can be extracted using a depth prediction model, such as the Video Depth Anything model.

[0069] Step S104: Construct a multimodal training dataset based on at least one first multimodal training data and at least one second multimodal training data.

[0070] Specifically, a multimodal training dataset can be constructed from each of the first multimodal training data and the corresponding second multimodal training data.

[0071] In this embodiment, based on existing first multimodal training data, second multimodal training data is constructed by adjusting text description information and using depth maps to assist in video generation. This eliminates the need to collect new robotic arm operation videos from scratch, thereby not only efficiently expanding the training data but also significantly reducing data acquisition costs. Simultaneously, the second multimodal training data is generated by adjusting the target object attribute information and / or background information of the first multimodal training data. This ensures that the constructed multimodal training dataset covers diverse scene variables (such as target objects with different attributes and background environments of varying complexity). This differentiated design allows the first visual language action model to be trained to access rich operation scene samples, learn strategies for coping with different scenarios, and effectively enhance its adaptability and task execution stability in complex and variable real-world operation scenarios.

[0072] Figure 3 This is a flowchart illustrating step S104 provided in an exemplary embodiment of this disclosure. In some alternative embodiments, such as Figure 3 As shown, step S104 may include the following steps:

[0073] Step S1041: For at least one second multimodal training data, extract the depth map of each video frame in the robotic arm operation video of the second multimodal training data to obtain a test depth map sequence.

[0074] The test depth map sequence may include the depth maps corresponding to each video frame in the robotic arm operation video of the second multimodal training data. For example, the depth map of each video frame can be extracted using a depth prediction model (e.g., the Video Depth Anything model).

[0075] Step S1042: Based on the test depth map sequence and the ground truth depth map sequence, determine the depth map fidelity corresponding to the robotic arm operation video of the second multimodal training data.

[0076] The ground truth depth map sequence is the depth map sequence of the robotic arm operation video that generated the second multimodal training data; that is, the ground truth depth map sequence is the depth map sequence corresponding to the robotic arm operation video in the first multimodal training data corresponding to the second multimodal training data. Depth map fidelity indicates the degree of similarity between the depth map sequence extracted from the robotic arm operation video of the second multimodal training data (test depth map sequence) and the depth map sequence of the robotic arm operation video that generated the second multimodal training data (ground truth depth map sequence).

[0077] For example, the difference or similarity between the test depth map sequence and the ground truth depth map sequence can be calculated, and this difference or similarity can be determined as the depth map reconstruction degree. For instance, the mean squared error (MSE), root mean squared error (RMSE), mean absolute error (MAE), or structural similarity index (SSIM) between the test depth map sequence and the ground truth depth map sequence can be used as the aforementioned depth map reconstruction degree.

[0078] Step S1043: Based on the depth map reconstruction degree corresponding to the robotic arm operation video of each second multimodal training data, determine the second multimodal training data used to construct the multimodal training dataset.

[0079] In one implementation, a preset depth map fidelity threshold can be set in advance. For example, assuming the depth map fidelity includes the MSE, RMSE, or MAE values ​​between the test depth map sequence and the ground truth depth map sequence, then if the depth map fidelity corresponding to the robotic arm operation video is less than the preset depth map fidelity threshold, the second multimodal training data corresponding to the robotic arm operation video is determined as the second multimodal training data for constructing the multimodal training dataset; otherwise, the second multimodal training data is discarded. Similarly, assuming the depth map fidelity includes the SSIM between the test depth map sequence and the ground truth depth map sequence, then if the depth map fidelity corresponding to the robotic arm operation video is greater than or equal to the preset depth map fidelity threshold, the second multimodal training data corresponding to the robotic arm operation video is determined as the second multimodal training data for constructing the multimodal training dataset; otherwise, the second multimodal training data is discarded.

[0080] Step S1044: Construct the training dataset based on the second multimodal training data used to construct the multimodal training dataset and at least one second multimodal training data.

[0081] For example, a multimodal training dataset can be constructed based on a second multimodal training data used to construct the multimodal training dataset, and a first multimodal training data corresponding to the second multimodal training data used to construct the multimodal training dataset.

[0082] In this embodiment of the disclosure, a test depth map sequence is extracted from the robotic arm operation video of the second multimodal training data and compared with the ground truth depth map sequence used to generate the robotic arm operation video to calculate the depth map restoration degree. Based on this, the second multimodal training data with qualified depth restoration quality is selected, thereby effectively ensuring the quality of the robotic arm operation video and providing reliable data support for subsequent model training.

[0083] Figure 4 This is a flowchart illustrating step S104 provided in another exemplary embodiment of this disclosure. In some alternative embodiments, such as Figure 4 As shown, step S104 may also include the following steps:

[0084] Step S1045: Generate multiple initial datasets.

[0085] The proportion of the first multimodal training data and the second multimodal training data differs in each initial dataset.

[0086] For example, four initial datasets can be constructed, in which the proportions of the first multimodal training data and the second multimodal training data in the four initial datasets are 25%, 50%, 75%, and 95%, respectively.

[0087] Step S1046: Based on each initial dataset, train the second visual language action model to be trained to obtain multiple test visual language action models.

[0088] Each initial dataset corresponds to a test visual language action model trained on that initial dataset. The second visual language action model may be, for example, the PiZero model.

[0089] For example, the second visual language action model can be trained by training the first visual language action model to be trained, or the initial dataset can be used to perform multiple rounds of iterative training on the second visual language action model to be trained until the preset end-of-training conditions are met, and the corresponding test visual language action model is obtained.

[0090] Step S1047: Based on the policy performance of each test visual language action model, determine the multimodal training dataset from multiple initial datasets.

[0091] Among them, strategy performance refers to the performance of the robotic arm driven by the test visual language action model in completing the task.

[0092] In one instance, policy performance can include a performance score, where a higher score indicates better policy performance of the tested visual language action model. The initial dataset corresponding to the best-performing tested visual language action model can be selected as the multimodal training dataset.

[0093] For example, the performance score of each test visual language action model can be determined based on the success rate of the robotic arm completing the task and human scoring. Specifically, the correspondence between task success rate and score can be set, as well as human scoring items can be set. For example, human scoring items may include the time taken for the robotic arm to complete the task, the response delay of the model's decision generation, and the policy's tolerance to noise or abnormal input.

[0094] In this embodiment of the disclosure, multiple initial datasets with different proportions of first and second multimodal training data are generated, and corresponding test visual language action models are trained respectively. Then, the optimal initial dataset is selected as the multimodal training dataset based on the strategy performance of each test visual language action model in driving the robotic arm to complete the task. This can accurately determine the optimal data composition ratio for model training, as well as the impact of different mixing ratios, different backgrounds or materials on model performance, avoiding training bias caused by blindly setting the data ratio, and enhancing the scientificity and efficiency of model training.

[0095] Figure 5 This is a flowchart illustrating a method for training a visual language action model provided in another exemplary embodiment of this disclosure. In some alternative implementations, such as... Figure 5 As shown, determining the associated video frames where the strategy failed may include the following steps:

[0096] Step S200: For each video frame, determine the motion deviation score based on the predicted joint angle sequence corresponding to the video frame and the ground truth joint angle information corresponding to the video frame.

[0097] The first visual language action model to be trained can predict the joint angle information of the next m (e.g., m=50) video frames after a video frame based on a video frame. The joint angle information of the video frame and the predicted joint angle information of the future video frames are composed of a predicted joint angle series. Based on the predicted joint angle sequence and the ground truth joint angle information of the corresponding video frames, the MSE value can be calculated. Based on the MSE value and the correspondence between the preset MSE value and the score, the score corresponding to the MSE is determined and the score is determined as the action deviation score corresponding to the video frame.

[0098] Step S210: Based on the predicted joint angle sequence corresponding to the video frame, determine the joint angular acceleration, and based on the joint angular acceleration, determine the motion smoothness score.

[0099] In one implementation, based on the time information and joint angle information corresponding to each future video frame (predicted from the video frame) corresponding to the predicted joint angle sequence, the angular velocity corresponding to each future video frame is determined. When the joint angle information includes multiple joint angles, the angular velocity corresponding to the future video frame can be the average value of the angular velocities corresponding to each joint angle. Then, based on the angular velocity corresponding to each future video frame and the time information corresponding to each video frame, the angular acceleration corresponding to each future video frame is determined. The average value of the angular acceleration corresponding to each future video frame is calculated and used as the joint angular acceleration corresponding to the video frame.

[0100] Based on the joint angular acceleration corresponding to the video frame and the pre-defined correspondence between joint angular acceleration and score, the score corresponding to the joint angular acceleration is determined, and this score is used as the motion smoothness score corresponding to the video frame.

[0101] Step S220: Based on the predicted joint angle sequence and preset joint angle threshold information corresponding to the video frame, determine the joint angle compliance score.

[0102] The preset joint angle threshold information includes the angle thresholds for each joint of the robotic arm, i.e., the maximum angle that each joint can open. The difference between each joint angle in the predicted joint angle sequence and the preset joint angle threshold information can be calculated. The difference between each joint angle and the preset joint angle threshold information can be considered as the average of the differences between the joint angles in that specific joint angle sequence and their corresponding thresholds in the preset joint angle threshold information. The average of the differences between each joint angle in the predicted joint angle sequence and the preset joint angle threshold information can be used as the joint compliance value corresponding to that video frame. Based on this joint compliance value and the correspondence between the preset joint compliance value and the score, the score corresponding to the joint compliance value is determined, and this score is designated as the joint angle compliance score for that video frame.

[0103] Step S230: Based on the motion deviation score, motion smoothness score, and joint angle compliance score corresponding to each video frame, a preset reward function is used to determine the video frames associated with strategy failure in each video frame.

[0104] Among them, the reward function value corresponding to each video can be determined based on the preset reward function (Rewards), and video frames with reward function values ​​lower than the preset reward threshold are identified as video frames associated with policy failure.

[0105] In one implementation, the preset reward function can be R. total = W1×a1+W2×a2+W3×a3, where W1, W2, and W3 are the weights for motion deviation, smoothness, and angle compliance, respectively; a1, a2, and a3 are the scores for motion deviation, motion smoothness, and joint angle compliance, respectively; R total This is the reward function value.

[0106] In this implementation, a scoring system is constructed based on three dimensions: motion deviation, motion smoothness, and joint angle compliance. Combined with a preset reward function, the system accurately locates video frames associated with strategy failures, ensuring that the iteration process can specifically focus on strengthening the weak links of the strategy and efficiently promote the continuous optimization of the model's strategy performance.

[0107] Figure 6 This is a flowchart illustrating a method for training a visual language action model provided in yet another exemplary embodiment of this disclosure. In some alternative implementations, such as Figure 6 As shown, the video generation model can be trained in the following ways:

[0108] Step S300: Obtain the video model training dataset.

[0109] The video model training dataset includes multiple first training data sets and multiple second training data sets. Each first training data set and each second training data set includes a sequence of labeled depth maps and corresponding shared descriptive information. This shared descriptive information includes target object attribute information, background information, and interaction information within the labeled depth map sequence. The multiple labeled depth map sequences are obtained from labeled robotic arm operation videos collected from different robotic arm perspectives. Each first training data set corresponds to at least one second training data set. The corresponding first training data set and second training data set have the same labeled depth map sequence, but different target object attribute information and / or background information.

[0110] In one implementation, multiple labeled robotic arm operation videos are acquired. For example, robot videos from a standard robot dataset can be selected as labeled robotic arm operation videos. For instance, robot videos from the AgiBot dataset, also known as AgiBot World, is a dataset of millions of real-world robots released by Logic Robotics in collaboration with the Shanghai Artificial Intelligence Laboratory, the National and Local Jointly Built Humanoid Robotics Innovation Center, and Shanghai Coopers. This dataset is based on real-world scenarios, a versatile hardware platform, and comprehensive quality control. The AgiBot dataset includes robot videos from different robotic arm perspectives.

[0111] For each labeled robotic arm operation video, a visual language model can be used to generate descriptive information describing the video content of the labeled robotic arm operation video. This descriptive information is used as the shared descriptive information of the first training data. A depth prediction model is used to extract the depth map sequence corresponding to the labeled robotic arm operation video as the labeled depth map sequence of the first training data.

[0112] For each first training data, the target object attribute information and / or background information in the shared description information of the first training data can be adjusted to obtain the shared description information of the second training data, and the labeled depth image sequence in the first training data can be used as the labeled depth image sequence of the second training data.

[0113] Step S310: Based on the video model training dataset, perform multiple rounds of iterative training on the video generation model to be trained until the preset model training conditions are met, and obtain the video generation model from the video generation model to be trained.

[0114] The video generation model to be trained can be the Cosmos-Transfer model.

[0115] In one implementation, the first and second training data are sequentially input into the video generation model to be trained. The model outputs corresponding predicted videos sequentially, and depth maps corresponding to each video frame in each predicted video are extracted to obtain a corresponding predicted depth map sequence. A loss function value can be determined based on the difference between the predicted depth map sequences corresponding to each of the first and second training data and the corresponding standard depth map sequences, using a preset loss function. The preset loss function may include, but is not limited to, cross-entropy error function or mean squared error function. The operation of determining the loss function value can be performed iteratively, and the parameters of the model to be trained can be adjusted iteratively to continuously reduce the loss function value until the loss function value converges, confirming that the preset model training conditions are met. The trained video generation model is then used as the video generation model. Stochastic gradient descent, adaptive gradient algorithm, adaptive moment estimation, root mean square, and other parameter optimizers can be used to adjust the parameters of the model to be trained. For example, the gradient of each parameter of the video generation model to be trained can be calculated using a parameter optimizer. The parameters can then be adjusted along the direction of the gradient, where the gradient represents the direction in which the loss function value decreases the most. The operation of determining the loss function value is iterated until the loss function value no longer decreases. The training of the video generation model to be trained is then completed, and the video generation model (MultiviewTransfer model) is obtained.

[0116] In this embodiment of the disclosure, a training dataset including labeled depth map sequences from multiple robotic arm perspectives is constructed. The first training data in this training dataset and the corresponding second training data have the same depth map sequence but different target object attributes and / or background information. Based on this training dataset, the video generation model to be trained is subjected to multiple rounds of iterative training. This allows the model to learn the consistency of depth information under multiple perspectives while mastering the video generation rules in different object attributes and different background scenes. This effectively improves the depth information fidelity and scene adaptation generalization of the model-generated video, ensuring that the finally trained model meets the needs of practical applications.

[0117] For example, Figure 7 This is an application diagram of the training method for visual language action models provided in this disclosure. For example... Figure 7 As shown, it specifically includes:

[0118] Multiple robotic arm operation videos (robotic arm operation videos in the first multimodal training data) from different perspectives were collected in both simulation and real environments, along with the ground truth values ​​corresponding to each video frame. Depth map sequences and text description information (text description information in the first multimodal training data) corresponding to each robotic arm operation video were obtained. The target object attribute information and / or background information in each text description information were adjusted. The adjusted text description information (text description information in the second multimodal training data) and the corresponding depth map sequences were input into a video generation model to obtain multiple synthetic videos (robotic arm operation videos in the second multimodal training data). Video filtering was performed based on the depth map fidelity of each synthetic video to obtain high-quality synthetic videos (robotic arm operation videos in the second multimodal training data used to construct the multi-model dataset). Multiple first multimodal training data sets were constructed based on each robotic arm operation video, the corresponding text description information, and the ground truth joint angle information corresponding to each video frame. Multiple second multimodal training data sets were also constructed based on each high-quality synthetic video, the corresponding adjusted text description information, and the ground truth joint angle information corresponding to each video frame.

[0119] The first multimodal training data and the second multimodal training data are input into the VLA model (first visual language action model) to be trained, and multiple rounds of iterative training are performed. The strategy of each round of iterative training is as follows: based on the VLA model to be trained, the prediction policy inference results corresponding to each video frame are output, the video frames associated with policy failure are identified, and the frame weights of the video frames associated with policy failure are increased in the next round of iterative training. Among them, a reward function can be preset, and the video frames associated with policy failure are identified by using the action deviation score, action smoothness score and joint angle compliance score corresponding to each video frame.

[0120] Figure 8This is a schematic diagram of a robotic arm operating device provided in an exemplary embodiment of this disclosure. Figure 8 As shown, the robotic arm manipulation device 400 can be applied to a robot. The robotic arm manipulation device 400 may include: a robotic arm 401, a robotic arm control device 402, and the aforementioned target visual language action model 403. The target visual language action model 403 is communicatively connected to both the robotic arm 401 and the robotic arm control device 402, and the robotic arm 401 is communicatively connected to the robotic arm control device 402.

[0121] The target visual language action model 403 is used to perform reasoning based on the input operation instructions and the video of the robotic arm's working environment, and outputs the strategy reasoning result, which includes the joint angle sequence of the robotic arm; the robotic arm control device 402 controls the robotic arm 401 to execute the strategy reasoning result.

[0122] The operation instructions can be natural language data input by the user. For example, operation instructions can include actions and tasks that the robotic arm needs to perform, such as picking up an apple or folding clothes. Video of the robotic arm's working environment can be acquired through a vision sensor on the robotic arm, such as a camera. The robotic arm control device can include, for example, a motion controller and a servo driver. The motion controller receives the strategy reasoning results from the target visual language action model and, through complex mathematical operations (such as inverse kinematics and trajectory planning interpolation algorithms), converts them into low-power command signals such as position, speed, or torque commands required by each joint motor. The motion controller can be, for example, a DSP or FPGA. The servo driver receives the low-power command signals from the motion controller and amplifies them into high-power current / voltage signals to drive the servo motors of each joint of the robotic arm to rotate precisely.

[0123] The robotic arm operating device of this disclosure corresponds to the embodiment of the visual language action model training method of this disclosure, and the relevant contents can be referred to each other, which will not be repeated here.

[0124] The beneficial technical effects of the exemplary embodiments of the robotic arm operating device of this disclosure can be found in the corresponding beneficial technical effects of the exemplary method section above, and will not be repeated here.

[0125] Figure 9 This is a schematic diagram of the structure of one embodiment of the training device for the visual language action model of this disclosure. Figure 9 As shown, the apparatus of this embodiment may include:

[0126] The first training data acquisition module 500 acquires a multimodal training dataset, which includes multiple first multimodal training data and multiple second multimodal training data. Each first multimodal training data and each second multimodal training data includes a robotic arm operation video, text description information, and ground truth joint angle information of the robotic arm corresponding to each video frame in the robotic arm operation video. The text description information is used to describe the video content of the robotic arm operation video.

[0127] The first model training module 510 is used to perform multiple rounds of iterative training on the first visual language action model to be trained based on the multimodal training dataset until a preset end-of-training condition is met, thereby obtaining a target visual language action model from the first visual language action model to be trained. In each round of iterative training, based on the prediction policy inference results corresponding to each video frame in each robotic arm operation video output by the first visual language action model to be trained, the policy failure associated video frames are determined, and the frame weight of the policy failure associated video frames in the next round of iterative training is increased. The prediction policy inference result corresponding to each video frame includes a sequence of predicted joint angles of the robotic arm. The sequence of predicted joint angles includes the predicted joint angle information corresponding to the future video frames predicted based on the video frame, and the predicted joint angle information corresponding to the video frame. The policy failure associated video frames represent the video frames that cause the first visual language action model to be trained to generate policy inference results that lead to task execution failure.

[0128] In some possible implementations of this disclosure, the text description information in the embodiments of this disclosure includes foreground information, background information, and interaction relationships of the robotic arm operation video. The foreground information includes target object attribute information, and the interaction relationships represent the process and goal of the strategy reasoning result performed by the robotic arm for the target object. The multiple robotic arm operation videos are acquired based on different robotic arm perspectives.

[0129] The first multimodal training data corresponds to at least one second multimodal training data, and the corresponding first multimodal training data and second multimodal training data have different target object attribute information and / or background information.

[0130] In some possible implementations of this disclosure, the first training data acquisition module 500 in this embodiment is specifically used to acquire multiple first multimodal training data, wherein the robotic arm operation videos of the multiple first multimodal training data are acquired based on different robotic arm perspectives; for the multiple first multimodal training data, the target object attribute information and / or background information in the text description information of the first multimodal training data are adjusted to obtain the text description information of the second multimodal training data; based on the robotic arm operation videos of the first multimodal training data and the text description information of the second multimodal training data, the robotic arm operation videos of the second multimodal training data are determined, wherein the robotic arm operation videos of the first multimodal training data and the robotic arm operation videos of the second multimodal training data have different target object attribute information and / or background information; and the multimodal training dataset is constructed based on at least one first multimodal training data and at least one second multimodal training data.

[0131] In some possible implementations of this disclosure, the step of determining the robotic arm operation video of the second multimodal training data based on the robotic arm operation video of the first multimodal training data and the text description information of the second multimodal training data in the embodiments of this disclosure is further used to: extract the depth map corresponding to each video frame in the robotic arm operation video of the first multimodal training data to obtain a depth map sequence; and obtain the robotic arm operation video of the second multimodal training data using a video generation model based on the depth map sequence and the text description information of the second multimodal training data.

[0132] In some possible implementations of this disclosure, the construction of the multimodal training dataset based on at least one first multimodal training data and at least one second multimodal training data in the embodiments of this disclosure is further configured to: for at least one second multimodal training data, extract the depth map of each video frame in the robotic arm operation video of the second multimodal training data to obtain a test depth map sequence; determine the depth map fidelity corresponding to the robotic arm operation video based on the test depth map sequence and the ground truth depth map sequence, wherein the ground truth depth map sequence is the depth map sequence of the robotic arm operation video that generated the second multimodal training data; determine the second multimodal training data for constructing the multimodal training dataset based on the depth map fidelity corresponding to the robotic arm operation video of each second multimodal training data; and construct the multimodal training dataset based on the second multimodal training data for constructing the multimodal training dataset and at least one first multimodal training data.

[0133] In some possible implementations of this disclosure, the construction of the multimodal training dataset based on at least one first multimodal training data and at least one second multimodal training data in the embodiments of this disclosure is further configured to: generate multiple initial datasets, wherein the ratio of the number of first multimodal training data to the number of second multimodal training data in each initial dataset is different; train the second visual language action model to be trained based on each initial dataset to obtain multiple test visual language action models; and determine the multimodal training dataset from the multiple initial datasets based on the policy performance of each test visual language action model, wherein the policy performance represents the performance of the robotic arm driven by the test visual language action model in completing the task.

[0134] In some possible implementations of this disclosure, the determination of policy failure-related video frames based on the prediction policy inference results corresponding to each video frame in the robotic arm operation video output by the first visual language action model to be trained is further used for: determining a motion deviation score for each video frame based on the predicted joint angle sequence corresponding to the video frame and the corresponding ground truth joint angle information; determining joint angular acceleration based on the predicted joint angle sequence corresponding to the video frame, and determining a motion smoothness score based on the joint angular acceleration; determining a joint angle compliance score based on the predicted joint angle sequence corresponding to the video frame and preset joint angle threshold information; and determining the policy failure-related video frames in each video frame using a preset reward function based on the motion deviation score, motion smoothness score, and joint angle compliance score corresponding to each video frame.

[0135] In some possible implementations of this disclosure, the training device for the visual language action model in the embodiments of this disclosure further includes:

[0136] The second training data acquisition module is used to acquire a video model training dataset. The video model training dataset includes multiple first training data sets and multiple second training data sets. Each first training data set and each second training data set includes a labeled depth map sequence and shared descriptive information corresponding to the depth map sequence. The shared descriptive information includes target object attribute information, background information, and interaction information in the labeled depth map sequence. The multiple labeled depth map sequences are obtained from labeled robotic arm operation videos collected from different robotic arm perspectives. Each first training data set corresponds to at least one second training data set. The corresponding first training data set and the second training data set have the same labeled depth map sequence and different target object attribute information and / or background information.

[0137] The second model training module is used to perform multiple rounds of iterative training on the video generation model to be trained based on the video model training dataset until the preset model training conditions are met, and to obtain the video generation model from the video generation model to be trained.

[0138] The training device for the visual language action model in this disclosure corresponds to the training method for the visual language action model in this disclosure, and the relevant contents can be referred to each other, which will not be repeated here.

[0139] The beneficial technical effects of the exemplary embodiments of the visual language action model training device of this disclosure can be found in the corresponding beneficial technical effects of the exemplary method section above, and will not be repeated here.

[0140] In addition, this disclosure also provides an electronic device, including:

[0141] Memory, used to store computer programs;

[0142] A processor is configured to execute a computer program stored in the memory, wherein when the computer program is executed, it implements the training method for the visual language action model described in any of the above embodiments of the present disclosure.

[0143] Figure 10 This is a schematic diagram illustrating the structure of an application embodiment of the electronic device disclosed herein. Below, reference is made to… Figure 10 This describes an electronic device according to embodiments of the present disclosure. The electronic device may be either or both of a first device and a second device, or a standalone device independent of them, which may communicate with the first device and the second device to receive acquired input signals from them.

[0144] like Figure 10 As shown, the electronic device includes one or more processors and memory.

[0145] A processor can be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and can control other components in an electronic device to perform desired functions.

[0146] The memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor may execute the program instructions to implement the training methods for visual language action models of the various embodiments of this disclosure described above, and / or other desired functions.

[0147] In one example, the electronic device may also include input devices and output devices, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0148] In addition, the input device may include, for example, a keyboard, a mouse, etc.

[0149] This output device can output various information to the outside, including determined distance information, direction information, etc. The output device may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0150] Of course, for the sake of simplicity, Figure 10 Only some of the components of the electronic device relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.

[0151] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the training methods for visual language action models according to various embodiments of this disclosure as described in the foregoing portions of this specification.

[0152] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0153] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the steps in the training method for the visual language action model according to various embodiments of this disclosure as described in the foregoing portion of this specification.

[0154] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0155] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as ROM, RAM, magnetic disk, or optical disk.

[0156] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0157] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0158] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0159] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0160] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0161] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0162] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A training method for a visual language action model, characterized in that, include: A multimodal training dataset is obtained, comprising multiple first multimodal training data sets and multiple second multimodal training data sets. Each first multimodal training data set and each second multimodal training data set includes a robotic arm operation video, text description information, and ground truth joint angle information of the robotic arm corresponding to each video frame in the robotic arm operation video. The text description information is used to describe the video content of the robotic arm operation video. This includes: acquiring multiple first multimodal training data sets, wherein the robotic arm operation videos of the multiple first multimodal training data sets are acquired based on different robotic arm perspectives; for the multiple first multimodal training data sets, the... The target object attribute information and / or background information in the text description information of the first multimodal training data are adjusted to obtain the text description information of the second multimodal training data; based on the robotic arm operation video of the first multimodal training data and the text description information of the second multimodal training data, the robotic arm operation video of the second multimodal training data is determined, wherein the robotic arm operation video of the first multimodal training data and the robotic arm operation video of the second multimodal training data have different target object attribute information and / or background information; the multimodal training dataset is constructed based on at least one first multimodal training data and at least one second multimodal training data; Based on the multimodal training dataset, the first visual language action model to be trained is subjected to multiple rounds of iterative training until a preset end-of-training condition is met. The target visual language action model is obtained from the first visual language action model to be trained. In each round of iterative training, based on the prediction policy inference results corresponding to each video frame in each robotic arm operation video output by the first visual language action model to be trained, the policy failure associated video frames are determined, and the frame weight of the policy failure associated video frames is increased in the next round of iterative training. The prediction policy inference result corresponding to each video frame includes the predicted joint angle sequence of the robotic arm. The predicted joint angle sequence includes the predicted joint angle information corresponding to the future video frame predicted based on the video frame, and the predicted joint angle information corresponding to the video frame. The policy failure associated video frame represents the video frame that causes the first visual language action model to be trained to generate a policy inference result that leads to task execution failure.

2. The method according to claim 1, characterized in that, The text description information includes foreground information, background information, and interaction relationships of the robotic arm operation video. The foreground information includes target object attribute information, and the interaction relationships represent the process and goal of the strategy reasoning result performed by the robotic arm for the target object. The multiple robotic arm operation videos are collected based on different robotic arm perspectives. The first multimodal training data corresponds to at least one second multimodal training data, and the corresponding first multimodal training data and second multimodal training data have different target object attribute information and / or background information.

3. The method according to claim 1, characterized in that, The process of determining the robotic arm operation video based on the robotic arm operation video of the first multimodal training data and the text description information of the second multimodal training data includes: Extract the depth map corresponding to each video frame in the robotic arm operation video of the first multimodal training data to obtain a depth map sequence; Based on the depth map sequence and the text description information of the second multimodal training data, a video generation model is used to obtain the robotic arm operation video of the second multimodal training data.

4. The method according to claim 1, characterized in that, The construction of the multimodal training dataset based on at least one first multimodal training data and at least one second multimodal training data includes: For at least one of the second multimodal training data, the depth map of each video frame in the robotic arm operation video of the second multimodal training data is extracted to obtain a test depth map sequence; Based on the test depth map sequence and the ground truth depth map sequence, the depth map fidelity corresponding to the robotic arm operation video is determined, wherein the ground truth depth map sequence is the depth map sequence of the robotic arm operation video used to generate the second multimodal training data; Based on the depth map reconstruction degree corresponding to the robotic arm operation video of each of the second multimodal training data, the second multimodal training data used to construct the multimodal training dataset is determined. The multimodal training dataset is constructed based on second multimodal training data used to construct the multimodal training dataset and at least one first multimodal training dataset.

5. The method according to claim 4, characterized in that, The construction of the multimodal training dataset based on at least one first multimodal training data and at least one second multimodal training data further includes: Multiple initial datasets are generated, and the ratio of the amount of the first multimodal training data to the amount of the second multimodal training data is different in each initial dataset; Based on the initial datasets, the second visual language action model to be trained is trained separately to obtain multiple test visual language action models. Based on the policy performance of each of the test visual language action models, the multimodal training dataset is determined from the multiple initial datasets, and the policy performance represents the performance of the robotic arm driven by the test visual language action model in completing the task.

6. The method according to any one of claims 1-5, characterized in that, The prediction policy inference results corresponding to each video frame in each of the robotic arm operation videos, based on the first visual language action model to be trained, determine the video frames associated with policy failure, including: For each video frame, a motion deviation score is determined based on the predicted joint angle sequence corresponding to the video frame and the ground truth joint angle information corresponding to the video frame. Based on the predicted joint angle sequence corresponding to the video frame, the joint angular acceleration is determined, and based on the joint angular acceleration, the motion smoothness score is determined. Based on the predicted joint angle sequence and preset joint angle threshold information corresponding to the video frame, a joint angle compliance score is determined. Based on the motion deviation score, motion smoothness score, and joint angle compliance score corresponding to each video frame, a preset reward function is used to determine the video frames associated with the failure of the strategy in each video frame.

7. The method according to claim 3, characterized in that, The video generation model is obtained in the following way: Obtain a video model training dataset, which includes multiple first training data and multiple second training data. Each first training data and each second training data includes a labeled depth map sequence and shared descriptive information corresponding to the depth map sequence. The shared descriptive information includes target object attribute information, background information and interaction information in the labeled depth map sequence. The multiple labeled depth map sequences are obtained from labeled robotic arm operation videos collected from different robotic arm perspectives. Each first training data corresponds to at least one second training data. The corresponding first training data and second training data have the same labeled depth map sequence, as well as different target object attribute information and / or background information. Based on the video model training dataset, the video generation model to be trained is iterated multiple times until the preset model training conditions are met, and the video generation model is obtained from the video generation model to be trained.

8. A robotic arm operating device, characterized in that, Applied to robots, the robotic arm operating device includes a robotic arm, a robotic arm control device, and the target visual language action model as described in any one of claims 1-7 above; The target visual language action model is used to perform reasoning based on the input operation instructions and the video of the robotic arm's working environment, and outputs the strategy reasoning result, which includes the joint angle sequence of the robotic arm. The robotic arm control device controls the robotic arm to execute the strategy reasoning result.

9. A training device for a visual language action model, characterized in that, include: The first training data acquisition module is used to acquire a multimodal training dataset. The multimodal training dataset includes multiple first multimodal training data sets and multiple second multimodal training data sets. Each first multimodal training data set and each second multimodal training data set includes a robotic arm operation video, text description information, and ground truth joint angle information of the robotic arm corresponding to each video frame in the robotic arm operation video. The text description information is used to describe the video content of the robotic arm operation video. The module includes: acquiring multiple first multimodal training data sets, wherein the robotic arm operation videos of the multiple first multimodal training data sets are acquired based on different robotic arm perspectives; and for the multiple first multimodal training data sets... The data is used to adjust the target object attribute information and / or background information in the text description information of the first multimodal training data to obtain the text description information of the second multimodal training data; based on the robotic arm operation video of the first multimodal training data and the text description information of the second multimodal training data, the robotic arm operation video of the second multimodal training data is determined, wherein the robotic arm operation video of the first multimodal training data and the robotic arm operation video of the second multimodal training data have different target object attribute information and / or background information; the multimodal training dataset is constructed based on at least one first multimodal training data and at least one second multimodal training data; The first model training module is used to perform multiple rounds of iterative training on the first visual language action model to be trained based on the multimodal training dataset until a preset end-of-training condition is met, thereby obtaining a target visual language action model from the first visual language action model to be trained. In each round of iterative training, based on the prediction policy inference results corresponding to each video frame in each robotic arm operation video output by the first visual language action model to be trained, policy failure-associated video frames are determined, and the frame weights of the policy failure-associated video frames are increased in the next round of iterative training. The prediction policy inference result corresponding to each video frame includes a sequence of predicted joint angles of the robotic arm. The sequence of predicted joint angles includes the predicted joint angle information corresponding to future video frames predicted based on the video frame, and the predicted joint angle information corresponding to the video frame. The policy failure-associated video frame represents the video frame that causes the first visual language action model to be trained to generate a policy inference result that leads to task execution failure.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing a computer program stored in the memory, wherein when the computer program is executed, it implements the method described in any one of claims 1-7.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-7.

12. A computer program product comprising computer program instructions, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-7.

Images