Double-mechanical-arm grabbing control method and device, electronic equipment and storage medium

Abstract

The invention discloses a double-mechanical-arm grabbing control method and device, electronic equipment and a storage medium, relates to the field of mechanical arm control, can be applied to financial science and technology and medical health business scenes, and comprises the steps that in response to a natural language task instruction, a double-mechanical-arm grabbing model is used for conducting dynamic cooperative control over double-arm actions on double mechanical arms. In the model, a visual feature vector of a target object is extracted by using a visual encoder, and a natural language task instruction is analyzed by using a language encoder to obtain a semantic feature vector; generating global planning information by using an LLM large model based on a fusion feature vector of a visual feature vector and a semantic feature vector; and an action decoder reinforcement learning model is used for dynamically generating action instructions of the double mechanical arms based on the global planning information, and closed-loop adjustment is conducted on the action instructions according to the real-time execution states of the double mechanical arms. The flexibility and the accuracy of grabbing control of the double mechanical arms can be improved.

Description

Technical Field

[0001] This application relates to the field of robotic arm control, and in particular to a dual-robotic arm grasping control method, device, electronic device, and storage medium. Background Technology

[0002] With the rapid development of industrial automation, home service robots, smart logistics, fintech, and healthcare, embodied intelligent fixed robotic arms, especially dual-arm systems, have become key technologies for improving operational efficiency and intelligence across various industries due to their ability to simulate the collaborative work of human arms in complex grasping, handling, and precision operations. In fintech scenarios, dual robotic arms can be used for cash sorting, document organization, and valuables transfer in smart vaults, ensuring the security and efficiency of financial asset processing through precise grasping and collaborative operation. In healthcare scenarios, they can assist in drug sorting, surgical instrument delivery, and even the assembly and debugging of precision medical equipment, providing technical support for the standardization and precision of medical services. The precise grasping and collaborative control capabilities of dual robotic arms are their core performance indicators, directly determining their application value in various fields. They are of great significance for reducing labor costs, minimizing human error, and pushing the limits of manual operation.

[0003] Existing dual-arm grasping control technologies are primarily based on traditional visual recognition and structured control algorithms. In fintech scenarios, they typically use preset image templates to match the features of cash or bills, controlling the arms to grasp and sort along a fixed path. In healthcare scenarios, they often target known-specification pharmaceutical packaging or surgical instruments, employing simple force feedback control to achieve the grasping action. These technical solutions rely on the fixed features and preset rules of specific items, achieving repetitive operations through closed-loop control.

[0004] However, in fintech and healthcare scenarios, existing technologies suffer from insufficient flexibility in dual-arm collaborative control. In multi-step sorting of financial documents (such as classification, stacking, and binding) or collaborative transfer of medical surgical instruments (such as precise delivery to a specific location in the surgical area), the lack of a dynamic coordination mechanism in motion planning often leads to conflicting or asynchronous movements of the two arms, resulting in low operational efficiency or even damage to the items. Summary of the Invention

[0005] In view of this, this application provides a dual-arm gripping control method, device, electronic device, and storage medium, which can improve the flexibility and accuracy of dual-arm gripping control.

[0006] According to a first aspect of this application, a dual-arm gripping control method is provided, comprising:

[0007] Receive natural language task instructions, which are used to instruct the execution of a target task of grasping a target object;

[0008] In response to the natural language task instruction, a dual-arm grasping model is used to perform dynamic collaborative control of the movements of the two robotic arms. The dual-arm grasping model includes: a visual encoder, a language encoder, an LLM large model, and an action decoder reinforcement learning model.

[0009] The dynamic collaborative control of the two robotic arms' movements in response to the natural language task instruction, using a dual-arm grasping model, includes:

[0010] In response to the natural language task instruction, the visual encoder is used to extract the visual feature vector of the target item, and the language encoder is used to parse the natural language task instruction to obtain the semantic feature vector.

[0011] The LLM large model is used to fuse the visual feature vector and the semantic feature vector, and global planning information containing the coordinated movements of both arms is generated based on the fused feature vector.

[0012] The motion decoder reinforcement learning model dynamically generates motion commands for the dual robotic arms based on the global planning information, and performs closed-loop adjustments to the motion commands according to the real-time execution status of the dual robotic arms. The motion commands are used to achieve dynamic collaborative control of the movements of the dual arms.

[0013] According to a second aspect of this application, a dual-arm gripping control device is provided, comprising:

[0014] A receiving module is used to receive natural language task instructions, which are used to instruct the execution of a target task of grasping a target item.

[0015] The control module is used to respond to the natural language task instructions and use a dual-arm grasping model to perform dynamic collaborative control of the dual-arm movements. The dual-arm grasping model includes: a visual encoder, a language encoder, an LLM large model, and an action decoder reinforcement learning model.

[0016] The control module is specifically used for:

[0017] In response to the natural language task instruction, the visual encoder is used to extract the visual feature vector of the target item, and the language encoder is used to parse the natural language task instruction to obtain the semantic feature vector.

[0018] The LLM large model is used to fuse the visual feature vector and the semantic feature vector, and global planning information containing the coordinated movements of both arms is generated based on the fused feature vector.

[0019] The motion decoder reinforcement learning model dynamically generates motion commands for the dual robotic arms based on the global planning information, and performs closed-loop adjustments to the motion commands according to the real-time execution status of the dual robotic arms. The motion commands are used to achieve dynamic collaborative control of the movements of the dual arms.

[0020] According to a third aspect of this application, a storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the above-described dual-arm grasping control method.

[0021] According to a fourth aspect of this application, an electronic device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor executes the program to implement the above-described dual-arm gripping control method.

[0022] By employing the aforementioned technical solutions, the dual-arm grasping control method, device, electronic equipment, and storage medium provided in this application, through the construction of a dual-arm grasping model integrating a visual encoder, a language encoder, an LLM large-scale model, and an action decoder reinforcement learning model, can effectively solve the problem of insufficient flexibility in dual-arm collaboration in various scenarios such as fintech and healthcare. Specifically, the visual encoder and language encoder accurately extract visual features and parse language commands respectively. The global plan generated after fusion by the LLM large-scale model can clearly define the action priority and collaboration sequence, enabling the dual robotic arms to accurately cooperate according to task requirements in complex tasks, such as avoiding action conflicts and synchronization deviations in multi-step sorting of financial documents or collaborative transfer of medical surgical instruments. The action decoder reinforcement learning model dynamically generates action commands based on the global plan and optimizes parameters through reinforcement learning in combination with real-time status to achieve closed-loop adjustment, ensuring that action commands are adapted to the real-time execution state, significantly improving operational efficiency, reducing the risk of damage to objects, and enhancing the adaptability of the robotic arm to complex scenarios, thereby expanding its application scope in fintech, healthcare, and other fields.

[0023] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0024] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0025] Figure 1 A flowchart illustrating a dual-robotic arm grasping control method provided in an embodiment of this application is shown.

[0026] Figure 2 A flowchart illustrating a dual-robotic arm grasping control method according to another embodiment of this application is shown;

[0027] Figure 3 This illustration shows a structural schematic diagram of a dual-arm gripping control device provided in an embodiment of this application;

[0028] Figure 4 A schematic diagram of another dual-arm gripping control device provided in an embodiment of this application is shown. Detailed Implementation

[0029] The present application will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of the present application can be combined with each other.

[0030] Existing dual-arm grasping control technologies are primarily based on traditional visual recognition and structured control algorithms. In fintech scenarios, they typically use preset image templates to match the features of cash or bills, controlling the arms to grasp and sort along a fixed path. In healthcare scenarios, they often target known-specification pharmaceutical packaging or surgical instruments, employing simple force feedback control to achieve the grasping action. These technical solutions rely on the fixed features and preset rules of specific items, achieving repetitive operations through closed-loop control.

[0031] However, in fintech and healthcare scenarios, existing technologies suffer from insufficient flexibility in dual-arm collaborative control. In multi-step sorting of financial documents (such as classification, stacking, and binding) or collaborative transfer of medical surgical instruments (such as precise delivery to a specific location in the surgical area), the lack of a dynamic coordination mechanism in motion planning often leads to conflicting or asynchronous movements of the two arms, resulting in low operational efficiency or even damage to the items.

[0032] To address the aforementioned technical problems, embodiments of the present invention provide a dual-robotic arm grasping control method, such as... Figure 1 As shown, the method includes:

[0033] Step 110: Receive natural language task instructions. Natural language task instructions are used to instruct the execution of the target task of grabbing the target item.

[0034] Natural language task instructions refer to task instructions given in the form of everyday human language (such as Chinese, English, etc.) to instruct the robotic arm to perform the task of grasping target objects. They include information such as task objectives, objects to be manipulated, and positional requirements, and can be parsed by the language encoder into machine-understandable semantic representations. Target objects are the objects specified in the natural language task instructions that the robotic arm needs to grasp. Their visual features (such as shape, position, and posture) are extracted by the visual encoder and used as a key basis for subsequent motion planning. Target tasks are the specific work that the robotic arm needs to complete, as defined by the natural language task instructions. They usually revolve around the grasping of target objects and related operations, and serve as the core guide for the robotic arm's motion planning and collaborative control.

[0035] The design for receiving natural language task instructions in this embodiment breaks through the limitations of traditional robotic arms that rely on fixed programs or specific instruction formats, significantly improving the convenience and flexibility of human-computer interaction. Operators do not need to possess professional programming knowledge; they can simply issue task instructions to the robotic arm using everyday language. This significantly reduces the operational threshold and improves task deployment efficiency, especially in scenarios such as industrial production and home services.

[0036] Step 120: In response to the natural language task instructions, use the dual-arm grasping model to perform dynamic collaborative control of the dual-arm movements.

[0037] The dual-arm grasping model is a model pre-trained for grasping tasks. It includes a visual encoder, a language encoder, a large language model (LLM), and a motion decoder reinforcement learning model. The visual encoder extracts visual features (such as shape and color) of the environment and target objects. The language encoder is a model module for parsing natural language task instructions, processing text information to transform semantic logic (such as task objectives, objects to be manipulated, and positional requirements) into machine-processable semantic feature vectors. The LLM integrates visual and linguistic features, inferring global motion planning and semantic understanding for the task, providing global planning for motion generation. The motion decoder reinforcement learning model generates motion instructions based on the global planning, optimizes motion parameters through reinforcement learning, and adjusts instructions in real-time to achieve collaborative control of both arms.

[0038] In this embodiment of the disclosure, through multi-module collaboration and real-time adjustment, the dual robotic arms can dynamically optimize their movements based on natural language commands and environmental changes, reducing conflicts and deviations in complex tasks and improving operational efficiency and quality. Simultaneously, the pre-trained model combined with reinforcement learning ensures the accuracy and adaptability of motion generation, expanding the application scope of robotic arms in industrial, domestic service, and other scenarios.

[0039] In specific application scenarios, the training of a dual-arm grasping model can include the following steps: selecting multiple sets of different but related grasping tasks, each with corresponding second training data; based on the second training data of each grasping task, using a meta-learning mechanism to train the dual-arm grasping model, during which a set of initial basic parameters is set for the dual-arm grasping model as the starting point for training all grasping tasks; training the dual-arm grasping model on each grasping task using its corresponding second training data, generating temporary parameters adapted to the individual grasping task using gradient descent; calculating the meta-loss function for all grasping tasks, optimizing the meta-loss function based on the temporary parameters of each grasping task to adjust the basic parameters until the meta-loss function converges, obtaining the initial model parameters of the dual-arm grasping model; and using transfer learning technology, fine-tuning the initial model parameters using the third training data of the target task, obtaining a dual-arm grasping model adapted to the target task by minimizing the target loss function.

[0040] Training methods for dual-arm grasping models can significantly improve their generalization ability and adaptation efficiency. Meta-learning allows dual-arm grasping models to master general collaborative strategies across scenarios through multi-task training, reducing reliance on large amounts of labeled data; transfer learning enables dual-arm grasping models to quickly adapt to new tasks, shortening the cycle from deployment to practical use.

[0041] In summary, the dual-arm grasping control method provided by this invention, by constructing a dual-arm grasping model that integrates a visual encoder, a language encoder, an LLM large-scale model, and an action decoder reinforcement learning model, can effectively solve the problem of insufficient flexibility in dual-arm collaboration in various scenarios such as fintech and healthcare. Specifically, the visual encoder and language encoder accurately extract visual features and parse language commands respectively. The global plan generated after fusion by the LLM large-scale model can clearly define the action priority and collaboration sequence, enabling the dual robotic arms to cooperate precisely according to task requirements in complex tasks, such as avoiding action conflicts and synchronization deviations in multi-step sorting of financial documents or collaborative transfer of medical surgical instruments. The action decoder reinforcement learning model dynamically generates action commands based on the global plan and optimizes parameters through reinforcement learning in combination with real-time status to achieve closed-loop adjustment, ensuring that the action commands are adapted to the real-time execution state and guaranteeing the flexibility and accuracy of dual-arm grasping control. In addition, during the training of the dual robotic arm grasping model, meta-learning optimizes the basic parameters through multi-task training, and transfer learning uses target task data for fine-tuning, allowing the model to quickly adapt to new tasks. This can improve the operational efficiency and adaptability in scenarios such as financial bill sorting and medical device delivery, reduce damage to items, and expand the scope of application.

[0042] Furthermore, as a refinement and extension of the specific implementation of the above embodiments, and to fully illustrate the implementation of this embodiment, this embodiment also provides another dual-robotic arm grasping control method, such as... Figure 2 As shown, the method includes:

[0043] Step 210: Receive natural language task instructions. Natural language task instructions are used to instruct the execution of the target task of grabbing the target item.

[0044] Step 220: In response to the natural language task instruction, use a visual encoder to extract the visual feature vector of the target object, and use a language encoder to parse the natural language task instruction to obtain the semantic feature vector.

[0045] The visual encoder comprises a local encoder and a global encoder. The local encoder, a submodule of the visual encoder, extracts features from local images (such as close-up shots) of a target object using a convolutional neural network (CNN), focusing on the object's detailed features (such as shape, color, and texture) to provide a basis for accurate identification of object attributes. The global encoder, another submodule of the visual encoder, extracts features from global scene images (such as the overall environment) containing the target object using a Transformer architecture, focusing on capturing the spatial relationship between the target object and its surrounding environment (such as relative position, placement posture, and association with other objects). The visual feature vector is a numerical vector output by the visual encoder, which integrates local detailed features and global spatial relationship features, and is a structured form for the machine to understand the visual attributes of the target object and its relationship with the environment.

[0046] In this embodiment of the disclosure, perceptual information can be processed by a visual encoder and a language encoder respectively: In the visual encoder, the local encoder extracts detailed features (such as surface texture and corner shape) from the local image (such as close-up area) of the target object based on a convolutional neural network, and the global encoder extracts spatial relationship features (such as relative position with other objects and location) from the global scene image (such as overall placement environment) containing the target object based on a Transformer architecture. The two are fused to form a visual feature vector representing the visual attributes of the target object; at the same time, the language encoder parses the semantic logic of the natural language task instruction (such as the object and location information in "grab the blue medicine bottle on the left tray") and converts it into a machine-understandable semantic feature vector.

[0047] By employing a dual-module architecture of local and global visual encoders, the limitations of traditional visual recognition, which focuses solely on local features, can be overcome. This allows for the precise capture of details of target objects (such as the serial number of financial instruments or the interface shape of medical devices) and an understanding of their spatial relationship with the surrounding environment (such as the layering of instruments in a stack or the placement of instruments on an operating table), avoiding recognition biases caused by a narrow perspective. The language encoder ensures accurate parsing of the semantics of natural language commands. The feature vectors generated by the combination of these two components lay a high-quality foundation for subsequent cross-modal fusion of large-scale LLM models, significantly improving the dual-arm grasping control system's ability to understand complex scenarios (such as mixed objects or commands with implicit requirements), and reducing collaborative errors caused by missing information or misjudgments.

[0048] For embodiments of this disclosure, step 220, which involves extracting the visual feature vector of the target item using a visual encoder, may include the following steps:

[0049] Step 220-1: Use a local encoder to extract features from the local image of the target object through a convolutional neural network to obtain local detail features.

[0050] As a core submodule of the visual encoder, the local encoder can extract features from local images of a target object (such as close-ups of the object or details of key parts) using a convolutional neural network. Specifically, the convolutional neural network can utilize the local receptive field characteristics of the convolutional layers to capture discriminative local information in the target object image layer by layer, such as the surface texture of the target object (e.g., metallic luster, fabric texture), geometric details (e.g., corner curvature, hole location), and color distribution (e.g., color block boundaries, gradient levels), and finally output local detail features that can accurately represent these subtle features.

[0051] The precise extraction of local detailed features can effectively distinguish items that are similar in appearance but different in details (such as screws of different specifications or reagent bottles with different labels), avoiding misidentification due to blurred features. At the same time, these detailed features are directly related to the gripping adaptability of the items (such as smooth surfaces requiring adjustment of gripping force, and grooved parts being suitable for gripping), which can provide key basis for the subsequent motion decoder to generate accurate gripping instructions, significantly reducing the risk of gripping failure or item damage due to misjudgment of item details, and improving the accuracy and reliability of dual robotic arm operation.

[0052] Step 220-2: Using a global encoder based on the Transformer architecture, feature extraction is performed on the global scene image containing the target object to obtain the global spatial relationship features between the target object and its surrounding environment.

[0053] As a key submodule of the visual encoder, the global encoder can extract features from a global scene image containing the target object (such as a complete picture covering the target object and its surrounding environment) based on the Transformer architecture. Specifically, the Transformer's self-attention mechanism can capture long-distance dependencies between different regions in the image. By calculating the relative position, spatial layout, and interaction relationships (such as which shelf the target is on, the distance between it and adjacent items, and whether it is occluded) of the target object and other objects in the surrounding environment (such as obstacles, operating platforms, and other items to be processed), it ultimately generates global spatial relationship features that represent these global associations.

[0054] Acquiring global spatial relationship features allows dual robotic arms to fully consider environmental constraints when performing tasks (such as avoiding knocking over stacks of documents when grasping tickets in financial scenarios, or avoiding the operating range of surgical personnel when passing instruments in medical scenarios), thereby optimizing motion paths and reducing collision risks. At the same time, this understanding of the global environment can help the LLM large model to more accurately plan the timing of dual-arm collaboration (such as adjusting the right arm to avoid obstacles before allowing the left arm to grasp), significantly improving the coordination and safety of dual-arm movements in complex scenarios, and avoiding operational errors caused by misjudgment of spatial relationships.

[0055] Step 220-3: Perform feature fusion between local detail features and global spatial relationship features to obtain the visual feature vector of the target object.

[0056] When fusing local detail features and global spatial relationship features, the fusion method can be achieved through mechanisms such as splicing, attention weighting, or cross-modal interaction, so that the two features complement and enhance each other, ultimately forming a visual feature vector that contains both the subtle attributes of the target object itself and its relationship with the environmental space.

[0057] By fusing local detail features and global spatial relationship features, visual feature vectors can simultaneously possess detail recognition and environmental adaptability. This enables the subsequent LLM large model to determine the optimal grasping method based on object details (such as adjusting the force for smooth surfaces) and plan safe action paths based on global spatial relationships (such as avoiding surrounding objects) when generating dual-arm collaborative planning. This can significantly improve the accuracy and safety of dual robotic arms operating in complex scenarios (such as mixed objects and narrow spaces) and reduce action errors caused by incomplete information.

[0058] Step 230: Use the LLM large model to fuse the visual feature vector and semantic feature vector, and generate global planning information containing the coordinated movements of both arms based on the fused feature vector.

[0059] Among them, the fused feature vector is a vector generated by the LLM large model through the fusion of visual feature vector and semantic feature vector through cross-modal attention mechanism. It contains the relationship between visual information and semantic instructions and is the core basis for global planning of model inference. The global planning information is a high-level strategy generated by the LLM large model to guide the collaborative action of the two robotic arms. It contains elements such as the division of labor logic between the two arms, action priority, and collaboration timing, and is used to ensure that the action generation conforms to the global logic of the task.

[0060] In the dual-arm grasping control process, the LLM large model, after receiving the visual feature vectors output by the visual encoder and the semantic feature vectors output by the language encoder, can fuse the two through a cross-modal attention mechanism. This allows the model to learn the association between visual information and semantic instructions (e.g., the visual features of the "red box" in "grab the red box" correspond to the semantic meaning of the "grab" action), generating a fused feature vector. Based on this fused feature vector, the LLM large model further infers the core requirements of the target task (e.g., the multi-step logic of sorting, stacking, and binding documents in "organizing tickets"), and plans the division of labor between the two robotic arms (e.g., the left arm is responsible for grasping and passing, and the right arm is responsible for stacking and fixing), action priorities (e.g., sorting is completed before stacking), and collaboration timing (e.g., the right arm must catch the left arm within 0.5 seconds after it passes the document). Finally, it forms global planning information containing the above elements to provide high-level guidance for subsequent action generation.

[0061] By leveraging the multimodal fusion and logical reasoning capabilities of the LLM large model, the problems of fragmented visual information and task instructions, and the lack of global collaborative logic in motion planning, in traditional technologies can be solved. Integrating visual and semantic features ensures that planning is based on complete information about "what the goal is, what the environment is like, and what needs to be done," avoiding planning deviations caused by incomplete information. The generated global plan clearly defines the division of labor and timing between the two arms, enabling the robotic arms to coordinate logically and orderly in complex tasks (such as multi-step processing of financial documents and precise delivery of medical devices), reducing motion conflicts (such as both arms extending into the same space simultaneously) and synchronization deviations (such as motion connection delays), significantly improving operational efficiency and smoothness.

[0062] Accordingly, for the embodiments of this disclosure, step 230 may include the following steps:

[0063] Step 230-1: The visual feature vector and the semantic feature vector are fused through a cross-modal attention mechanism to obtain a fused feature vector.

[0064] When fusing visual and semantic feature vectors through a cross-modal attention mechanism, a bidirectional attention interaction is first constructed. The semantic feature vector (containing key semantics of the task instruction, such as "red," "left," and "grab" in "grab the red bottle on the left") calculates the association weights with each element in the visual feature vector (containing local details of the target object, such as color and shape, and global spatial location, such as left / right orientation), focusing on features in the visual representation that match the semantic keywords. Simultaneously, the visual feature vector also pays attention to the parts of the semantic features related to its own attributes (e.g., the visual feature of "red surface" will be strongly associated with the semantic description of "red bottle"). Through this bidirectional attention weight allocation and weighted summation, the features of the two modalities are deeply bound together, ultimately integrating into a fused feature vector that includes both the visual attributes of the target object (what it is and where it is) and the semantic logic of the task (what to do and how to do it), achieving a precise correspondence between visual information and linguistic instructions.

[0065] Step 230-2: Determine the task objective of the target task based on the fused feature vector, and infer the division of labor logic of the two robotic arms based on the task objective, and determine the action priority and cooperation sequence of the left and right arms.

[0066] In this embodiment of the disclosure, the core task objective of the target task can first be extracted from the fused feature vector, that is, the "specific result to be achieved" can be clearly defined. For example, the task objective can be determined as "grab the surgical forceps and pass them to the left side of the sterile tray" from "hand the surgical forceps to the doctor and place them on the left side of the sterile tray". Then, based on the task objective, the division of labor logic of the two robotic arms is generated through the logical reasoning capability of the LLM model. That is, according to the task steps (such as "grab-pass-place") and the operating characteristics of the two arms (such as the left arm being more flexible and suitable for fine grasping, and the right arm having a larger load and suitable for stable placement), the specific responsibilities of the left and right arms are assigned. Then, the action priority (such as "grab first and then pass" or "avoid obstacles first and then perform grasping") and the cooperation sequence (such as the left arm holding stable for 0.3 seconds after grasping, and the right arm moving synchronously to the receiving position) are determined to ensure that the actions of the two arms are logically and orderly coordinated.

[0067] By accurately extracting task objectives from fused feature vectors, ambiguity in task understanding can be avoided; and the division of labor logic, action priority, and collaborative timing planning based on task objectives can solve the problems of inefficient collaboration or conflict caused by unclear responsibilities and chaotic action sequence of dual robotic arms.

[0068] Step 230-3: Generate global planning information including bi-arm coordinated movements based on action priority, collaboration timing, and spatial collaboration range.

[0069] In this embodiment of the disclosure, the process of generating global planning information can comprehensively consider three major factors: action priority (e.g., the order of "avoiding obstacles before grasping"), cooperation timing (e.g., the time required for the right arm to move to the receiving position within 0.5 seconds after the left arm grasps), and spatial cooperation range (the spatial boundaries of the arm movements, such as the operating radius of the left arm and the safe distance between the right arm and surrounding objects). Specifically, action priority can ensure that key steps of the task are executed first to avoid invalid operations; cooperation timing can standardize the timing connection of the arm movements to prevent action delays or overlaps; and the spatial cooperation range can be combined to delineate the no-go zones and safe zones for the arm movements to avoid spatial conflicts (e.g., both arms extending into the same area simultaneously).

[0070] By integrating action priority, collaboration timing, and spatial collaboration range, the problems of time asynchrony, spatial conflicts, and lack of logical core in steps during dual-arm collaboration can be solved. The global plan generated by the combination of these three factors enables the dual-arms to maintain logical continuity of actions and achieve spatiotemporal collaboration in complex scenarios, which can significantly improve the stability and accuracy of task completion.

[0071] Step 240: Using the motion decoder reinforcement learning model, the motion commands of the two robotic arms are dynamically generated based on global planning information. The motion commands are then adjusted in a closed loop according to the real-time execution status of the two robotic arms. The motion commands are used to achieve dynamic coordinated control of the movements of the two arms.

[0072] The motion decoder reinforcement learning model, serving as the execution layer for dual-arm control, takes the global planning information generated by the LLM large model as input and dynamically generates specific motion commands for the dual robotic arms through reinforcement learning algorithms. These commands can include parameters such as joint angles, movement speed, and gripping force. For example, based on the plan of "left arm prioritizing gripping and right arm assisting in fixation," the extension trajectory of the left arm and the waiting posture command for the right arm are generated. Simultaneously, force and position sensors can collect the execution status of the two arms in real time (such as actual joint angles, spatial positions, and contact forces), comparing this with the preset parameters of the motion commands. If there is a deviation (such as excessive gripping force causing deformation of the object), the motion command for the next time node is adjusted immediately through the reinforcement learning feedback mechanism (such as reducing force or correcting the trajectory), forming a closed loop of "generation-execution-feedback-adjustment," ultimately achieving dynamic collaborative control of the two arm movements in time and space.

[0073] By leveraging the dynamic generation and closed-loop adjustment capabilities of reinforcement learning, the pain points of traditional pre-programmed control, such as "fixed instructions and difficulty in coping with environmental changes," can be addressed. The instructions generated by the action decoder reinforcement learning model based on global planning not only conform to high-level strategies but also flexibly adapt to unexpected situations through real-time feedback (such as adjusting the gripping angle immediately when an object slips). Closed-loop adjustment ensures that deviations in the movements of both arms are corrected in a timely manner, avoiding collaborative failures caused by accumulated errors (such as misalignment of the arms during coordination).

[0074] In the embodiments of this disclosure, when dynamically generating motion commands for the dual robotic arms based on global planning information using the motion decoder reinforcement learning model, step 240 may include the following steps:

[0075] Step 240-1: Input the visual feature vector into the pre-trained grasping point prediction model, and use the convolutional neural network in the grasping point prediction model to regress and calculate the coordinates of the target grasping point.

[0076] Among them, the gripping point prediction model is a pre-trained model used to predict the optimal gripping point coordinates of the target object based on the visual feature vector. Its training data includes the visual features of various objects and the corresponding gripping point position labels. The target gripping point coordinates are the spatial coordinates output by the gripping point prediction model, representing the most suitable gripping position on the target object (such as a structurally stable area that is easy to exert force), which is a precise spatial guide for the dual robotic arms to perform gripping actions.

[0077] During the dynamic generation of dual robotic arm motion commands, the system inputs the visual feature vectors output by the visual encoder into a pre-trained grasping point prediction model. This grasping point prediction model, based on a convolutional neural network architecture, outputs the coordinates of the target grasping point through regression calculation (i.e., predicting continuous values). Specifically, the convolutional neural network architecture can utilize its deep extraction capability of visual features to learn the mapping relationship between visual feature vectors and the optimal grasping point of the object (such as the center of gravity position, structurally stable region), ultimately outputting accurate three-dimensional or two-dimensional coordinate values, providing precise spatial positioning basis for the grasping actions of the dual robotic arms.

[0078] In specific application scenarios, when pre-training the grasping point prediction model, the implementation steps may include the following: collecting sample visual feature vectors of various sample items and their corresponding grasping point coordinates as first training data, wherein the grasping point coordinates are determined based on the geometric features of the items and grasping stability analysis; using the first training data to iteratively train the convolutional neural network in the grasping point prediction model, so that the grasping point prediction model learns the mapping relationship between the visual features of the items and the grasping point coordinates; when the prediction error of the grasping point prediction model on the validation set is less than a first preset threshold, or the number of iterative training times reaches a preset upper limit, it is determined that the training of the grasping point prediction model is complete.

[0079] The first training data is a dataset used to train the grasping point prediction model, containing sample visual feature vectors of various sample items and corresponding grasping point coordinates, which is the basis for the model to learn the mapping relationship. The sample visual feature vectors are visual feature vectors extracted from the sample item images, containing local detail features and global spatial relationship features of the sample items. The grasping point coordinates are the spatial coordinates of the optimal grasping position on the sample item. The item geometric features are the spatial attributes of the item, such as shape, size, structural layout, and center of gravity. The grasping stability analysis is an analysis process to evaluate whether the grasping point can keep the item stable during the grasping process (such as not slipping, not deforming, and being uniformly stressed), which is used to select the optimal grasping position. The first preset threshold is the error critical value for determining whether the model's prediction accuracy meets the standard. When the prediction error on the validation set is less than this value, it is determined that the grasping point prediction model meets the practical requirements, that is, the training process is completed.

[0080] The training data from diverse sample items enables the gripping point prediction model to adapt to different types of objects (such as regular geometric shapes and irregular flexible objects), avoiding overfitting caused by single data sets. Gripping point coordinates determined based on geometric features and stability analysis provide high-quality labeled data for the model, ensuring that the learned mapping closely matches actual gripping needs. Iterative training combined with validation set evaluation effectively controls model prediction errors, avoiding the problem of good training results but large deviations in practical applications. Ultimately, the trained gripping point prediction model can accurately output the target gripping point coordinates of the object, providing a reliable gripping and positioning basis for the dual robotic arms. This significantly reduces the risk of slippage and object damage due to improper gripping point selection, especially when handling irregularly shaped or fragile objects, greatly improving gripping success rate and operational safety.

[0081] Step 240-2a: Using the action decoder reinforcement learning model, based on global planning information, target grasping point coordinates, and the first historical action of the left arm in the previous moment, generate the first action command of the right arm in the current moment.

[0082] The first historical action is the action information of the left arm at the previous moment, including the motion trajectory, joint angle, posture, etc., reflecting the historical state of the left arm, and is used to ensure the coordination of the current action of the right arm with it in time and space; the first action instruction is the specific control parameters (such as motion trajectory, joint angle, force magnitude, etc.) generated by the action decoder reinforcement learning model for the right arm at the current moment, which directly drives the right arm to perform the operation, so as to achieve dynamic coordination with the left arm.

[0083] In this embodiment of the disclosure, when generating the first action command of the right arm at the current moment, the action decoder reinforcement learning model integrates three types of key information: first, global planning information (including high-level strategies such as the division of labor logic between the two arms, action priority, and cooperation sequence), providing an overall framework for action generation; second, the coordinates of the target grasping point (the optimal grasping position output by the grasping point prediction model), clarifying the spatial positioning benchmark of the action; and third, the first historical action of the left arm at the previous moment (such as the movement trajectory and grasping posture of the left arm), ensuring that the action of the right arm is temporally continuous and spatially coordinated with the historical state of the left arm. The action decoder reinforcement learning model integrates this information through reinforcement learning algorithms, learns the mapping relationship between "global strategy-spatial positioning-historical action" and "current action", and finally generates specific action commands for the right arm (such as movement trajectory, joint angle, force magnitude, etc.), ensuring that the action of the right arm not only conforms to the global planning, but also forms a dynamic coordination with the left arm (such as after the left arm grasps, the right arm moves to the receiving position in a timely manner according to the continuation of the historical action).

[0084] By introducing the first historical movement of the left arm, the movement of the right arm can "take over" the previous state of the left arm, preventing time gaps (such as the left arm has moved to point A, but the right arm is still planning the movement according to the initial position) or spatial collisions (such as the right arm rashly entering the same area before the left arm has completely withdrawn).

[0085] Step 240-2b: Using the action decoder reinforcement learning model, based on global planning information, target grasping point coordinates, and the second historical action of the right arm in the previous moment, generate the second action command of the left arm in the current moment.

[0086] The second historical action is the action information of the right arm at the previous moment, including the movement trajectory, joint angle, and holding posture, which reflects the historical state of the right arm and is used to ensure the coordination of the current action of the left arm with it in time and space. The second action instruction is the specific control parameters (such as extension angle, movement speed, grasping preparation posture, etc.) generated by the action decoder reinforcement learning model for the left arm at the current moment, which directly drives the left arm to perform the operation and realize the dynamic coordination with the right arm.

[0087] In this embodiment of the disclosure, similar to steps 240-2a of the embodiment, when the action decoder reinforcement learning model generates the second action command for the left arm at the current moment, it also integrates three types of key information: global planning information (including high-level strategies such as division of labor between the two arms, action priority, and cooperation sequence) to provide an overall framework for the action; the target grasping point coordinates (the optimal grasping position determined by the grasping point prediction model) to clarify the spatial reference of the action; and the second historical action of the right arm at the previous moment (such as the movement trajectory and holding posture of the right arm) to ensure that the action of the left arm is connected to the historical state of the right arm in time and adapted in space. The action decoder reinforcement learning model learns the correlation between this information and the current action of the left arm through reinforcement learning algorithms, and generates specific second action commands (such as the extension angle, movement speed, and grasping preparation posture of the left arm), so that the action of the left arm not only conforms to the global task logic, but also forms dynamic coordination with the preceding action of the right arm (such as after the right arm completes positioning, the left arm adjusts its own trajectory according to its historical position to avoid collision).

[0088] Introducing the second historical movement of the right arm allows the left arm to "respond" to the previous state of the right arm, preventing time delays in coordination (such as the left arm following up to assist when the right arm has moved to the target area) or spatial conflicts (such as the left arm mistakenly entering the same operating range when the right arm has not withdrawn).

[0089] In this embodiment of the present disclosure, when the motion commands are adjusted in a closed loop according to the real-time execution status of the two robotic arms to achieve dynamic collaborative control of the two arm movements, the preset motion parameters corresponding to the motion commands can be clearly defined first. These parameters can include key control parameters such as the joint angles, movement speeds, and gripping forces of the two robotic arms, serving as the benchmark for motion execution. Subsequently, the actual execution data of the two arms is collected in real time through force sensors (detecting contact force) and position sensors (capturing joint angles and spatial positions) to form feedback. Then, the deviation value between the real-time execution data and the preset motion parameters (such as the difference between the actual joint angle and the preset angle, and the difference between the contact force and the standard force) is calculated. If the deviation value exceeds a second preset threshold (such as a joint angle deviation greater than 5° or a force deviation greater than 2N), a reinforcement learning algorithm is activated to adjust the motion commands at the next time node based on the deviation analysis (such as correcting the joint movement trajectory and increasing or decreasing the gripping force). The above process continues to cycle until the gripping task is completed or the deviation value falls back to the allowable range, ensuring that the movements of the two arms are always dynamically adapted to the actual execution status.

[0090] Accordingly, step 240 may include the following steps: determining the preset motion parameters corresponding to the motion command, the preset motion parameters including the joint angle parameters, motion speed parameters, and grasping force parameters of the dual robotic arms; collecting real-time execution data of the dual robotic arms through force sensors and position sensors, the real-time execution data including joint angles, spatial positions, and contact forces; calculating the deviation value between the real-time execution data and the preset motion parameters; if the deviation value exceeds a second preset threshold, adjusting the motion command at the next time node through a reinforcement learning algorithm; repeating the above steps until the grasping task is completed or the deviation value is within the allowable range.

[0091] Among them, the preset motion parameters are the dual-arm execution benchmark parameters set in the motion command, including joint angle parameters (target rotation angle of each joint), motion speed parameters (movement rate of the end effector), gripping force parameters (force range of the gripping object), etc., which are the reference standards for measuring the motion execution accuracy; force sensors are sensors used to detect force data (such as gripping force, collision force) when the dual-arm comes into contact with the target object or environment; position sensors are sensors that collect the joint angles of the dual-arm (such as the rotation angle of the shoulder joint and elbow joint) and the spatial coordinates of the end effector; real-time execution data are the actual motion data of the dual-arm obtained through sensors, including joint angles, spatial position (reflecting posture and position), and contact force (reflecting the interaction state), etc.; deviation value is the difference between real-time execution data and preset motion parameters, used to quantify the degree of deviation of motion execution; the second preset threshold is the critical value for determining the deviation and triggering adjustment, which is determined by the task accuracy requirements (such as the lower threshold in precision assembly scenarios and the higher threshold in handling scenarios), and if this value is exceeded, the command correction is triggered.

[0092] This closed-loop adjustment mechanism effectively solves the problem of motion deviation caused by various interferences (such as mechanical wear and environmental changes) during the execution of dual robotic arms, significantly improving the accuracy and stability of dual-arm coordination. Preset motion parameters provide clear execution targets for the robotic arms, avoiding blind movements; real-time data acquisition from force and position sensors captures the actual state of the robotic arms in a timely manner, providing a reliable basis for deviation analysis; the calculation of deviation values ​​quantifies the deviation of the robotic arms' movements, facilitating the determination of whether adjustments are needed; when the deviation value exceeds a threshold, the reinforcement learning algorithm can respond quickly, making the motion commands at the next time point more aligned with actual needs, avoiding the accumulation of deviations. This continuous cyclical adjustment allows the dual robotic arms to flexibly respond to various emergencies in complex grasping tasks, ensuring that the movements of both arms remain coordinated, improving the success rate of grasping tasks, and reducing damage to objects or task failures caused by uncoordinated movements.

[0093] In summary, the technical solution of this application employs a visual encoder that includes a local encoder and a global encoder to extract and fuse local detail features and global spatial relationship features of the target object, thereby improving visual perception accuracy and providing accurate guidance for grasping. The LLM large model fuses visual and semantic features through a cross-modal attention mechanism to generate a global plan containing action priority, collaboration timing, and spatial range, enabling precise coordination between the two arms. The action decoder generates real-time instructions based on the global plan, grasping point coordinates, and historical actions, and combines real-time data from force and position sensors for closed-loop adjustment. Through reinforcement learning, it optimizes action parameters to avoid conflicts and deviations, further improving the efficiency and stability of dual-arm grasping operations. The grasping point prediction model, after training, can accurately locate the grasping point, further ensuring grasping accuracy. These features work together to enhance the collaborative flexibility, grasping accuracy, and environmental adaptability of the dual-arm robotic system in complex scenarios, reduce damage to objects, and expand the application scope.

[0094] Furthermore, as Figure 1 and Figure 2 The specific implementation of the method shown in this embodiment provides a dual-arm gripping control device, such as... Figure 3 As shown, the device includes: a receiving module 31 and a control module 32.

[0095] The receiving module 31 can be used to receive natural language task instructions. The natural language task instructions for the dual robotic arm grasping control are used to instruct the execution of the target task of grasping the target object.

[0096] Control module 32 can be used to respond to natural language task commands for dual-arm grasping control, and use the dual-arm grasping model to perform dynamic collaborative control of the dual-arm movements. The dual-arm grasping control dual-arm grasping model includes: a visual encoder, a language encoder, an LLM large model and an action decoder reinforcement learning model.

[0097] The dual-arm gripping control module 32 can be specifically used for:

[0098] The dual-arm grasping control responds to the dual-arm grasping control natural language task instructions. The visual feature vector of the target object is extracted by the dual-arm grasping control visual encoder, and the semantic feature vector is obtained by parsing the dual-arm grasping control natural language task instructions using the dual-arm grasping control language encoder.

[0099] The visual feature vector and semantic feature vector of dual-arm grasping control are fused using the dual-arm grasping control LLM large model, and global planning information containing the collaborative movements of the two arms is generated based on the fused feature vector.

[0100] By utilizing a reinforcement learning model of a dual-arm grasping control motion decoder, motion commands for the dual arms are dynamically generated based on the global planning information of the dual-arm grasping control. The motion commands are then adjusted in a closed loop according to the real-time execution status of the dual arms. These motion commands are used to achieve dynamic coordinated control of the dual-arm movements.

[0101] In some embodiments of this application, the visual encoder includes a local encoder and a global encoder. When the dual-arm grasping control uses the visual encoder to extract the visual feature vector of the target object, the control module 32 can specifically be used to: use the local encoder to extract features from the local image of the target object using a convolutional neural network to obtain local detail features; use the global encoder based on the Transformer architecture to extract features from the global scene image containing the target object to obtain global spatial relationship features between the target object and its surrounding environment; and fuse the local detail features and global spatial relationship features to obtain the visual feature vector of the target object.

[0102] In some embodiments of this application, when the dual-arm grasping control visual feature vector and dual-arm grasping control semantic feature vector are fused using the dual-arm grasping control LLM large model, and global planning information containing dual-arm collaborative actions is generated based on the fused feature vector, the control module 32 can specifically be used to: fuse the dual-arm grasping control visual feature vector and dual-arm grasping control semantic feature vector through a cross-modal attention mechanism to obtain a fused feature vector; determine the task objective of the dual-arm grasping control target task based on the dual-arm grasping control fused feature vector, and infer the division of labor logic of the dual arms based on the dual-arm grasping control task objective, and determine the action priority and collaboration sequence of the left and right arms; generate global planning information containing dual-arm collaborative actions based on the dual-arm grasping control action priority, dual-arm grasping control collaboration sequence, and combined with the spatial collaboration range.

[0103] In some embodiments of this application, when dynamically generating action commands for the dual robotic arms based on the global planning information of the dual robotic arm grasping control motion decoder reinforcement learning model, the control module 32 can specifically be used to: input the visual feature vector of the dual robotic arm grasping control into a pre-trained grasping point prediction model; use the convolutional neural network in the grasping point prediction model to regress and calculate the coordinates of the target grasping point; use the dual robotic arm grasping control motion decoder reinforcement learning model to generate the first action command for the right arm at the current moment based on the global planning information of the dual robotic arm grasping control, the coordinates of the target grasping point of the dual robotic arm grasping control, and the first historical action of the left arm at the previous moment; and use the dual robotic arm grasping control motion decoder reinforcement learning model to generate the second action command for the left arm at the current moment based on the global planning information of the dual robotic arm grasping control, the coordinates of the target grasping point of the dual robotic arm grasping control, and the second historical action of the right arm at the previous moment.

[0104] In some embodiments of this application, such as Figure 4 As shown, the device also includes a training module 33;

[0105] In training the grasping point prediction model, the training module 33 can specifically be used to: collect sample visual feature vectors of various sample items and their corresponding grasping point coordinates as the first training data; the grasping point coordinates of the dual robotic arm grasping control are determined based on the geometric features of the items and the grasping stability analysis; use the first training data of the dual robotic arm grasping control to iteratively train the convolutional neural network in the grasping point prediction model of the dual robotic arm grasping control, so that the grasping point prediction model of the dual robotic arm grasping control learns the mapping relationship between the visual features of the items and the grasping point coordinates; when the prediction error of the grasping point prediction model of the dual robotic arm grasping control on the validation set is less than a first preset threshold, or the number of iterative training times reaches a preset upper limit, it is determined that the training of the grasping point prediction model of the dual robotic arm grasping control is complete.

[0106] In some embodiments of this application, when the dual-arm grasping control action commands are adjusted in a closed loop based on the real-time execution state of the dual-arm grasping control to achieve dynamic collaborative control of the dual-arm movements, the control module 32 can specifically be used to: determine the preset action parameters corresponding to the dual-arm grasping control action commands, the preset action parameters of the dual-arm grasping control including the joint angle parameters, motion speed parameters, and grasping force parameters of the dual-arm grasping control; collect the real-time execution data of the dual-arm grasping control through force sensors and position sensors, the real-time execution data of the dual-arm grasping control including joint angles, spatial positions, and contact forces; calculate the deviation value between the real-time execution data of the dual-arm grasping control and the preset action parameters of the dual-arm grasping control; if the deviation value of the dual-arm grasping control exceeds a second preset threshold, adjust the action command at the next time node through a reinforcement learning algorithm; repeat the above steps until the grasping task is completed or the deviation value of the dual-arm grasping control is within the allowable range.

[0107] In some embodiments of this application, the training module 33, when training the dual-arm grasping model, can specifically select multiple different but related grasping task sets, each grasping task being configured with corresponding second training data; based on the second training data of each grasping task, a meta-learning mechanism is used to perform meta-learning training on the dual-arm grasping model. During the meta-learning training process, a set of initial basic parameters is set for the dual-arm grasping control model, serving as the starting point for training all grasping tasks; the dual-arm grasping control model is trained on each grasping task using its corresponding second training data, through gradient... The descent method generates temporary parameters adapted to the single grasping task; the meta-loss function for all grasping tasks is calculated, and the meta-loss function for dual-arm grasping control is optimized based on the temporary parameters of each grasping task to adjust the basic parameters of dual-arm grasping control until the meta-loss function converges, thus obtaining the initial model parameters of the dual-arm grasping control model; transfer learning technology is used to fine-tune the initial model parameters of dual-arm grasping control using the third training data of the dual-arm grasping control target task, and by minimizing the target loss function, a dual-arm grasping model adapted to the dual-arm grasping control target task is obtained.

[0108] It should be noted that other corresponding descriptions of the functional units involved in the dual-arm gripping control device provided in this embodiment can be found in [reference needed]. Figure 1 and Figure 2 The corresponding descriptions in [the document] will not be repeated here.

[0109] Based on the above, Figure 1 and Figure 2 Accordingly, this embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements the above-described method. Figure 1 and Figure 2 The dual robotic arm grasping control method is shown.

[0110] Based on this understanding, the technical solution of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as CD-ROM, USB flash drive, mobile hard drive, etc.) and includes several instructions to cause an electronic device (such as a personal computer, server, or network device, etc.) to execute the methods of various implementation scenarios of this application.

[0111] Based on the above, Figure 1 and Figure 2 The method shown, and Figure 3 and Figure 4 To achieve the above objectives, the present application also provides an electronic device, specifically a personal computer, tablet computer, server, or other network device, as shown in the virtual device embodiment. This device includes a storage medium and a processor; the storage medium stores a computer program; the processor executes the computer program to achieve the above-described objectives. Figure 1 and Figure 2 The dual robotic arm grasping control method is shown.

[0112] Optionally, the aforementioned physical devices may also include a user interface, a network interface, a camera, radio frequency (RF) circuitry, sensors, audio circuitry, a Wi-Fi module, etc. The user interface may include a display screen, input units such as a keyboard, etc., and optional user interfaces may also include USB interfaces, card reader interfaces, etc. The network interface may optionally include standard wired interfaces, wireless interfaces (such as Wi-Fi interfaces), etc.

[0113] Those skilled in the art will understand that the physical device structure provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or combine certain components, or have different component arrangements.

[0114] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the aforementioned physical device, supporting the operation of information processing programs and other software and / or programs. The network communication module is used to enable communication between the various components within the storage medium, as well as communication with other hardware and software in the information processing physical device.

[0115] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platform, or it can be implemented by hardware.

[0116] This invention, through the construction of a dual-arm grasping model integrating a visual encoder, a language encoder, an LLM (Limited Linear Model), and an action decoder reinforcement learning model, effectively addresses the issue of insufficient flexibility in dual-arm collaboration across various scenarios, including fintech and healthcare. Specifically, the visual encoder and language encoder accurately extract visual features and parse language commands, respectively. The global plan generated after fusion by the LLM model clearly defines action priorities and collaboration sequences, enabling the dual robotic arms to precisely coordinate according to task requirements in complex tasks. For example, it avoids action conflicts and synchronization deviations in multi-step financial document processing or collaborative transfer of surgical instruments. The action decoder reinforcement learning model dynamically generates action commands based on the global plan and optimizes parameters through reinforcement learning in conjunction with real-time status, achieving closed-loop adjustment. This ensures that action commands adapt to the real-time execution state, significantly improving operational efficiency, reducing the risk of item damage, and enhancing the robotic arm's adaptability to complex scenarios, thus expanding its application scope in fintech, healthcare, and other fields.

[0117] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing this application. Those skilled in the art will understand that the modules in the apparatus of the embodiment can be distributed within the apparatus of the embodiment as described, or can be modified to be located in one or more apparatuses different from this embodiment. The modules of the above-described embodiment can be combined into one module, or further divided into multiple sub-modules.

[0118] The serial numbers in this application are for descriptive purposes only and do not represent the superiority or inferiority of any particular implementation scenario. The above disclosures are merely a few specific implementation scenarios of this application; however, this application is not limited thereto, and any variations conceived by those skilled in the art should fall within the protection scope of this application.

Claims

1. A dual-arm gripping control method, characterized in that, include: Receive natural language task instructions, which are used to instruct the execution of a target task of grasping a target object; In response to the natural language task instruction, a dual-arm grasping model is used to perform dynamic collaborative control of the movements of the two robotic arms. The dual-arm grasping model includes: a visual encoder, a language encoder, an LLM large model, and an action decoder reinforcement learning model. The dynamic collaborative control of the two robotic arms' movements in response to the natural language task instruction, using a dual-arm grasping model, includes: In response to the natural language task instruction, the visual encoder is used to extract the visual feature vector of the target item, and the language encoder is used to parse the natural language task instruction to obtain the semantic feature vector. The LLM large model is used to fuse the visual feature vector and the semantic feature vector, and global planning information containing the coordinated movements of both arms is generated based on the fused feature vector. The motion decoder reinforcement learning model dynamically generates motion commands for the dual robotic arms based on the global planning information, and performs closed-loop adjustments to the motion commands according to the real-time execution status of the dual robotic arms. The motion commands are used to achieve dynamic collaborative control of the movements of the dual arms.

2. The method according to claim 1, characterized in that, The visual encoder includes a local encoder and a global encoder. Extracting the visual feature vector of the target object using the visual encoder includes: The local encoder is used to extract features from the local image of the target item through a convolutional neural network to obtain local detail features; The global encoder, based on the Transformer architecture, is used to extract features from the global scene image containing the target object to obtain the global spatial relationship features between the target object and its surrounding environment. The local detail features and the global spatial relationship features are fused to obtain the visual feature vector of the target object.

3. The method according to claim 1, characterized in that, The LLM large model is used to fuse the visual feature vector and the semantic feature vector, and global planning information containing the coordinated movements of both arms is generated based on the fused feature vector, including: The visual feature vector and the semantic feature vector are fused using a cross-modal attention mechanism to obtain a fused feature vector; Based on the fused feature vector, the task objective of the target task is determined, and based on the task objective, the division of labor logic of the two robotic arms is inferred to determine the action priority and cooperation sequence of the left and right arms. Based on the action priority, the collaboration timing, and the spatial collaboration range, global planning information including the coordinated movements of both arms is generated.

4. The method according to claim 1, characterized in that, The motion decoder reinforcement learning model dynamically generates motion commands for the dual robotic arms based on the global planning information, including: The visual feature vector is input into a pre-trained grasping point prediction model, and the target grasping point coordinates are calculated by using the convolutional neural network in the grasping point prediction model to regress. The action decoder reinforcement learning model generates the first action command for the right arm at the current moment based on the global planning information, the target grasping point coordinates, and the first historical action of the left arm at the previous moment; and, The action decoder reinforcement learning model generates the second action command for the left arm at the current moment based on the global planning information, the coordinates of the target grasping point, and the second historical action of the right arm at the previous moment.

5. The method according to claim 4, characterized in that, The method also includes a training method for the grab point prediction model: Collect visual feature vectors of various sample items and their corresponding grasping point coordinates as the first training data. The grasping point coordinates are determined based on the geometric features of the items and the grasping stability analysis. The convolutional neural network in the grasping point prediction model is iteratively trained using the first training data, so that the grasping point prediction model learns the mapping relationship between the visual features of the object and the coordinates of the grasping point. When the prediction error of the grasp point prediction model on the validation set is less than a first preset threshold, or the number of iterations reaches a preset upper limit, the grasp point prediction model is judged to have completed training.

6. The method according to claim 1, characterized in that, The step of adjusting the motion commands in a closed loop based on the real-time execution status of the dual robotic arms to achieve dynamic coordinated control of the dual-arm movements includes: Determine the preset motion parameters corresponding to the motion command, the preset motion parameters including the joint angle parameters, motion speed parameters and gripping force parameters of the dual robotic arms; Real-time execution data of the dual robotic arms is collected by force sensors and position sensors. The real-time execution data includes joint angles, spatial positions, and contact forces. Calculate the deviation between the real-time execution data and the preset action parameters; If the deviation value exceeds the second preset threshold, the action command at the next time node is adjusted through a reinforcement learning algorithm; Repeat the above steps until the capture task is completed or the deviation value is within the allowable range.

7. The method according to claim 1, characterized in that, The method also includes a training method for the dual-robotic arm grasping model: Select multiple different but related sets of crawling tasks, and configure corresponding second training data for each crawling task; Based on the second training data for each grasping task, a meta-learning mechanism is used to train the dual-arm grasping model. During the meta-learning training process, a set of initial basic parameters is set for the dual-arm grasping model as the starting point for training all grasping tasks. The dual-arm grasping model is trained on each grasping task using its corresponding second training data, and temporary parameters adapted to the individual grasping task are generated by gradient descent. The meta-loss function for all grasping tasks is calculated, and the meta-loss function is optimized based on the temporary parameters of each grasping task to adjust the basic parameters until the meta-loss function converges, thus obtaining the initial model parameters of the dual-arm grasping model. By employing transfer learning techniques and using third training data for the target task to fine-tune the parameters of the initial model, a dual-arm grasping model adapted to the target task is obtained by minimizing the target loss function.

8. A dual-arm gripping control device, characterized in that, include: A receiving module is used to receive natural language task instructions, which are used to instruct the execution of a target task of grasping a target item. The control module is used to respond to the natural language task instructions and use a dual-arm grasping model to perform dynamic collaborative control of the dual-arm movements. The dual-arm grasping model includes: a visual encoder, a language encoder, an LLM large model, and an action decoder reinforcement learning model. The control module is specifically used for: In response to the natural language task instruction, the visual encoder is used to extract the visual feature vector of the target item, and the language encoder is used to parse the natural language task instruction to obtain the semantic feature vector. The LLM large model is used to fuse the visual feature vector and the semantic feature vector, and global planning information containing the coordinated movements of both arms is generated based on the fused feature vector. The motion decoder reinforcement learning model dynamically generates motion commands for the dual robotic arms based on the global planning information, and performs closed-loop adjustments to the motion commands according to the real-time execution status of the dual robotic arms. The motion commands are used to achieve dynamic collaborative control of the movements of the dual arms.

9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

10. An electronic device comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 7.

Images