Dual-robot collaborative picking method and system based on multi-source vision cross-view fusion

Abstract

The application provides a dual-robot arm cooperative goods taking method and system based on multi-source vision cross-view fusion, relates to the technical field of intelligent control, and comprises the following steps: acquiring external depth camera and wrist camera data, analyzing language task description to generate a weighted directed task graph, fusing depth data and near-field image data by using geometric projection transformation and cross-modal attention mechanism, forming a confidence field to guide the mechanical arm to reacquire data, and finally outputting a division scheme and a grasping posture sequence by a vision-language-action model. The application improves the cooperative goods taking accuracy and flexibility in a complex scene.

Description

Technical Field

[0001] This invention relates to the field of intelligent control technology, and in particular to a dual-robotic arm collaborative picking method and system based on multi-source vision cross-view fusion. Background Technology

[0002] With the development of robotics technology, the demand for collaborative dual-arm robotic operations in industrial automation, warehousing and logistics, and service robotics is growing. Dual-arm systems can accomplish complex tasks that a single arm cannot, such as handling large, irregular, or flexible objects, through collaborative work. Currently, vision-based robotic arm grasping technology has made significant progress, but challenges remain in perception fusion, task planning, and division of labor coordination in multi-arm collaborative operations.

[0003] Traditional robotic arm vision grasping systems typically rely on perception data from a single viewpoint, failing to comprehensively capture the spatial geometry of the workspace and the detailed features of the target object. This results in a reduced grasping success rate in complex scenarios. In particular, when objects are occluded, stacked, or irregularly arranged, the perception information from a single viewpoint is often insufficient to support accurate grasping decisions.

[0004] Existing dual-arm collaborative systems generally suffer from the following drawbacks: First, they lack an effective multi-source visual data fusion mechanism. Information acquired by the external global camera and the near-field camera on the robotic arm's wrist is difficult to integrate in a unified coordinate system, making it impossible to simultaneously utilize global geometric features and local detail features. Second, existing systems typically employ fixed collaborative modes, making it difficult to dynamically adjust the robotic arm's task allocation strategy based on verbally described task requirements, thus limiting the system's adaptability to complex and variable tasks. Furthermore, when there is uncertainty regarding the grasping target, the system lacks proactive perception capabilities and cannot improve the reliability of grasping decisions by incrementally acquiring more information, leading to frequent failures when facing partially visible or difficult-to-identify objects. Summary of the Invention

[0005] The present invention provides a method and system for collaborative picking up goods by dual robotic arms based on multi-source vision cross-view fusion, which can solve the problems in the prior art.

[0006] A first aspect of the present invention provides a dual-robotic arm collaborative picking method based on multi-source vision cross-view fusion, comprising:

[0007] Acquire workspace depth data from an external depth camera and near-field image data from the wrist cameras of each of the two robotic arms; receive a language task description, parse the language task description to extract constraints, and generate a weighted directed task graph;

[0008] The depth data and the near-field image data are aligned to a unified coordinate system through geometric projection transformation, and the geometric features of the depth data and the semantic features of the near-field image data are fused using a cross-modal attention mechanism to generate a fused feature representation.

[0009] Based on the fusion feature representation, candidate points for grasping are extracted and confidence scores are calculated to form a confidence field. When there are candidate points for grasping in the confidence field with confidence scores lower than a preset gate threshold, the corresponding robotic arm is controlled to approach the candidate point for grasping and re-acquire near-field image data to update the fusion feature representation and the confidence field.

[0010] The weighted directed task graph, the fused feature representation, and the confidence field are input into the vision-language-action model. The vision-language-action model outputs a division of labor scheme and a grasping posture sequence for the two robotic arms. The division of labor scheme specifies the support action or object-grabbing action to be performed by each robotic arm and their synchronization relationship.

[0011] Parsing the language task description to extract constraints and generating a weighted directed task graph includes:

[0012] The language task description is semantically parsed to identify the attribute information of the pickup object, including color constraints, size constraints, category constraints, placement constraints, and order constraints.

[0013] A set of task nodes is constructed based on the attribute information, and each task node corresponds to a pickup sub-task.

[0014] Based on the order constraints, directed edges are determined between each task node in the task node set, and the directed edges represent the sequential dependencies of task execution.

[0015] Assign a weight to each of the directed edges, where the weight represents the priority of the corresponding task dependency;

[0016] The weighted directed task graph is formed by combining the set of task nodes, the directed edges, and the weights.

[0017] The depth data and the near-field image data are aligned to a unified coordinate system through geometric projection transformation, and a cross-modal attention mechanism is used to fuse the geometric features of the depth data and the semantic features of the near-field image data to generate a fused feature representation, including:

[0018] Based on the intrinsic and extrinsic parameter matrices of the external depth camera, the depth data is transformed to the unified coordinate system to obtain aligned depth data;

[0019] Based on the intrinsic and extrinsic parameter matrices of the wrist cameras of the two robotic arms, the near-field image data is transformed to the unified coordinate system to obtain aligned near-field image data.

[0020] Geometric features are extracted from the alignment depth data, and semantic features are extracted from the alignment near-field image data;

[0021] Construct a cross-modal attention matrix, wherein the cross-modal attention matrix represents the association weight between each feature element in the geometric feature and each feature element in the semantic feature;

[0022] The geometric features and semantic features are weighted and fused based on the cross-modal attention matrix to generate the fused feature representation.

[0023] The weighted directed task graph, the fused feature representation, and the confidence field are input into the vision-language-action model. The vision-language-action model outputs a division of labor scheme and grasping posture sequence for the two robotic arms, including:

[0024] The weighted directed task graph is encoded as a task embedding vector, the fused feature representation is encoded as a visual embedding vector, and the confidence field is encoded as a confidence embedding vector.

[0025] The task embedding vector, the visual embedding vector, and the confidence embedding vector are input into the encoder of the vision-language-action model to generate a joint representation;

[0026] Based on the joint representation, an action sequence is generated by the decoder of the vision-language-action model, the action sequence including the action of the left robotic arm and the action of the right robotic arm;

[0027] The action sequence is decomposed into the division of labor scheme according to the executing subject of each action in the action sequence. The division of labor scheme specifies that the action of the left robotic arm is a supporting action and the action of the right robotic arm is a picking action, or specifies that the action of the left robotic arm is a picking action and the action of the right robotic arm is a supporting action.

[0028] Extract the grab position and grab posture corresponding to each action from the action sequence to form the grab posture sequence.

[0029] Generating action sequences based on the joint representation using the decoder of the vision-language-action model includes:

[0030] The joint representation is input into the attention layer of the decoder to calculate the cross-attention weights between the task embedding vector, the visual embedding vector, and the confidence embedding vector.

[0031] The joint representation is weighted and aggregated according to the cross-attention weights to generate a context representation;

[0032] The context representation is input into the feedforward network of the decoder to generate the action sequence step by step. Each action in the action sequence includes an execution subject identifier, an action type identifier, a grasping position coordinate, and a grasping posture parameter.

[0033] The method further includes:

[0034] Based on the depth data, a spatial occupancy grid is constructed, and the distribution of obstacles and accessibility areas are determined according to the spatial occupancy grid.

[0035] Based on the reachability area, the feasibility of the grasping posture sequence is screened, and unreachable grasping postures are eliminated.

[0036] The execution trajectory of the dual robotic arms is generated based on the screened grasping posture sequence, and speed constraints, torque constraints and cooperative safety area constraints are superimposed on the execution trajectory.

[0037] The dual robotic arms are controlled to perform a picking operation according to the execution trajectory. During the execution, the deviation between the actual trajectory and the execution trajectory is monitored. When the deviation exceeds a preset deviation threshold, the near-field image data acquisition is re-executed and the execution trajectory is updated.

[0038] A second aspect of the present invention provides a dual-robotic arm collaborative picking system based on multi-source vision cross-view fusion, comprising:

[0039] The first unit is used to acquire workspace depth data collected by an external depth camera and near-field image data collected by the wrist cameras of each of the two robotic arms; receive a language task description, parse the language task description to extract constraints, and generate a weighted directed task graph.

[0040] The second unit is used to align the depth data and the near-field image data to a unified coordinate system through geometric projection transformation, and to fuse the geometric features of the depth data and the semantic features of the near-field image data using a cross-modal attention mechanism to generate a fused feature representation.

[0041] The third unit is used to extract grasping candidate points based on the fusion feature representation and calculate confidence scores to form a confidence field; when there are grasping candidate points in the confidence field with confidence scores lower than a preset gate threshold, the corresponding robotic arm is controlled to approach the grasping candidate point and re-acquire near-field image data to update the fusion feature representation and the confidence field.

[0042] The fourth unit is used to input the weighted directed task graph, the fused feature representation and the confidence field into the vision-language-action model. The vision-language-action model outputs the division of labor scheme and grasping posture sequence of the two robotic arms. The division of labor scheme specifies the support action or object picking action to be performed by each robotic arm and their synchronization relationship.

[0043] A third aspect of the present invention provides an electronic device, comprising:

[0044] processor;

[0045] Memory used to store processor-executable instructions;

[0046] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0047] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0048] The beneficial effects of this application are as follows:

[0049] By fusing multi-view data acquired from external depth cameras and dual robotic arm wrist cameras, the system effectively combines global geometric information with local semantic details, significantly improving the accuracy of target recognition and localization in complex scenes, especially for object grasping in challenging situations such as occlusion, reflection, or complex textures.

[0050] By adopting a confidence field dynamic adjustment strategy, when the system detects a grasping candidate point with low confidence, it can actively control the robotic arm to adjust its position and re-collect data, forming a closed-loop feedback mechanism, which effectively improves the grasping success rate and reduces potential risks.

[0051] Based on a vision-language-action model, the intelligent division of labor and collaboration between the two robotic arms is realized. It can adaptively plan support and object retrieval actions according to language commands and scene characteristics, solving complex operation tasks that are difficult to handle by traditional single-arm systems, such as picking up and placing unstable objects and handling large items that require coordination, thereby improving the overall system's flexibility and task execution efficiency. Attached Figure Description

[0052] Figure 1 This is a flowchart illustrating the collaborative picking method of dual robotic arms based on multi-source vision cross-view fusion according to an embodiment of the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0055] Figure 1 This is a flowchart illustrating the dual-robotic arm collaborative picking method based on multi-source vision cross-view fusion, as described in an embodiment of the present invention. Figure 1 As shown, the method includes:

[0056] Acquire workspace depth data from an external depth camera and near-field image data from the wrist cameras of each of the two robotic arms; receive a language task description, parse the language task description to extract constraints, and generate a weighted directed task graph;

[0057] The depth data and the near-field image data are aligned to a unified coordinate system through geometric projection transformation, and the geometric features of the depth data and the semantic features of the near-field image data are fused using a cross-modal attention mechanism to generate a fused feature representation.

[0058] Based on the fusion feature representation, candidate points for grasping are extracted and confidence scores are calculated to form a confidence field. When there are candidate points for grasping in the confidence field with confidence scores lower than a preset gate threshold, the corresponding robotic arm is controlled to approach the candidate point for grasping and re-acquire near-field image data to update the fusion feature representation and the confidence field.

[0059] The weighted directed task graph, the fused feature representation, and the confidence field are input into the vision-language-action model. The vision-language-action model outputs a division of labor scheme and a grasping posture sequence for the two robotic arms. The division of labor scheme specifies the support action or object-grabbing action to be performed by each robotic arm and their synchronization relationship.

[0060] In one alternative implementation, parsing the language task description to extract constraints and generating a weighted directed task graph includes:

[0061] The language task description is semantically parsed to identify the attribute information of the pickup object, including color constraints, size constraints, category constraints, placement constraints, and order constraints.

[0062] A set of task nodes is constructed based on the attribute information, and each task node corresponds to a pickup sub-task.

[0063] Based on the order constraints, directed edges are determined between each task node in the task node set, and the directed edges represent the sequential dependencies of task execution.

[0064] Assign a weight to each of the directed edges, where the weight represents the priority of the corresponding task dependency;

[0065] The weighted directed task graph is formed by combining the set of task nodes, the directed edges, and the weights.

[0066] The language task description is semantically parsed to identify the attribute information of the picking object, including color constraints, size constraints, category constraints, placement constraints, and order constraints. Then, a set of task nodes is constructed based on the attribute information. Next, the directed edges between task nodes are determined based on the order constraints. Then, a weight is assigned to each directed edge. Finally, the set of task nodes, directed edges, and weights are combined to form a weighted directed task graph.

[0067] Semantic parsing of language task descriptions is performed in this embodiment, employing a semantic slot-filling method. For example, for the task description "first pick up the red cube on the table, then take the blue sphere from the shelf," the system uses predefined semantic framework templates for matching. These templates include action slots (e.g., "pick up," "take away"), object slots (e.g., "cube," "sphere"), attribute slots (e.g., "red," "blue"), location slots (e.g., "on the table," "on the shelf"), and temporal slots (e.g., "first," "then"). The system maps words in the task description to corresponding semantic slots through part-of-speech tagging and dependency parsing. In the example above, the system identifies two sub-tasks for retrieving goods. The attribute information of the first sub-task includes the color constraint "red," the category constraint "cube," and the location constraint "on the table." The attribute information of the second sub-task includes the color constraint "blue," the category constraint "sphere," and the location constraint "on the shelf," and there is a sequence constraint "execute the first sub-task first, then execute the second sub-task."

[0068] After identifying the attributes of the item to be picked up, a set of task nodes is constructed based on these attributes. Each task node corresponds to a pickup subtask and contains complete attribute information for that subtask. During the construction process, the system creates a unique identifier for each subtask and associates the subtask's attribute information with the identifier. For the example above, the system creates two task nodes: Node 1 represents "pick up the red cube on the table," containing attribute information {color: "red", category: "cube", location: "on the table", action: "pick up"}; Node 2 represents "take the blue sphere from the shelf," containing attribute information {color: "blue", category: "sphere", location: "on the shelf", action: "take away"}.

[0069] Determining directed edges between task nodes based on sequence constraints is a crucial step in constructing task dependencies. The execution order is determined by extracting temporal keywords (such as "first," "then," "after," and "finally") and logical connectors (such as "and," "simultaneously," and "or") from the language task description. For explicit sequence constraints, the system directly establishes directed edges between the corresponding task nodes. For implicit sequence constraints, the system infers the execution order by analyzing the syntactic structure and semantic relationships of the task description. In the example above, based on the "first...then..." sequence constraint, the system establishes a directed edge from node 1 to node 2, indicating that node 1 must be executed before node 2.

[0070] Assigning weights to each directed edge reflects the priority of task dependencies. In this implementation, the weight values ​​are set from 1 to 10, with higher values ​​indicating higher priority. Weight allocation is based on several factors: the strength of temporal keywords (e.g., "must first" has a higher weight than "best first"), task complexity (e.g., tasks involving multiple attribute constraints have higher weights), object characteristics (e.g., tasks related to fragile items have higher weights), and the logical relationships between tasks (e.g., mutually exclusive relationships have higher weights than parallel relationships). In the example above, because an explicit "first...then..." structure is used, the system assigns a weight of 8 to the directed edge from node 1 to node 2, indicating a high-priority sequential dependency.

[0071] The system combines task node sets, directed edges, and weights to form a weighted directed task graph. This graph is stored using an adjacency list structure, containing a node list and an edge list. Each element in the node list contains a node identifier and corresponding attribute information; each element in the edge list contains a starting node identifier, a target node identifier, and a weight value. For complex task descriptions, such as "first pick up the red cube and blue sphere on the table, then take the yellow cylinder from the shelf, and finally put down the green cube," the system identifies four retrieval subtasks, creates four task nodes, and establishes three directed edges according to the order constraint "first...then...last...": the weight from node 1 (red cube) to node 3 (yellow cylinder) is 7, the weight from node 2 (blue sphere) to node 3 is 7, and the weight from node 3 to node 4 (green cube) is 9. There is no directed edge between node 1 and node 2, indicating that they can be executed in parallel.

[0072] The weighted directed task graph generated through the above steps provides a structured representation of the task, capturing object attribute information and execution order constraints. This task graph can be used for subsequent task planning and execution, especially in the field of robot operation, guiding robots to complete complex picking tasks in the correct order. The system also has the ability to handle conditional branches. For example, a task description such as "If there is a red square on the table, take it first; otherwise, take the blue ball directly" can be represented in the task graph as condition nodes and multiple execution paths, each with different weights to reflect the execution priority under different conditions.

[0073] In one optional implementation, the depth data and the near-field image data are aligned to a unified coordinate system through geometric projection transformation, and a cross-modal attention mechanism is used to fuse the geometric features of the depth data and the semantic features of the near-field image data to generate a fused feature representation, including:

[0074] Based on the intrinsic and extrinsic parameter matrices of the external depth camera, the depth data is transformed to the unified coordinate system to obtain aligned depth data;

[0075] Based on the intrinsic and extrinsic parameter matrices of the wrist cameras of the two robotic arms, the near-field image data is transformed to the unified coordinate system to obtain aligned near-field image data.

[0076] Geometric features are extracted from the alignment depth data, and semantic features are extracted from the alignment near-field image data;

[0077] Construct a cross-modal attention matrix, wherein the cross-modal attention matrix represents the association weight between each feature element in the geometric feature and each feature element in the semantic feature;

[0078] The geometric features and semantic features are weighted and fused based on the cross-modal attention matrix to generate the fused feature representation.

[0079] The system acquires depth data from an external depth camera and near-field image data from the wrist cameras of each of the two robotic arms. The external depth camera, which can be a structured light or time-of-flight (ToF) type depth camera, is mounted above the operating scene and has a large field of view. The acquired depth data is represented as point cloud data containing multiple points, each with three-dimensional coordinates (x, y, z). The wrist cameras of the two robotic arms are RGB cameras, and the acquired near-field image data is a high-resolution color image.

[0080] Based on the intrinsic and extrinsic parameter matrices of the external depth camera, the depth data is transformed to a unified coordinate system to obtain aligned depth data. The intrinsic parameter matrix of the external depth camera contains parameters such as focal length and principal point coordinates, and is typically a 3×3 matrix; the extrinsic parameter matrix contains camera position and pose information, and is typically a 4×4 matrix. Specifically, for each 3D point (x... d y d z d The following transformation steps are used to map the point to a unified coordinate system: multiply the point in the depth camera coordinate system by the extrinsic parameter matrix of the depth camera to obtain the point in the world coordinate system (x). w y w z w In practical implementation, the extrinsic parameter matrix of a depth camera can be obtained through calibration using a calibration board. For example, for a depth camera installed at a height of 1.5 meters and with a top-down angle of 60 degrees, the rotation part of its extrinsic parameter matrix reflects the 60-degree top-down angle, and the translation part reflects the height position of 1.5 meters.

[0081] Based on the intrinsic and extrinsic parameter matrices of the wrist cameras of the two robotic arms, the near-field image data is transformed to a unified coordinate system to obtain aligned near-field image data. The intrinsic parameter matrix of the wrist cameras of the two robotic arms also includes parameters such as focal length and principal point coordinates; the extrinsic parameter matrix changes with the movement of the robotic arms, representing the position and orientation of the camera in the world coordinate system. For each pixel (u, v) in the image captured by the wrist camera, its corresponding 3D point (x, v) is first estimated using depth information. c y c z c Then, the extrinsic parameter matrix of the wrist camera is used to transform it to the world coordinate system to obtain the point (x). w y w z wIn practical applications, the extrinsic parameter matrix of the wrist camera can be calculated through the forward kinematics of the robotic arm and hand-eye calibration. For example, when the end effector of the robotic arm is located at the center of the workspace (0.5, 0, 0.3) meters and the camera is facing vertically downwards, the corresponding extrinsic parameter matrix reflects the position and orientation information.

[0082] Geometric features are extracted from aligned depth data, and semantic features are extracted from aligned near-field image data. For aligned depth data, a 3D convolutional network is used to extract geometric features. Specifically, the point cloud data is voxelized into a 64×64×64 3D grid with a grid resolution of 5 mm. The value of each voxel indicates whether a point exists at that location. After processing through three 3D convolutional layers with kernel sizes of 3×3×3 and channel numbers of 32, 64, and 128 respectively, geometric features with dimensions of 16×16×16×128 are obtained. For aligned near-field image data, a 2D convolutional neural network is used to extract semantic features. Images with a resolution adjusted to 256×256 are processed through five 2D convolutional layers with kernel sizes of 3×3 and channel numbers of 32, 64, 128, 256, and 512 respectively, resulting in semantic features with dimensions of 16×16×512.

[0083] Construct a cross-modal attention matrix to represent the association weight between each feature element in the geometric features and each feature element in the semantic features. Reshape the geometric features into a matrix F of size (16×16×16)×128. g The semantic features are reshaped into a matrix F of size (16×16)×512. s F is transformed through a linear mapping. g Transform F into a query matrix Q of size (16×16×16)×256. s The key matrix K is transformed into a (16×16)×256 key matrix. The dot product of Q and the transpose of K is calculated to obtain an attention score matrix of size (16×16×16)×(16×16). The attention score matrix is ​​then softmax normalized to obtain the cross-modal attention matrix A. In practical applications, when the geometric features of the object edge region in the depth data are highly correlated with the semantic features of the corresponding region in the near-field image, the corresponding attention weight is high, for example, the weight value can reach above 0.8.

[0084] Geometric and semantic features are weighted and fused using a cross-modal attention matrix to generate a fused feature representation. The semantic features F... sThe linear transformation yields a value matrix V of size (16×16)×128. The matrix product of the attention matrix A and the value matrix V is calculated to obtain a weighted semantic feature of size (16×16×16)×128. This weighted semantic feature is then reshaped back to a 16×16×16×128 shape and element-wise added to the original geometric features to obtain the fused feature representation F. fusion In practical applications, fused feature representations can simultaneously preserve the precise geometric information of depth data and the rich semantic information of near-field images. For example, for a cylindrical object in a scene, the fused features contain both its accurate three-dimensional shape information and semantic information such as surface texture and color.

[0085] Through the above technical solutions, effective alignment and fusion of depth data and near-field image data are achieved. The generated fused feature representation can provide more comprehensive and accurate scene understanding information for subsequent dual-robotic arm collaborative operation tasks.

[0086] In one optional implementation, the weighted directed task graph, the fused feature representation, and the confidence field are input into the vision-language-action model, and the vision-language-action model outputs a division of labor scheme and grasping posture sequence for the two robotic arms, including:

[0087] The weighted directed task graph is encoded as a task embedding vector, the fused feature representation is encoded as a visual embedding vector, and the confidence field is encoded as a confidence embedding vector.

[0088] The task embedding vector, the visual embedding vector, and the confidence embedding vector are input into the encoder of the vision-language-action model to generate a joint representation;

[0089] Based on the joint representation, an action sequence is generated by the decoder of the vision-language-action model, the action sequence including the action of the left robotic arm and the action of the right robotic arm;

[0090] The action sequence is decomposed into the division of labor scheme according to the executing subject of each action in the action sequence. The division of labor scheme specifies that the action of the left robotic arm is a supporting action and the action of the right robotic arm is a picking action, or specifies that the action of the left robotic arm is a picking action and the action of the right robotic arm is a supporting action.

[0091] Extract the grab position and grab posture corresponding to each action from the action sequence to form the grab posture sequence.

[0092] The weighted directed task graph is encoded as a task embedding vector. The weighted directed task graph is represented as G=(V,E,W), where V represents the set of task nodes, E represents the set of directed edges between nodes, and W represents the set of edge weights. A graph neural network is used for encoding. For each task node v... i The node features are updated by aggregating information from its neighboring nodes. The aggregation process uses an attention mechanism, assigning different importance to different neighboring nodes based on edge weights. After multiple layers of graph convolution operations, all node features are concatenated and passed through a fully connected layer to obtain a 512-dimensional task embedding vector t. emb For example, for a "grab an object" task graph containing 5 task nodes, each node represents "identify target", "approach target", "grab object", "lift object", and "place object", after encoding, a task embedding vector representing the overall task semantics and execution order is generated.

[0093] The fused feature representation is encoded as a visual embedding vector, which includes RGB image features and depth map features, with dimensions H×W×C, where H and W represent the height and width of the feature map, respectively, and C represents the number of feature channels. Visual features are further extracted using a convolutional neural network, comprising five convolutional blocks, each containing two 3×3 convolutional layers and one 2×2 max-pooling layer. Finally, adaptive pooling and fully connected layers transform the feature map into a 768-dimensional visual embedding vector v. emb For example, for an input scene image with a resolution of 640×480, after fusing feature extraction and visual encoding, a visual embedding vector representing information such as the shape, texture, and spatial location of objects in the scene is obtained.

[0094] The confidence field is encoded as a confidence embedding vector. The confidence field is a two-dimensional matrix of size H×W, where the value at each position represents the probability that the position is suitable for capture. Two consecutive convolutional layers (with kernel sizes of 5×5 and 3×3, respectively) are used to extract local features of the confidence field. Then, global average pooling and a fully connected layer are applied to generate a confidence embedding vector c with dimension 256. emb For example, for a confidence field where the confidence level around an object is high (value close to 1.0) while the confidence level in other areas is low (value close to 0.0), the encoded vector can effectively represent the potential grab location distribution information.

[0095] The task embedding vector, visual embedding vector, and confidence embedding vector are input into the encoder of the vision-language-action model to generate a joint representation. The encoder adopts a Transformer architecture and contains 6 encoding layers. First, the three embedding vectors are concatenated and positional encoding is added. Then, a multi-head self-attention mechanism and a feedforward neural network are used for processing. Each encoding layer has 8 attention heads, and the hidden layer dimension is 2048. The encoding process allows information from different modalities to be correlated and fused, ultimately generating a joint representation j with a dimension of 1536. rep For example, for a task of "taking an item out of a drawer", the joint representation can simultaneously include task semantics (the drawer needs to be opened first), visual information (the location of the drawer handle), and grasp confidence (the area suitable for grasping).

[0096] Action sequences are generated using a decoder based on a joint representation and a vision-language-action model. The decoder also employs a Transformer architecture, containing six decoding layers and generating the action sequences via an autoregressive approach. At each decoding step, the previously generated action is used as input, and a cross-attention mechanism is used to focus on relevant information in the joint representation. The output action sequence includes actions from the left and right robotic arms, with each action represented as a tuple containing quaternions for the executing entity, action type, position coordinates, and orientation. For example, for the "collaborative bottle opening" task, the generated action sequence is: [(left arm, grasp, [0.2, 0.3, 0.5], [0.707, 0.0, 0.0, 0.707]), (right arm, grasp, [0.2, 0.3, 0.7], [0.5, 0.5, 0.5, 0.5]), (left arm, fixed, [0.2, 0.3, 0.5], [0.707, 0.0, 0.0, 0.707]), (right arm, rotate, [0.2, 0.3, 0.7], [0.3, 0.3, 0.3, 0.9])].

[0097] The action sequence is decomposed into a division of labor scheme based on the executing entity of each action. By analyzing the executing entity (left arm or right arm) and the action type of each action, the role division of the two robotic arms is determined. Supporting actions typically include fixing, stabilizing, and supporting, while picking actions typically include grasping, extracting, and rotating. The role division of the left and right arms is determined based on the semantics and sequence of the actions. For example, in the bottle-opening task mentioned above, the left arm performs the action of fixing the bottle body, and the right arm performs the action of rotating the bottle cap. Therefore, the division of labor scheme is "the left robotic arm performs the supporting action, and the right robotic arm performs the picking action."

[0098] The grasping position and grasping posture corresponding to each action are extracted from the action sequence to form a grasping posture sequence. For each action, the position coordinates and orientation quaternions are extracted from its tuple representation to form the grasping posture. The position coordinates are represented as points [x, y, z] in three-dimensional space, and the orientation quaternions are represented as [qw, qx, qy, qz], which together define the pose of the robotic arm's end effector when performing the action. The final grasping posture sequence contains the pose information of all actions, arranged in the chronological order of action execution. For example, the grasping posture sequence for the above bottle cap opening task is: [([0.2, 0.3, 0.5], [0.707, 0.0, 0.0, 0.707]), ([0.2, 0.3, 0.7], [0.5, 0.5, 0.5, 0.5]), ([0.2, 0.3, 0.5], [0.707, 0.0, 0.0, 0.707]), ([0.2, 0.3, 0.7], [0.3, 0.3, 0.3, 0.9])].

[0099] In one alternative implementation, generating an action sequence based on the joint representation using the decoder of the vision-language-action model includes:

[0100] The joint representation is input into the attention layer of the decoder to calculate the cross-attention weights between the task embedding vector, the visual embedding vector, and the confidence embedding vector.

[0101] The joint representation is weighted and aggregated according to the cross-attention weights to generate a context representation;

[0102] The context representation is input into the feedforward network of the decoder to generate the action sequence step by step. Each action in the action sequence includes an execution subject identifier, an action type identifier, a grasping position coordinate, and a grasping posture parameter.

[0103] The joint representation is input into the attention layer of the decoder to calculate the cross-attention weights between the task embedding vector, the visual embedding vector, and the confidence embedding vector. Specifically, the attention layer of the decoder receives the joint representation as input, which includes the task embedding vector, the visual embedding vector, and the confidence embedding vector. The attention layer determines the cross-attention weights by calculating the relationship between the query vector, the key vector, and the value vector. In this embodiment, the task embedding vector is mapped to the query vector, and the visual embedding vector and the confidence embedding vector are mapped to the key vector and the value vector, respectively. The original attention score is obtained by performing a dot product operation on the query vector and the key vector, and then subjected to scaling and softmax normalization to finally obtain the cross-attention weights. For example, for a crawling task, the task embedding vector has a dimension of 256, the visual embedding vector has a dimension of 512, and the confidence embedding vector has a dimension of 128. After attention calculation, an attention weight matrix with a dimension of [number of tasks × (number of visual features + number of confidence features)] is obtained.

[0104] This method generates a context representation by weighted aggregation of the joint representation based on cross-attention weights. Specifically, the visual embedding vector and the confidence embedding vector are weighted and summed using the cross-attention weights obtained in the previous step to obtain the context representation. Features with higher weights have a larger proportion in the final context representation. In this embodiment, the context representation is a comprehensive representation vector reflecting visual and confidence information under the current task, and its dimension is the same as the task embedding vector, which is 256 dimensions. During the weighted aggregation process, the attention weight matrix is ​​multiplied by the value vector to obtain the attention-weighted feature representation. For example, for the identified object "water cup", if the visual features show that it is a blue cylinder and the confidence is high, then the visual features and confidence features related to "water cup" will receive higher attention weights and occupy a more important position in the context representation.

[0105] This method inputs the context representation into the feedforward network of the decoder, progressively decoding to generate action sequences. Specifically, the feedforward network of the decoder consists of multiple fully connected layers, each with activation functions and regularization processing. The feedforward network receives the context representation as input and outputs the action prediction result after multiple transformations. In this embodiment, the feedforward network contains three fully connected layers: the first layer maps the 256-dimensional context representation to a 512-dimensional hidden representation; the second layer maps the 512-dimensional hidden representation to a 256-dimensional hidden representation; and the third layer maps the 256-dimensional hidden representation to the action parameter space. The generation of the action sequence adopts an autoregressive approach, meaning that the generated actions serve as conditions for the generation of the next action. After each action is generated, it is encoded and combined with the context representation as a condition for generating the next action. For example, for a task containing three steps, "grab a water cup and put it in the sink," the first action, "robotic arm grabs the water cup," is decoded first. Then, this action is encoded and combined with the context representation to continue decoding the second action, "robotic arm lifts the water cup," and so on to complete the generation of the entire action sequence.

[0106] In this embodiment, each action in the action sequence includes an execution subject identifier, an action type identifier, grasping position coordinates, and grasping posture parameters. The execution subject identifier specifies the robot component performing the current action, such as "left robotic arm" or "right robotic arm"; the action type identifier specifies the basic type of action, such as "grasp," "place," or "push"; the grasping position coordinates represent the position of the target object in space using three-dimensional coordinates (x, y, z); and the grasping posture parameters represent the posture of the robotic arm's end effector using quaternions (w, x, y, z). For example, the complete representation of the action of grasping a water cup is: Execution subject = "right robotic arm", Action type = "grasp", Grasp position coordinates = (0.25, 0.40, 0.15), Grasp posture parameters = (0.707, 0.0, 0.707, 0.0). These parameters collectively define a complete robot action, enabling the robot to accurately perform the corresponding task.

[0107] In practical applications, the action sequence generation process also incorporates the robot's kinematic constraints and environmental interaction limitations to ensure the generated actions are executable. For example, for the task of "grabbing a water glass from a table," the model calculates the optimal gripping point and posture based on the visual features and confidence level of the water glass, while also considering the working range of the robotic arm and collision risks. The decoder analyzes key information in the context representation to determine the precise position of the water glass and the appropriate gripping method, and then generates a series of action instructions containing precise parameters, such as "move the right robotic arm to coordinates (0.25, 0.40, 0.15), adjust the posture to (0.707, 0.0, 0.707, 0.0), and close the gripper to 80% force," thereby completing the precise gripping of the water glass.

[0108] In an optional implementation, the method further includes:

[0109] Based on the depth data, a spatial occupancy grid is constructed, and the distribution of obstacles and accessibility areas are determined according to the spatial occupancy grid.

[0110] Based on the reachability area, the feasibility of the grasping posture sequence is screened, and unreachable grasping postures are eliminated.

[0111] The execution trajectory of the dual robotic arms is generated based on the screened grasping posture sequence, and speed constraints, torque constraints and cooperative safety area constraints are superimposed on the execution trajectory.

[0112] The dual robotic arms are controlled to perform a picking operation according to the execution trajectory. During the execution, the deviation between the actual trajectory and the execution trajectory is monitored. When the deviation exceeds a preset deviation threshold, the near-field image data acquisition is re-executed and the execution trajectory is updated.

[0113] According to the method described, after acquiring depth data, a spatial occupancy grid needs to be constructed to determine the distribution of obstacles and accessible areas in the working environment. The spatial occupancy grid is a three-dimensional representation method that divides the working space into uniform grid cells, each marked as occupied or vacant. Specifically, the system divides the working space into 0.05m × 0.05m × 0.05m cubic grid cells, assigning an occupancy probability value to each cell. Initially, the occupancy probability of all grid cells is set to 0.5, representing an unknown state. Point cloud data acquired by the depth camera is mapped to the corresponding grid cells. When a point is detected, the occupancy probability of its corresponding grid cell is increased to 0.85, while the occupancy probability of grid cells traversed by the point cloud ray decreases to 0.2, representing an vacant state. After fusion and updating of multiple frames of depth images, grid cells with an occupancy probability greater than 0.7 are marked as obstacles, and grid cells with an occupancy probability less than 0.3 are marked as passable areas, forming a complete spatial occupancy model.

[0114] Based on the constructed spatial occupancy grid, the system determines the distribution of obstacles and reachable areas. In practice, starting from the base position of the robotic arm, the system uses a breadth-first search algorithm to explore the reachable spatial areas of the robotic arm. During the search, the system considers the kinematic constraints of the robotic arm, including joint angle limitations and joint velocity limitations. For the left and right robotic arms, their respective reachable spaces are calculated, and their intersection area is marked as the cooperative area. The reachability analysis results are stored in the form of a three-dimensional spatial mapping, where each grid cell is labeled with a reachability flag: 0 indicates unreachable, 1 indicates the left robotic arm is reachable, 2 indicates the right robotic arm is reachable, and 3 indicates both arms are reachable. In practical applications, the system pre-calculates the workspace coverage of the robotic arm under different joint configurations, forming a lookup table, which greatly improves the efficiency of reachability analysis, reducing the computation time from 350 milliseconds to 42 milliseconds.

[0115] When screening the feasibility of grasping posture sequences based on reachability regions, the system checks whether the end effector position of each candidate grasping posture is within the reachable region. The screening process employs inverse kinematics to attempt to find a reasonable set of joint angle configurations for each grasping posture. In practice, the system does not check all grasping postures, but rather checks them in descending order of grasping quality score. The search stops once 10 feasible grasping postures are found, thus improving computational efficiency while ensuring grasping quality. For a dual-arm system, the reachability analysis of the left and right arms is performed in parallel, with each arm maintaining its own list of feasible grasping postures. In experimental verification, an average of 65% of candidate grasping postures were eliminated, significantly reducing the computational burden of subsequent path planning.

[0116] Based on the screened grasping posture sequence, the execution trajectory of the dual robotic arms is generated. The trajectory generation employs a variant of the sampling tree fast exploration algorithm, dividing the joint space into multiple regions and prioritizing sampling in regions closer to the target. In specific implementation, the system divides trajectory planning into three stages: a preparation stage, an approach stage, and a grasping stage. In the preparation stage, the robotic arm moves from its initial position to a preparation position near the target object; in the approach stage, the robotic arm approaches the target object from the preparation position along the planned path; in the grasping stage, the robotic arm precisely moves to the grasping position and performs the grasping action. During trajectory generation, the system calculates the robotic arm's velocity and acceleration at each sampling point to ensure they do not exceed preset limits. For a seven-DOF robotic arm, the joint velocity limit is set to 2.5 radians / second, and the acceleration limit is set to 1.5 radians / second². Regarding torque constraints, the system calculates the joint torque at each sampling point based on the robotic arm's dynamic model to ensure it does not exceed the rated value.

[0117] In collaborative dual-arm scenarios, a collaborative safety zone constraint must also be added. To prevent collisions between the two arms, the system defines a safety distance threshold of 0.12 meters, meaning that the minimum distance between any part of the two arms at any given time must not be less than this threshold. During trajectory planning, the system uses a spherical approximation model to represent the geometry of the arms. Each arm's links are composed of multiple spheres, and potential collisions are detected by calculating the distances between these spheres. When a potential collision risk is detected, the system adjusts the trajectory points to maintain a safe distance between the two arms. In practical applications, the system utilizes GPU parallel computing technology to accelerate the collision detection process, keeping the detection time within 5 milliseconds.

[0118] When controlling the dual robotic arms to perform a picking operation according to the execution trajectory, a position-force hybrid control strategy is adopted. Position control mode is used before approaching the object; after contact with the object, it switches to impedance control mode to adapt to changes in the object's actual position and shape. During execution, the system monitors the actual position and force of the robotic arms in real time through joint encoders and force / torque sensors, calculating the deviation between the actual trajectory and the planned trajectory. Deviation calculation is based on the position and attitude of the end effector. When the position deviation exceeds 15 mm or the attitude deviation exceeds 0.1 radians, the system determines that the deviation exceeds a preset threshold. At this point, near-field image data is re-acquired, and the environmental model and execution trajectory are updated. Experiments show that this adaptive adjustment mechanism can effectively cope with environmental changes and positioning errors, increasing the success rate from 78% to 95%, significantly improving the system's robustness.

[0119] The method further includes:

[0120] An external depth camera is mounted above or to the side of the workspace, acquiring depth data using structured light or time-of-flight principles. It outputs a 16-bit depth image of 640×480 pixels, with a depth measurement range of 0.3 meters to 3.0 meters and millimeter-level accuracy. Each of the two robotic arms is equipped with a wrist camera using a color CMOS sensor with a resolution of 1280×720 pixels, a 60-degree field of view, and a fixed focal length of 6 mm. These cameras are mounted near the end effectors of each arm, approximately 15 cm from the gripping point. The depth and wrist cameras are connected to the computing unit via USB 3.0 or Ethernet interfaces, with a data acquisition frequency set at 10 frames per second to ensure real-time performance and data synchronization.

[0121] The language task description is received via a text input interface, supporting natural language formats such as "grab the red circular object and place it in the specified location" or "remove obstructing items before picking up the target goods." The language parsing module uses a pre-trained natural language processing model to extract key constraints, including semantic information such as color attributes, shape features, size range, location description, and operation order. Constraints are stored in structured data format, containing fields such as attribute name, constraint value, weight coefficient, and priority identifier. The weighted directed task graph is represented by an adjacency matrix, where nodes represent subtasks or operation steps, and edge weights reflect the dependencies and execution order between tasks. Weight values ​​range from 0.1 to 1.0, with larger values ​​indicating stronger dependencies.

[0122] The geometric projection transformation module first completes camera calibration, obtaining the intrinsic parameter matrix and distortion coefficients. The extrinsic parameter matrix is ​​estimated using a checkerboard or feature point matching method. Depth data is transformed from the camera coordinate system to the world coordinate system using a homogeneous coordinate transformation with a 4×4 real matrix. Color images acquired by the wrist camera are simultaneously transformed to the same world coordinate system, ensuring precise spatial correspondence between depth point clouds and color pixels. Coordinate alignment errors are controlled within 2 millimeters, finely adjusted using an iterative nearest-point algorithm.

[0123] A cross-modal attention mechanism fuses geometric features from depth data and semantic features from near-field images. Geometric features extracted from depth data include surface normals, curvature information, edge contours, and local smoothness, with the feature vector dimension set to 128. Semantic features extracted from near-field images include texture descriptors, color histograms, edge responses, and corner detection results, also encoded as a 128-dimensional feature vector. Attention weights are calculated using dot-product attention; the query vector comes from semantic features, and the key-value pairs come from geometric features. Attention scores are normalized using a softmax function. The fused feature is represented as a weighted linear combination of geometric and semantic features, with weight coefficients determined by the attention scores, ultimately outputting a 256-dimensional fused feature vector.

[0124] Candidate point extraction is based on dense sampling using fused feature representations, generating candidate locations within the workspace at a 2mm grid spacing. A grasping feasibility score is calculated for each candidate point, comprehensively considering factors such as the angle between the surface normal vector and the grasper's opening / closing direction, local surface smoothness, distance to surrounding obstacles, and distance to the target object's boundary. The confidence score is calculated using a weighted summation method, with the weights of each factor determined based on historical success rate statistics: normal vector alignment weight 0.4, surface smoothness weight 0.3, obstacle distance weight 0.2, and boundary distance weight 0.1. The confidence field is stored in a 3D grid format, with each grid cell recording the confidence score at the corresponding location, ranging from 0 to 1, and the resolution consistent with the candidate point sampling spacing.

[0125] A preset gating threshold is set to 0.7. When a candidate point in the confidence field has a confidence score lower than this threshold, an adaptive re-observation process is triggered. The control module calculates the corresponding robotic arm movement path, bringing the wrist camera closer to the low-confidence area. The movement distance is typically 5 to 10 centimeters to ensure higher resolution near-field images. The re-acquired near-field image data undergoes the same feature extraction and fusion process to update the fused feature representation of the corresponding region. The confidence field uses an incremental update method, recalculating only the confidence scores of the affected areas while keeping the values ​​of other areas unchanged to reduce computational overhead. The update process uses a moving average method to fuse the old and new confidence values, with a smoothing coefficient set to 0.3 to avoid frequent fluctuations.

[0126] The vision-language-action model employs an end-to-end neural network architecture. Inputs include an adjacency matrix representation of a weighted directed task graph, a sequence of feature vectors representing fused feature representations, and a 3D array of confidence fields. The task graph is encoded using a graph neural network, with node features at 64 dimensions and edge features at 32 dimensions, and then processed through three layers of graph convolution to generate task representation vectors. The fused feature representations are processed using a convolutional neural network, employing a ResNet50 backbone to extract high-level semantic features. The confidence field serves as the spatial constraint input, encoded into spatial constraint vectors through a 3D convolutional network. These three representation vectors are fused using a multi-head attention mechanism to generate a unified decision feature vector with a dimension of 512.

[0127] The dual-arm task allocation scheme is output through a classification network, including three branches: left-arm task allocation, right-arm task allocation, and collaboration mode selection. Left and right arm task allocation uses multi-label classification, with candidate tasks including support actions, object retrieval actions, obstacle avoidance actions, and standby states, each task corresponding to a probability value. Collaboration mode selection includes sequential execution, parallel execution, alternating execution, and master-slave collaboration modes, outputting probability distributions through a softmax function. The grasping posture sequence is output through a regression network, including parameters such as the 3D coordinates of the grasping position, the rotation matrix of the grasping direction, the grasper opening and closing angle, and the contact force magnitude. Position coordinate accuracy is controlled at the millimeter level, rotation angle accuracy at the degree level, and force control accuracy at the Newton level.

[0128] Synchronization relationships are determined through the timing planning module, which generates a timestamp sequence based on the dependencies in the task graph and the robotic arm's motion constraints. Each action is assigned time parameters such as start time, duration, and end time, with a time resolution set to 100 milliseconds. A time synchronization constraint is established between the support action and the object-retrieving action to ensure that the object-retrieving operation begins only after the support is in place. The handover action requires both arms to reach the handover position within a specified time window, with a time window tolerance set to 200 milliseconds.

[0129] A second aspect of the present invention provides a dual-robotic arm collaborative picking system based on multi-source vision cross-view fusion, comprising:

[0130] The first unit is used to acquire workspace depth data collected by an external depth camera and near-field image data collected by the wrist cameras of each of the two robotic arms; receive a language task description, parse the language task description to extract constraints, and generate a weighted directed task graph.

[0131] The second unit is used to align the depth data and the near-field image data to a unified coordinate system through geometric projection transformation, and to fuse the geometric features of the depth data and the semantic features of the near-field image data using a cross-modal attention mechanism to generate a fused feature representation.

[0132] The third unit is used to extract grasping candidate points based on the fusion feature representation and calculate confidence scores to form a confidence field; when there are grasping candidate points in the confidence field with confidence scores lower than a preset gate threshold, the corresponding robotic arm is controlled to approach the grasping candidate point and re-acquire near-field image data to update the fusion feature representation and the confidence field.

[0133] The fourth unit is used to input the weighted directed task graph, the fused feature representation and the confidence field into the vision-language-action model. The vision-language-action model outputs the division of labor scheme and grasping posture sequence of the two robotic arms. The division of labor scheme specifies the support action or object picking action to be performed by each robotic arm and their synchronization relationship.

[0134] A third aspect of the present invention provides an electronic device, comprising:

[0135] processor;

[0136] Memory used to store processor-executable instructions;

[0137] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0138] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0139] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0140] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A collaborative picking method for dual robotic arms based on multi-source vision cross-view fusion, characterized in that, include: Acquire workspace depth data from an external depth camera and near-field image data from cameras on the wrists of both robotic arms; Receive a language task description, parse the language task description to extract constraints, and generate a weighted directed task graph; The depth data and the near-field image data are aligned to a unified coordinate system through geometric projection transformation, and a cross-modal attention mechanism is used to fuse the geometric features of the depth data and the semantic features of the near-field image data to generate a fused feature representation, including: Based on the intrinsic and extrinsic parameter matrices of the external depth camera, the depth data is transformed to the unified coordinate system to obtain aligned depth data; Based on the intrinsic and extrinsic parameter matrices of the wrist cameras of the two robotic arms, the near-field image data is transformed to the unified coordinate system to obtain aligned near-field image data. Geometric features are extracted from the alignment depth data, and semantic features are extracted from the alignment near-field image data; Construct a cross-modal attention matrix, wherein the cross-modal attention matrix represents the association weight between each feature element in the geometric feature and each feature element in the semantic feature; The geometric features and semantic features are weighted and fused based on the cross-modal attention matrix to generate the fused feature representation; Based on the fusion feature representation, candidate points are extracted and confidence scores are calculated to form a confidence field; When there is a grasping candidate point in the confidence field with a confidence score lower than a preset gating threshold, the corresponding robotic arm is controlled to approach the grasping candidate point and re-acquire near-field image data, updating the fused feature representation and the confidence field; The weighted directed task graph, the fused feature representation, and the confidence field are input into the vision-language-action model. The vision-language-action model outputs the division of labor scheme and grasping posture sequence of the two robotic arms. The division of labor scheme specifies the support action or object picking action to be performed by each robotic arm and their synchronization relationship. The method further includes: Based on the depth data, a spatial occupancy grid is constructed, and the distribution of obstacles and accessibility areas are determined according to the spatial occupancy grid. Based on the reachability area, the feasibility of the grasping posture sequence is screened, and unreachable grasping postures are eliminated. The execution trajectory of the dual robotic arms is generated based on the screened grasping posture sequence, and speed constraints, torque constraints and cooperative safety area constraints are superimposed on the execution trajectory. The dual robotic arms are controlled to perform a picking operation according to the execution trajectory. During the execution, the deviation between the actual trajectory and the execution trajectory is monitored. When the deviation exceeds a preset deviation threshold, the near-field image data acquisition is re-executed and the execution trajectory is updated.

2. The method according to claim 1, characterized in that, Parsing the language task description to extract constraints and generating a weighted directed task graph includes: The language task description is semantically parsed to identify the attribute information of the pickup object, including color constraints, size constraints, category constraints, placement constraints, and order constraints. A set of task nodes is constructed based on the attribute information, and each task node corresponds to a pickup sub-task. Based on the order constraints, directed edges are determined between each task node in the task node set, and the directed edges represent the sequential dependencies of task execution. Assign a weight to each of the directed edges, where the weight represents the priority of the corresponding task dependency; The weighted directed task graph is formed by combining the set of task nodes, the directed edges, and the weights.

3. The method according to claim 1, characterized in that, The weighted directed task graph, the fused feature representation, and the confidence field are input into the vision-language-action model. The vision-language-action model outputs a division of labor scheme and grasping posture sequence for the two robotic arms, including: The weighted directed task graph is encoded as a task embedding vector, the fused feature representation is encoded as a visual embedding vector, and the confidence field is encoded as a confidence embedding vector. The task embedding vector, the visual embedding vector, and the confidence embedding vector are input into the encoder of the vision-language-action model to generate a joint representation; Based on the joint representation, an action sequence is generated by the decoder of the vision-language-action model, the action sequence including the action of the left robotic arm and the action of the right robotic arm; The action sequence is decomposed into the division of labor scheme according to the executing subject of each action in the action sequence. The division of labor scheme specifies that the action of the left robotic arm is a supporting action and the action of the right robotic arm is a picking action, or specifies that the action of the left robotic arm is a picking action and the action of the right robotic arm is a supporting action. Extract the grab position and grab posture corresponding to each action from the action sequence to form the grab posture sequence.

4. The method according to claim 3, characterized in that, Generating action sequences based on the joint representation using the decoder of the vision-language-action model includes: The joint representation is input into the attention layer of the decoder to calculate the cross-attention weights between the task embedding vector, the visual embedding vector, and the confidence embedding vector. The joint representation is weighted and aggregated according to the cross-attention weights to generate a context representation; The context representation is input into the feedforward network of the decoder to generate the action sequence step by step. Each action in the action sequence includes an execution subject identifier, an action type identifier, a grasping position coordinate, and a grasping posture parameter.

5. A dual-robotic arm collaborative picking system based on multi-source vision cross-view fusion, used to implement the method as described in any one of claims 1-4, characterized in that, include: The first unit is used to acquire workspace depth data collected by an external depth camera and near-field image data collected by the wrist cameras of each of the two robotic arms. Receive a language task description, parse the language task description to extract constraints, and generate a weighted directed task graph; The second unit is used to align the depth data and the near-field image data to a unified coordinate system through geometric projection transformation, and to fuse the geometric features of the depth data and the semantic features of the near-field image data using a cross-modal attention mechanism to generate a fused feature representation. The third unit is used to extract candidate points based on the fused feature representation and calculate confidence scores to form a confidence field; When there is a grasping candidate point in the confidence field with a confidence score lower than a preset gating threshold, the corresponding robotic arm is controlled to approach the grasping candidate point and re-acquire near-field image data, updating the fused feature representation and the confidence field; The fourth unit is used to input the weighted directed task graph, the fused feature representation and the confidence field into the vision-language-action model. The vision-language-action model outputs the division of labor scheme and grasping posture sequence of the two robotic arms. The division of labor scheme specifies the support action or object picking action to be performed by each robotic arm and their synchronization relationship.

6. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 4.

Images