A dual-arm cooperative operation high-level behavior topology construction method and system

Abstract

The application discloses a kind of dual-arm collaborative operation high-level behavior topology construction method and system, belong to robot control technical field, this method includes: obtaining multi-modal dual-arm body demonstration data, and the semantic features corresponding to left and right hand are extracted;Respectively construct left and right hand action semantic rule model;Build dual-hand collaborative model based on mixed master-slave relationship diagram, to infer multi-level collaborative mode and output corresponding dual-hand operation collaborative constraint;Based on dual-hand operation collaborative constraint, left and right hand action semantic rule model is integrated, and the priori knowledge of large language model is introduced, generates the task semantic rule generalization strategy for new scene, and generates robot dual-arm task state transition topology atlas accordingly.The application effectively overcomes the dimension disaster caused by dual-arm collaborative action, greatly improves the cognitive inference and cross-scene generalization ability of robot in complex long sequence operation.

Description

Technical Field

[0001] This invention relates to the fields of embodied intelligence, robot control and artificial intelligence, and in particular to a method and system for constructing high-level behavioral topologies through dual-arm collaborative operation. Background Technology

[0002] In recent years, embodied autonomous learning technology for dual-arm robots has received considerable attention. It aims to achieve autonomous operation of complex service tasks using data-driven modern artificial intelligence methods such as deep reinforcement learning and imitation learning. This approach leverages the superior mobility and flexibility of dual-arm systems compared to single-arm systems, fully utilizing their adaptability to complex tasks to comprehensively address many challenges currently facing dual-arm robots in their journey towards general intelligence. The continuous expansion of robot applications places increasingly higher demands on the generalization efficiency of embodied learning algorithms. Only with the ability to cognitively reason about complex tasks can dual-arm robots quickly and autonomously adapt to different open service domains.

[0003] Current autonomous learning technologies for dual-arm robots involve agents jointly understanding their own action boundaries and environmental state information. However, due to the lack of prior knowledge such as high-level task behavior rules, traditional end-to-end dual-arm learning strategies often neglect the reasoning behind the internal logic of long, complex task sequences, failing to meet the operational requirements of intelligent and dynamic service scenarios. Therefore, embodied autonomous learning for long-sequence collaborative operations is being extensively researched. Long-sequence dual-arm collaborative operations refer to operations with a more complex policy space topology and more agile hand-hand coordination decision-making capabilities compared to ordinary single, fixed operations. Multiple agents (left and right arms) must rapidly acquire the real-time state space based on the changing, dynamic, and open service environment to formulate globally optimal action coordination strategies, avoiding self-collisions in arm movements and task logic failures. Long-sequence collaborative operations have an exponentially expanded joint state-action space compared to the execution of ordinary single actions. Designing a reasonable dual-arm robot autonomous learning fusion strategy, the efficiency of structured modeling of task knowledge, and efficient dimensionality reduction strategies for hand-hand coordination are crucial issues.

[0004] However, most traditional data-driven models currently have weak cognitive reasoning capabilities at the task level, and the acquired action strategies lack logical interpretability, which limits the generalization accuracy of dual-arm collaborative operation models in new environments. Consequently, this significantly reduces the efficiency and success rate of dual-arm robots in completing complex cross-scene tasks in real-world application scenarios. Summary of the Invention

[0005] This invention provides a method and system for constructing high-level behavioral topology for dual-arm cooperative operation, in order to solve the technical problem of weak cognitive reasoning ability in the autonomous learning of long-sequence cooperative operation skills of existing dual-arm robots.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: On one hand, the present invention provides a method for constructing high-level behavioral topology for dual-arm cooperative operation, applicable to dual-arm robots, the method comprising: Acquire multimodal dual-arm embodied demonstration data of the robot, and extract semantic features corresponding to the robot's left hand and right hand based on the multimodal dual-arm embodied demonstration data; Action semantic recognition is performed on the semantic features corresponding to the robot's left hand and the robot's right hand respectively, thereby constructing a left-hand action semantic rule model and a right-hand action semantic rule model; Based on the semantic rule model of left-hand action and the semantic rule model of right-hand action, a two-hand collaboration model based on a hybrid master-slave relationship graph is constructed to infer multi-level collaboration patterns and output corresponding two-hand operation collaboration constraints. Based on the co-operation constraints of both hands, the semantic rule model of the left hand action and the semantic rule model of the right hand action are integrated, and the prior knowledge of the large language model is introduced to generate a generalization strategy of task semantic rules for new scenarios. Based on the task semantic rule generalization strategy, a topology map of the robot's dual-arm task state transition is generated.

[0007] Furthermore, the multimodal dual-arm embodied demonstration data includes left-hand demonstration data and right-hand demonstration data; the left-hand demonstration data and right-hand demonstration data respectively include image sequences acquired by the robot's vision sensor, six-dimensional force / torque of the end effector acquired by the force sensor, and joint angles acquired by the joint encoder.

[0008] Furthermore, the semantic features are quadruple semantic features, including operation object features, action primitive features, state change features, and spatial relationship features; wherein, The characteristics of the operation object include the object's category, geometric center coordinates, bounding box size, and physical properties; The action primitive features are represented by a predefined dictionary of action primitives to represent the actions to be performed. The state change characteristics characterize the environmental state transitions caused by the operation; The spatial relationship features characterize the spatial relative relationship between the end effector of the robotic arm and the environment or the object being operated.

[0009] Furthermore, action semantic recognition is performed on the semantic features corresponding to the robot's left hand and the robot's right hand respectively, thereby constructing a left-hand action semantic rule model and a right-hand action semantic rule model, including: The semantic features to be processed are constructed into an attribute graph, which is then input into a graph neural network. The graph neural network performs message passing and aggregation of node features to obtain graph feature representation. The graph feature representation is input into an ensemble learning model composed of multiple decision trees. By weighted fusion of the outputs of each decision tree, the action semantic rule recognition result of the corresponding robotic arm operation is generated.

[0010] Furthermore, based on the left-hand action semantic rule model and the right-hand action semantic rule model, a two-handed collaboration model based on a hybrid master-slave relationship graph is constructed to infer multi-level collaboration patterns and output corresponding two-handed operation collaboration constraints, including: Within a preset time window, based on the left-hand action semantic rule model, the semantic features corresponding to the robot's left hand are used to perform action semantic recognition, resulting in a left-hand temporal feature state vector; based on the right-hand action semantic rule model, the semantic features corresponding to the robot's right hand are used to perform action semantic recognition, resulting in a right-hand temporal feature state vector. Based on the temporal feature state vectors of the left and right hands, the cross-attention features related to the spatial and kinematic relationships of the left and right hands are calculated. The left-hand temporal feature state vector, the right-hand temporal feature state vector, and the cross-attention feature are concatenated. The concatenated vector is then input into a fully connected network. The network is used to infer the probability distribution of the cooperative state at the current moment, thereby determining the current cooperative mode.

[0011] Furthermore, based on the left-hand temporal feature state vector and the right-hand temporal feature state vector, the formula for calculating the cross-attention features related to the spatial and kinematic correlation between the left and right hands is as follows: ; in, Cross-attention, representing the spatial and kinematic correlation between the left and right hands; This represents the query matrix generated by linear mapping from the left-handed temporal feature state vector; and These represent the key matrix and value matrix, respectively, generated by linear mapping from the right-handed temporal characteristic state vector; for The transpose of the matrix; This represents the feature dimension of the key matrix in the attention mechanism; Softmax This represents the normalized exponential function.

[0012] Furthermore, the multi-level collaborative modes include non-coordinated mode, loosely coupled mode, and tightly symmetric mode; The non-coordinated mode corresponds to the anti-collision boundary constraint where the two hands do not interfere with each other. The loosely coupled mode corresponds to a master-slave following constraint with sequential timing. The tightly symmetrical mode corresponds to rigid or flexible cooperative transport constraints with relative orientation.

[0013] Furthermore, based on the co-operation constraints of both hands, the semantic rule model of the left hand action and the semantic rule model of the right hand action are integrated, and prior knowledge from the large language model is introduced to generate a task semantic rule generalization strategy for new scenarios, including: Under the aforementioned two-hand operation coordination constraint, the semantic rule model of the left hand action and the semantic rule model of the right hand action are integrated to generate a joint state space description; The new task instruction, the current scene context, and the joint state space description are concatenated and encoded together into a text prompt word sequence; The text prompt word sequence is input into a large language model, and the large language model is used to output a semantic rule sequence conditional probability distribution adapted to the current new scenario; A task semantic rule generalization strategy is generated by maximizing the conditional probability distribution of the semantic rule sequence.

[0014] Furthermore, the topological map is used for the representation and reasoning of knowledge related to the operational tasks of a dual-arm robot.

[0015] On the other hand, the present invention also provides a dual-arm cooperative high-level behavioral topology construction system, applicable to dual-arm robots, the system comprising: The semantic feature extraction module is used to acquire multimodal dual-arm embodied demonstration data of the robot, and extract the semantic features corresponding to the robot's left hand and the robot's right hand based on the multimodal dual-arm embodied demonstration data. The action semantic modeling module is used to perform action semantic recognition on the semantic features corresponding to the robot's left hand and the semantic features corresponding to the robot's right hand, thereby constructing a left-hand action semantic rule model and a right-hand action semantic rule model. The two-handed collaborative modeling module is used to construct a two-handed collaborative model based on a hybrid master-slave relationship graph, based on the left-hand action semantic rule model and the right-hand action semantic rule model, in order to infer multi-level collaborative patterns and output corresponding two-handed operation collaborative constraints. The rule integration and generalization module is used to integrate the left-hand action semantic rule model and the right-hand action semantic rule model based on the two-hand operation coordination constraint, and introduce the prior knowledge of the large language model to generate a task semantic rule generalization strategy for new scenarios. The topology graph generation module is used to generate a topology graph of the robot's dual-arm task state transition based on the task semantic rule generalization strategy.

[0016] In another aspect, the present invention also provides an electronic device comprising a processor and a memory; wherein the memory stores at least one instruction, which is loaded and executed by the processor to implement the above-described method.

[0017] In another aspect, the present invention also provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the above method.

[0018] The beneficial effects of the technical solution provided by this invention include at least the following: 1. By innovatively constructing a two-arm collaborative model based on a hybrid master-slave relationship graph, this invention can accurately infer multi-level collaborative modes such as "non-coordinated, loosely coupled, and tightly symmetrical". This structured dimensionality reduction and decoupling mechanism transforms complex two-arm constraints into clear spatiotemporal physical boundaries, significantly reducing the learning difficulty.

[0019] 2. This invention proposes to integrate Graph Neural Decision Tree (GNN-DT) with ensemble learning for application in robot variable-granularity action semantic recognition. This architecture not only fully leverages the powerful representational capabilities of graph neural networks in processing high-dimensional spatial topological features, but also inherits the transparency and high interpretability of decision trees and ensemble learning in rule-based reasoning.

[0020] 3. This invention sequentially integrates the rich prior knowledge of the open world inherent in the Large Language Model (LLM) with specific dual-arm operation collaborative constraints to generate a dynamic task semantic rule generalization strategy. The generated dual-arm state transition topology graph inherits the powerful logical generalization and thought chain reasoning capabilities of LLM, while also being subject to strict dual-arm kinematic body-involvement constraints. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is an overall flowchart of the dual-arm collaborative operation high-level behavior topology construction method provided in this embodiment of the invention; Figure 2 This is a schematic diagram of a dual-arm multi-level collaborative reasoning mode based on a hybrid master-slave relationship diagram provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the dual-arm task state transition topology generated in an embodiment of the present invention; Figure 4 This is a system block diagram of the electronic device provided in the embodiments of the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0024] First, it should be noted that in the embodiments of the present invention, the words "exemplarily," "for example," etc., are used to indicate that they are examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the term "exemplarily" is intended to present the concept in a specific manner. Furthermore, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either one or the other.

[0025] First Embodiment

[0026] This embodiment provides a method for constructing high-level behavioral topology for dual-arm cooperative operation, applicable to dual-arm robots. Its implementation framework is as follows: Figure 1 As shown. This method can be implemented by an electronic device, which can be a terminal or a server. Specifically, the execution flow of this method includes the following steps: S1, acquire multimodal dual-arm embodied demonstration data of the robot, and extract semantic features corresponding to the robot's left hand and the robot's right hand based on the multimodal dual-arm embodied demonstration data; Specifically, this embodiment uses the visual sensors (such as an RGB-D camera), force sensors, and joint encoders of the dual-arm robot to collect multimodal embodied demonstration data (such as image sequences, six-dimensional force / torque at the end effector, joint angles, etc.). For the decoupling of the two arms, independent four-tuple semantic features of the left and right arms are extracted separately, denoted as:

[0027] Characteristics of the operation object This includes the object's category, 3D coordinates of its geometric center, bounding box dimensions, and physical properties (such as predicted mass and stiffness).

[0028] Action primitive features : Use a predefined dictionary of action primitives, i.e. .

[0029] State change characteristics The environmental state transition resulting from the representation operation. Specifically, the change in object pose calculated through visual feedback. .

[0030] Spatial Relationship Characteristics Characterizing the end effector of a robotic arm With the target object or environment The spatial relative relationship between them. Specifically, it is represented by a combination of distance and attitude difference:

[0031] in, and Let represent the three-dimensional position vectors of the end effector and the target object in the base coordinate system, respectively. ; The L2 norm of a vector is represented, which is the Euclidean distance. and These represent the unit quaternions that characterize the three-dimensional spatial pose of the end effector and the target object, respectively. For the target object's pose quaternion The inverse quaternion; This represents a quaternion multiplication operator used to calculate relative attitude difference.

[0032] S2, perform action semantic recognition on the semantic features corresponding to the robot's left hand and the semantic features corresponding to the robot's right hand respectively, so as to construct the left hand action semantic rule model and the right hand action semantic rule model; Specifically, this embodiment uses graph neural decision trees and ensemble learning algorithms to perform variable-granularity action semantic recognition on the semantic features of left-hand independent quadruples and right-hand independent quadruples, and constructs left-hand action semantic rule models and right-hand action semantic rule models respectively. The specific process is as follows: Graph Neural Decision Tree (GNDT) models are used to perform variable-granularity action semantic recognition on the four-tuple semantic features of the left and right hands respectively: Attribute graph construction: Transforming quadruple features into attribute graphs Among them, nodes Represents an end effector, the object being manipulated, or an environmental obstacle. (Side) Represents the interaction relationships between nodes (i.e., spatial relationship features) ).

[0033] Graph Node Feature Aggregation: Graph neural networks (GNNs) are used for message passing and aggregation of node features to obtain graph feature representations. Among them, the first... Layer nodes Features The updated formula is:

[0034] in, This is the index of the hidden layer number in a graph neural network; and Representing nodes respectively In the With the The feature vector of the layer; Represents a node The set of first-order neighbor nodes; AGG represents the feature aggregation function in a graph neural network; Representing the graph network The learnable weight matrix of the layer; It is a non-linear activation function.

[0035] Ensemble decision tree inference: After multiple layers of propagation and global pooling, a graph-level feature representation is obtained. Enter it into the... In a random forest composed of many differentiable decision trees, each tree... Probability distribution of independent output action semantics Finally, the action recognition result is output by weighted fusion of the outputs of each decision tree. :

[0036] in, These are classification labels for action semantics; For the first The confidence weight coefficients of decision trees in weighted fusion; This represents the optimal action semantic recognition result that the model ultimately infers and outputs.

[0037] Based on this, semantic rule models for left-hand and right-hand actions are constructed respectively.

[0038] S3, based on the left-hand action semantic rule model and the right-hand action semantic rule model, constructs a two-hand collaboration model based on a hybrid master-slave relationship graph to infer multi-level collaboration modes and output corresponding two-hand operation collaboration constraints; Specifically, such as Figure 2 As shown, this embodiment, after clarifying the semantics of single-arm movements, further infers the multi-level coordination patterns between the two arms. The multi-level coordination pattern classification space is defined as follows: These correspond to the non-coordinated mode (collision avoidance), loosely coupled mode (master-slave timing follow), and tightly symmetric mode (rigid / flexible position constraints), respectively.

[0039] Feature extraction: extraction time window The left-hand and right-hand temporal characteristic states within are denoted as... and .

[0040] in, and These represent the left and right hands within the time window, respectively. The sequence of temporal characteristic state vectors within.

[0041] Cross-attention feature calculation: Calculate the spatial and kinematic correlation between the left and right hands, and introduce a multi-head cross-attention mechanism:

[0042] in, For left-hand characteristic state The query matrix generated by linear mapping; and These are respectively the right-hand characteristic states The key matrix and value matrix generated by linear mapping; Key matrix The transpose of the matrix; is the feature dimension of the key matrix in the attention mechanism, used to prevent gradient vanishing scaling; Softmax is the normalized exponential function.

[0043] This step outputs cross-attention features. .in, The output is the spatiotemporal cross-attention associated feature vector of the left and right hands.

[0044] Cooperative pattern inference: The input is concatenated into a fully connected network, and the probability distribution of the current cooperative mode is output. This determines the current multi-level cooperative mode, extracts the corresponding constraints, and generates a constraint vector. .

[0045] in, This is used to infer the generated coordinating constraint vector for the two-handed operation to guide the underlying execution. The non-coordinated mode corresponds to the collision-avoidance boundary constraint that the two hands do not interfere with each other; the loosely coupled mode corresponds to the master-slave follow constraint with sequential timing; and the tightly symmetric mode corresponds to the rigid or flexible coordinating transport constraint with strict relative pose.

[0046] S4. Based on the co-operation constraints of both hands, the semantic rule model of the left hand action and the semantic rule model of the right hand action are sequentially integrated, and the prior knowledge of the large language model is introduced to generate a task semantic rule generalization strategy for new scenarios. Specifically, in order to enable operations to be performed even in completely new and unseen task scenarios (new objects, new instructions), this embodiment introduces a large language model for zero-shot generalization inference. The specific process is as follows: Cue word encoding: This involves encoding the left and right hand action semantic models obtained from the above steps, along with collaborative constraints. Sequential integration is performed to form a "joint state space description text." This text is then concatenated with the natural language instructions and scene text information of the new task to form a sequence of prompt words. .

[0047] Probability Distribution Output and Maximization: Reception in Large Language Models Then, based on its inherent physical common sense and logical reasoning ability, it outputs the next action node sequence adapted to the new scenario. The conditional probability distribution is determined. The optimal task semantic rule generalization strategy is generated by maximizing this conditional probability distribution using a greedy search or beam search algorithm. :

[0048] in, This represents a sequence of text prompts encoded from a status description, instructions, and context. The total number of steps in the generated action node sequence predicted for a large language model; The sequence of action nodes generated for the candidates; To generate the first in the sequence The specific semantic meaning of the step; Indicates that in generating the first Before the step action, all previously generated historical action sequences; This represents the conditional probability of the next action semantics output by the autoregressive model of a large language model given a sequence of cue words and historical context. Represents a sequence Multiply the generation probabilities of each time step together; This represents the optimal task semantic rule generalization strategy sequence obtained by maximizing the conditional probability distribution search output.

[0049] S5. Based on the task semantic rule generalization strategy, generate a topology map of the robot dual-arm task state transition. The robot dual-arm task state transition topology map generated in this embodiment is as follows: Figure 3 As shown. The topology map is used for the representation and reasoning of knowledge related to the operational tasks of a dual-arm robot.

[0050] Specifically, this embodiment is based on the optimal strategy output in the previous step. Generate a dual-arm task state transition topology map that can be directly read and executed by the robot's underlying controller. The details are as follows: Node set : Represents the coordinated state of both arms, such as: "Left hand grasping - right hand waiting".

[0051] edge set : Represents the triggering condition for state transition, such as environmental event triggering or time threshold triggering.

[0052] Loading this topology map into the main control unit of the dual-arm robot allows the robot to search using a map search algorithm, such as A. The algorithm, or Dijkstra's algorithm, evaluates its own state and interaction with the environment in real time in a dynamic environment, adaptively adjusts the distribution of behavior sequences, and ultimately completes the collaborative operation task safely and efficiently.

[0053] In summary, this embodiment provides a method for constructing high-level behavioral topologies through dual-arm cooperative operation, which offers the following specific technical advantages: 1. The action semantic recognition scheme based on graph neural decision tree and ensemble learning proposed in this embodiment has the advantages of fine feature representation and clear discrimination boundary, which enables dual-arm robot to sensitively and accurately recognize variable granularity actions in complex unstructured environments. It solves the problem of overfitting that traditional methods are prone to occur when fusing multimodal features, and is more suitable for complex collaborative operation scenarios with high flexibility and high dynamics.

[0054] 2. The dual-arm collaborative modeling strategy based on a hybrid master-slave relationship graph proposed in this embodiment can perform cross-attention calculation on the correlation between the temporal features and spatial kinematics of the two arms, accurately infer multi-level collaborative modes and adaptively generate corresponding physical constraints, providing strong logical support for the underlying safety collision avoidance and refined synchronous collaboration of the dual-arm system. It has the advantages of high collaborative matching degree, fast constraint generation and strong security. 3. The rule generalization and topology graph generation strategy proposed in this embodiment, which introduces prior knowledge of a large language model, deeply integrates the semantic model of dual-arm action, collaborative constraints and new task scenario context, and performs zero-shot reasoning. This effectively overcomes the bottleneck of traditional hard-coding rules being difficult to adapt to new objects. The generated state transition topology graph can directly guide the robot's underlying execution, greatly improving the autonomous migration capability and execution efficiency of the dual-arm robot in operation tasks facing unknown scenarios.

[0055] Second Embodiment

[0056] This embodiment provides a dual-arm collaborative high-level behavioral topology construction system, which includes the following modules: The semantic feature extraction module is used to acquire multimodal dual-arm embodied demonstration data of the robot, and extract the semantic features corresponding to the robot's left hand and the robot's right hand based on the multimodal dual-arm embodied demonstration data. The action semantic modeling module is used to perform action semantic recognition on the semantic features corresponding to the robot's left hand and the semantic features corresponding to the robot's right hand, thereby constructing a left-hand action semantic rule model and a right-hand action semantic rule model. The two-handed collaborative modeling module is used to construct a two-handed collaborative model based on a hybrid master-slave relationship graph, based on the left-hand action semantic rule model and the right-hand action semantic rule model, in order to infer multi-level collaborative patterns and output corresponding two-handed operation collaborative constraints. The rule integration and generalization module is used to integrate the left-hand action semantic rule model and the right-hand action semantic rule model based on the two-hand operation coordination constraint, and introduce the prior knowledge of the large language model to generate a task semantic rule generalization strategy for new scenarios. The topology graph generation module is used to generate a topology graph of the robot's dual-arm task state transition based on the task semantic rule generalization strategy.

[0057] It should be noted that the dual-arm collaborative operation high-level behavior topology construction system of this embodiment corresponds to the dual-arm collaborative operation high-level behavior topology construction method of the first embodiment described above; the functions implemented by each functional module in the dual-arm collaborative operation high-level behavior topology construction system of this embodiment correspond one-to-one with the process steps in the dual-arm collaborative operation high-level behavior topology construction method of the first embodiment described above; therefore, it will not be described again here.

[0058] Third Embodiment

[0059] This embodiment provides an electronic device, such as... Figure 4 As shown, the electronic device includes a processor and a memory; wherein the processor and the memory can be connected via a communication bus; the memory stores at least one instruction, which is loaded and executed by the processor to implement the method of the first embodiment described above. Furthermore, the electronic device may also include a transceiver, the processor and the transceiver can be connected via a communication bus, and the transceiver is used to communicate with other devices.

[0060] Below, in conjunction with Figure 4 A detailed introduction to each component of this electronic device is provided below: The processor is the control center of the electronic device. The electronic device may include multiple processors, each of which can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The term "processor" can refer to a single processor or a collective term for multiple processing elements. For example, a processor can be one or more central processing units (CPUs), other general-purpose processors, application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), one or more field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor can perform various functions of the electronic device by running or executing software programs stored in memory and by calling data stored in memory.

[0061] In a specific implementation, as one example, the processor may include one or more CPUs, for example... Figure 4 CPU0 and CPU1 shown are, of course, merely illustrative examples.

[0062] The memory is used to store the software program that executes the solution of the present invention, and the processor controls its execution. For specific implementation methods, please refer to the above method embodiments, which will not be repeated here.

[0063] Optionally, the memory may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory may be integrated with the processor or exist independently, and may be accessed through the interface circuit of the electronic device ( Figure 4 (Not shown in the image) is coupled to the processor; however, this embodiment of the invention does not impose specific limitations on this.

[0064] The transceiver may include a receiver and a transmitter. Figure 4 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function. The transceiver can be integrated with the processor or exist independently, and can be connected through the interface circuit of the electronic device (…). Figure 4 (Not shown in the image) is coupled to the processor, and this embodiment of the invention does not specifically limit this.

[0065] In addition, it should be noted that, Figure 4 The structure of the electronic device shown is not intended to limit the device. Actual devices may include more or fewer components than shown, or combine certain components, or have different component arrangements. Furthermore, the technical effects achieved by this electronic device when performing the method of the first embodiment described above can be referenced to the technical effects described in the first embodiment; therefore, they will not be repeated here.

[0066] Fourth embodiment

[0067] This embodiment provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the method of the first embodiment described above. The computer-readable storage medium may be a ROM, random access memory, CD-ROM, magnetic tape, floppy disk, or optical data storage device, etc. The instruction stored therein can be loaded and executed by a processor in a terminal.

[0068] Furthermore, it should be noted that the present invention can be provided as a method, apparatus, or computer program product. Therefore, embodiments of the present invention can take the form of a completely or partially hardware embodiment, a completely or partially software embodiment, or an embodiment combining software and hardware aspects. Moreover, when implemented in software, embodiments of the present invention can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any usable medium accessible to a computer or a data storage device such as a server or data center containing one or more sets of usable media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive (SSD).

[0069] Embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0070] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing terminal equipment to cause a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0071] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element. Furthermore, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone, where A and B can be singular or plural. Additionally, the character " / " in this text generally indicates an "or" relationship between the preceding and following objects, but it can also indicate an "AND / OR" relationship. Please refer to the context for specific interpretations. "At least one" refers to one or more items, while "more than" refers to two or more items. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can be represented as: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0072] Furthermore, it is understood that in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0073] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0074] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of functional modules / units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the shown or discussed mutual couplings or direct couplings or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms. Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, i.e., they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs. Additionally, the functional units in the various embodiments of this invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0075] If the method is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0076] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention. It should be pointed out that although preferred embodiments of the present invention have been described, those skilled in the art, once they understand the basic inventive concept of the present invention, can make several improvements and modifications without departing from the principles described herein. These improvements and modifications should also be considered within the scope of protection of the present invention. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

Claims

1. A method for constructing high-level behavioral topology for dual-arm cooperative operation, applicable to dual-arm robots, characterized in that, The method for constructing high-level behavioral topology for dual-arm collaborative operation includes: Acquire multimodal dual-arm embodied demonstration data of the robot, and extract semantic features corresponding to the robot's left hand and right hand based on the multimodal dual-arm embodied demonstration data; Action semantic recognition is performed on the semantic features corresponding to the robot's left hand and the robot's right hand respectively, thereby constructing a left-hand action semantic rule model and a right-hand action semantic rule model; Based on the semantic rule model of left-hand action and the semantic rule model of right-hand action, a two-hand collaboration model based on a hybrid master-slave relationship graph is constructed to infer multi-level collaboration patterns and output corresponding two-hand operation collaboration constraints. Based on the co-operation constraints of both hands, the semantic rule model of the left hand action and the semantic rule model of the right hand action are integrated, and the prior knowledge of the large language model is introduced to generate a generalization strategy of task semantic rules for new scenarios. Based on the task semantic rule generalization strategy, a topology map of the robot's dual-arm task state transition is generated.

2. The method for constructing high-level behavioral topology for dual-arm cooperative operation as described in claim 1, characterized in that, The multimodal dual-arm embodied demonstration data includes left-hand demonstration data and right-hand demonstration data; the left-hand demonstration data and the right-hand demonstration data respectively include image sequences acquired by the robot's vision sensor, six-dimensional force / torque of the end effector acquired by the force sensor, and joint angle acquired by the joint encoder.

3. The method for constructing high-level behavioral topology for dual-arm cooperative operation as described in claim 1, characterized in that, The semantic features are quadruple semantic features, including operation object features, action primitive features, state change features, and spatial relationship features; among them, The characteristics of the operation object include the object's category, geometric center coordinates, bounding box size, and physical properties; The action primitive features are represented by a predefined dictionary of action primitives to represent the actions to be performed. The state change characteristics characterize the environmental state transitions caused by the operation; The spatial relationship features characterize the spatial relative relationship between the end effector of the robotic arm and the environment or the object being operated.

4. The method for constructing high-level behavioral topology for dual-arm cooperative operation as described in claim 1, characterized in that, Action semantic recognition is performed on the semantic features corresponding to the robot's left hand and right hand respectively, thereby constructing a left-hand action semantic rule model and a right-hand action semantic rule model, including: The semantic features to be processed are constructed into an attribute graph, which is then input into a graph neural network. The graph neural network performs message passing and aggregation of node features to obtain graph feature representation. The graph feature representation is input into an ensemble learning model composed of multiple decision trees. By weighted fusion of the outputs of each decision tree, the action semantic rule recognition result of the corresponding robotic arm operation is generated.

5. The method for constructing high-level behavioral topology for dual-arm cooperative operation as described in claim 1, characterized in that, Based on the semantic rule model of left-hand action and the semantic rule model of right-hand action, a two-handed collaboration model based on a hybrid master-slave relationship graph is constructed to infer multi-level collaboration patterns and output corresponding two-handed operation collaboration constraints, including: Within a preset time window, based on the left-hand action semantic rule model, the semantic features corresponding to the robot's left hand are used to perform action semantic recognition, resulting in a left-hand temporal feature state vector; based on the right-hand action semantic rule model, the semantic features corresponding to the robot's right hand are used to perform action semantic recognition, resulting in a right-hand temporal feature state vector. Based on the temporal feature state vectors of the left and right hands, the cross-attention features related to the spatial and kinematic relationships of the left and right hands are calculated. The left-hand temporal feature state vector, the right-hand temporal feature state vector, and the cross-attention feature are concatenated. The concatenated vector is then input into a fully connected network. The network is used to infer the probability distribution of the cooperative state at the current moment, thereby determining the current cooperative mode.

6. The method for constructing high-level behavioral topology for dual-arm cooperative operation as described in claim 5, characterized in that, Based on the left-hand temporal feature state vector and the right-hand temporal feature state vector, the formula for calculating the cross-attention feature of spatial and kinematic correlation between the left and right hands is as follows: ； in, Cross-attention, representing the spatial and kinematic correlation between the left and right hands; This represents the query matrix generated by linear mapping from the left-handed temporal feature state vector; and These represent the key matrix and value matrix, respectively, generated by linear mapping from the right-handed temporal characteristic state vector; for The transpose of the matrix; This represents the feature dimension of the key matrix in the attention mechanism; Softmax This represents the normalized exponential function.

7. The method for constructing high-level behavioral topology for dual-arm cooperative operation as described in claim 1, characterized in that, The multi-level collaborative modes include non-coordinated mode, loosely coupled mode, and tightly symmetric mode; The non-coordinated mode corresponds to the anti-collision boundary constraint where the two hands do not interfere with each other. The loosely coupled mode corresponds to a master-slave following constraint with sequential timing. The tightly symmetrical mode corresponds to rigid or flexible cooperative transport constraints with relative orientation.

8. The method for constructing high-level behavioral topology for dual-arm cooperative operation as described in claim 1, characterized in that, Based on the co-operation constraints of both hands, the semantic rule model of the left hand action and the semantic rule model of the right hand action are integrated, and prior knowledge from a large language model is introduced to generate a task semantic rule generalization strategy for new scenarios, including: Under the aforementioned two-hand operation coordination constraint, the semantic rule model of the left hand action and the semantic rule model of the right hand action are integrated to generate a joint state space description; The new task instruction, the current scene context, and the joint state space description are concatenated and encoded together into a text prompt word sequence; The text prompt word sequence is input into a large language model, and the large language model is used to output a semantic rule sequence conditional probability distribution adapted to the current new scenario; A task semantic rule generalization strategy is generated by maximizing the conditional probability distribution of the semantic rule sequence.

9. The method for constructing high-level behavioral topology for dual-arm cooperative operation as described in claim 1, characterized in that, The topology map is used for the representation and reasoning of knowledge related to the operation tasks of dual-arm robots.

10. A dual-arm cooperative high-level behavioral topology construction system, applicable to dual-arm robots, characterized in that, The dual-arm collaborative high-level behavioral topology construction system includes: The semantic feature extraction module is used to acquire multimodal dual-arm embodied demonstration data of the robot, and extract the semantic features corresponding to the robot's left hand and the robot's right hand based on the multimodal dual-arm embodied demonstration data. The action semantic modeling module is used to perform action semantic recognition on the semantic features corresponding to the robot's left hand and the semantic features corresponding to the robot's right hand, thereby constructing a left-hand action semantic rule model and a right-hand action semantic rule model. The two-handed collaborative modeling module is used to construct a two-handed collaborative model based on a hybrid master-slave relationship graph, based on the left-hand action semantic rule model and the right-hand action semantic rule model, in order to infer multi-level collaborative patterns and output corresponding two-handed operation collaborative constraints. The rule integration and generalization module is used to integrate the left-hand action semantic rule model and the right-hand action semantic rule model based on the two-hand operation coordination constraint, and introduce the prior knowledge of the large language model to generate a task semantic rule generalization strategy for new scenarios. The topology graph generation module is used to generate a topology graph of the robot's dual-arm task state transition based on the task semantic rule generalization strategy.

Images