Flexible Assembly Method, System and Medium for Humanoid Robots Based on Chain of Thought

Abstract

An embodiment of the present application provides a flexible assembly method, system and medium for a humanoid robot based on a chain of thought, belonging to the technical field of natural language processing. This solution obtains the assembly task instructions provided by the user, parses the assembly task instructions through a chain of thought model, calculates the weight of each word in the assembly task instructions through an attention mechanism during the parsing process, dynamically adjusts the parsing target related to the current task according to the weight, decomposes the task layer by layer into multiple subtasks according to the parsing target to obtain a task tree structure, traverses the task tree structure to determine the assembly plan, and the assembly plan includes the execution arrangement order, task path planning and assembly action parameters. According to the assembly plan, the humanoid robot is controlled to perform the assembly operation, which can automatically extract task information without presetting a fixed instruction template, utilize the combination of a hierarchical chain of thought and an attention mechanism, and transform it into an assembly operation to generate a clear execution plan, improving the flexibility of task parsing.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and particularly to a flexible assembly method, system and medium for humanoid robots based on the chain of thought. Background Art

[0002] With the development of the manufacturing industry towards intelligence and flexibility, traditional fixed-process assembly systems have been difficult to meet the needs of diversified and customized production. As a solution to complex and changing production environments, flexible manufacturing systems can quickly adapt to different types of assembly tasks and product changes, improving the efficiency and flexibility of production lines. In this process, humanoid robots have gradually become an important role in the field of flexible manufacturing due to their humanoid operation methods and flexible working capabilities.

[0003] However, current humanoid robots still face multiple challenges in complex task parsing and autonomous assembly. For example, when facing new tasks, they often need to reprogram or adjust the task process, which not only increases the development time of the system, but also raises the usage threshold and cost. Traditional robot task allocation and planning mostly rely on pre-written programs or fixed instruction templates, with insufficient task parsing capabilities and difficulty in understanding complex instructions. This method appears too rigid and lacks flexibility when dealing with changing assembly tasks, and cannot meet the requirements of quick adjustment and real-time response. At the same time, when carrying out task planning and executing tasks, there is often a lack of in-depth understanding and adaptive capabilities for complex assembly processes, resulting in operators needing to invest a large amount of time in task guidance and parameter adjustment, greatly limiting the automation level of the production line. Summary of the Invention

[0004] The main purpose of the embodiments of this application is to propose a flexible assembly method, system and medium for humanoid robots based on the chain of thought, aiming to reduce development and operation costs without relying on pre-written programs or fixed instruction templates, automatically extract key information accurately from complex instructions, generate a clear execution plan, and realize the intelligence of flexible assembly.

[0005] To achieve the above object, on the one hand, an embodiment of this application proposes a flexible assembly method for humanoid robots based on the chain of thought, and the method includes the following steps:

[0006] Obtain the assembly task instructions provided by the user;

[0007] Perform task parsing on the assembly task instructions through the chain of thought model, and calculate the weight of each word in the assembly task instructions through the attention mechanism during the parsing process. Dynamically adjust the parsing target related to the current task according to the weight, and decompose the task layer by layer into multiple subtasks according to the parsing target to obtain a task tree structure;

[0008] Traverse the task tree structure to determine the assembly plan, where the assembly plan includes the execution arrangement sequence, task path planning, and assembly action parameters;

[0009] Control the humanoid robot to perform assembly operations according to the assembly plan.

[0010] In some embodiments, obtaining the assembly task instructions provided by the user includes the following steps:

[0011] Obtain the multimodal data provided by the user, where the multimodal data includes text, voice, and visual information;

[0012] Use natural language processing technology to perform word segmentation, part-of-speech tagging, and semantic parsing on the text and voice, and extract the main content and intention of the task to obtain text information;

[0013] Use computer vision technology to perform feature extraction and recognition on the visual information to obtain symbol information;

[0014] Map the text information and the symbol information to the vector space to obtain the assembly task instructions.

[0015] In some embodiments, calculating the weight of each word in the assembly task instructions through the attention mechanism and dynamically adjusting the parsing target related to the current task according to the weight includes the following steps:

[0016] Calculate the attention score of each word in the assembly task instructions based on the context information through the attention mechanism, where the context information includes the task historical state and domain knowledge;

[0017] Perform normalization processing on the attention score to obtain the weight of the word;

[0018] Determine the words with weights greater than the preset threshold as the parsing targets related to the current task.

[0019] In some embodiments, traversing the task tree structure to determine the assembly plan includes the following steps:

[0020] Traverse the task tree structure to determine the sequence and dependency relationship of subtasks;

[0021] According to the sequence and dependency relationship of the subtasks, perform logical verification on the task tree structure, and optimize the arrangement of subtasks in the task tree structure based on the attention mechanism to obtain the execution arrangement sequence;

[0022] According to the execution arrangement sequence, use the motion planning algorithm to determine the task path planning;

[0023] Decompose the subtask until it is decomposed into multiple basic actions that the humanoid robot can directly execute, and determine the required assembly action parameters according to the basic actions;

[0024] Obtain an assembly plan based on the execution arrangement order, the task path planning, and the assembly action parameters.

[0025] In some embodiments, the method for logically verifying the task tree structure according to the sequence and dependency of the subtasks, and optimizing the arrangement of the subtasks in the task tree structure based on the attention mechanism to obtain the execution arrangement order includes the following steps:

[0026] When there are multiple subtasks to be executed simultaneously, evaluate the priority and resource requirements of each subtask through the attention mechanism to obtain an evaluation result;

[0027] Determine a task scheduling strategy according to the evaluation result through a reinforcement learning algorithm, and adjust the arrangement of the subtasks in the task tree structure according to the task scheduling strategy to obtain the execution arrangement order.

[0028] In some embodiments, the flexible assembly method for a humanoid robot based on the chain of thought further includes the following steps:

[0029] Obtain monitoring data, where the monitoring data includes environmental information and assembly status data;

[0030] Judge whether there is an abnormal situation in the currently executed subtask according to the monitoring data;

[0031] When there is an abnormal situation in the subtask, re-parse the subtask through the attention mechanism and dynamically adjust the assembly plan corresponding to the subtask.

[0032] In some embodiments, the flexible assembly method for a humanoid robot based on the chain of thought further includes the following steps:

[0033] Record the task tree structure and the assembly status data to obtain an assembly process result;

[0034] Display the assembly process result through an interaction interface.

[0035] To achieve the above object, another aspect of the embodiments of the present application proposes a flexible assembly virtual system for a humanoid robot based on the chain of thought, and the virtual system includes:

[0036] A first module for obtaining an assembly task instruction provided by a user;

[0037] A second module, configured to perform task parsing on the assembly task instruction through a chain of thought model, calculate the weight of each word in the assembly task instruction through an attention mechanism during the parsing process, dynamically adjust the parsing target related to the current task according to the weight, and decompose the task layer by layer into multiple subtasks according to the parsing target to obtain a task tree structure;

[0038] A third module, configured to traverse the task tree structure to determine an assembly plan, where the assembly plan includes an execution arrangement order, a task path planning, and assembly action parameters;

[0039] A fourth module, configured to control a humanoid robot to perform an assembly operation according to the assembly plan.

[0040] To achieve the above object, on the other hand, an embodiment of the present application provides a flexible assembly hardware system for a humanoid robot based on a chain of thought. The hardware system includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the above method is implemented.

[0041] To achieve the above object, on the other hand, an embodiment of the present application provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above method is implemented.

[0042] The embodiments of the present application at least include the following beneficial effects: The present application provides a flexible assembly method, system and medium for a humanoid robot based on a chain of thought. This solution obtains an assembly task instruction provided by a user, performs task parsing on the assembly task instruction through a chain of thought model, calculates the weight of each word in the assembly task instruction through an attention mechanism during the parsing process, dynamically adjusts the parsing target related to the current task according to the weight, and decomposes the task layer by layer into multiple subtasks according to the parsing target to obtain a task tree structure. It can automatically and accurately extract key information from complex instructions, ignore irrelevant content, adapt to the variable expressions in natural language, and improve the flexibility and accuracy of the autonomous task parsing ability without relying on a preset fixed instruction template or rule by combining a hierarchical chain of thought and an attention mechanism.

[0043] By traversing the task tree structure to determine an assembly plan, where the assembly plan includes an execution arrangement order, a task path planning, and assembly action parameters, and controlling a humanoid robot to perform an assembly operation according to the assembly plan, the task can be transformed into an assembly operation, a clear execution plan can be generated, the development and operation costs of the flexible assembly system can be reduced, and the intelligentization of flexible assembly can be realized. Description of the Drawings

[0044] Figure 1It is a flowchart of the flexible assembly method of a humanoid robot based on a chain of thought provided by an embodiment of the present application;

[0045] Figure 2 It is a schematic structural diagram of a task tree provided by an embodiment of the present application;

[0046] Figure 3 It is a schematic structural diagram of a flexible assembly software system of a humanoid robot based on a chain of thought provided by an embodiment of the present application;

[0047] Figure 4 It is a flowchart of the overall operation of the chain of thought provided by an embodiment of the present application;

[0048] Figure 5 It is a schematic structural diagram of a virtual system for flexible assembly of a humanoid robot based on a chain of thought provided by another embodiment of the present application;

[0049] Figure 6 It is a schematic hardware structure diagram of a hardware system for flexible assembly of a humanoid robot based on a chain of thought provided by an embodiment of the present application. Detailed implementation manners

[0050] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application detailed in the appended claims.

[0051] It can be understood that the terms "first", "second", etc. used in the present application can be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information. Depending on the context, the words "if", "when" as used herein can be interpreted as "when...", "while...", or "in response to determining".

[0052] The terms "at least one", "multiple", "each", "any one", etc. used in the present application, at least one includes one, two or more than two, multiple includes two or more than two, each refers to each of the corresponding multiple, and any one refers to any one of the multiple.

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the technical field to which this application belongs. The terms used herein are for the purpose of describing embodiments of this application only and are not intended to limit this application.

[0054] Before elaborating on the embodiments of this application in detail, some nouns and terms involved in the embodiments of this application are first explained.

[0055] (1) Hierarchical Chain of Thought (H-CoT): The hierarchical chain of thought technology decomposes complex tasks into subtasks at multiple levels. The chain of thought structure enables the robot to gradually understand the overall framework and specific steps of the task.

[0056] (2) Attention Mechanism: It refers to a mechanism in which, when processing information, the system or model can selectively focus on a part of the input information while ignoring other irrelevant or secondary information. This mechanism mimics the way humans allocate attention when processing complex perceptual information, enabling the system or model to process and understand information more efficiently. In deep learning, the attention mechanism is usually implemented as a weight parameter allocation mechanism. By calculating the importance of different parts of the information and accordingly allocating different weights, it assists the model in capturing relatively important information.

[0057] Next, some related technologies involved in the embodiments of this application are described.

[0058] In current flexible manufacturing systems, the following key problems exist in humanoid robots in task parsing and autonomous assembly, which seriously limit the efficiency and adaptability of flexible manufacturing systems.

[0059] (1) Insufficient task parsing ability, making it difficult to understand complex instructions.

[0060] Currently, when receiving assembly tasks, humanoid robots mostly rely on pre-written instruction scripts or fixed template rules. Although this method can perform well in highly standardized assembly tasks, in the face of a changing and complex production environment, it cannot flexibly understand and parse new task instructions, and it is difficult to cope with sudden changes and customized requirements in production. This results in poor adaptability of humanoid robots to new tasks, requires a large amount of manual intervention, and it is difficult to achieve true autonomous assembly.

[0061] (2) Lack of flexibility and self-adaptability in task planning and execution.

[0062] In the related art, humanoid robots usually adopt a fixed task planning process and lack the ability to dynamically adjust during the task process. When the production environment or assembly task changes, the humanoid robot cannot optimize and adjust the assembly process according to the real-time situation. This results in problems such as rigid movements and low efficiency for the humanoid robot during task execution, and it cannot meet the requirements of flexible manufacturing for high efficiency and flexibility.

[0063] (3) Insufficient information utilization and inability to effectively distinguish the key points of the task.

[0064] The task planning method of humanoid robots usually cannot make full use of the key information in the task description and is easily interfered by irrelevant information, resulting in low task parsing efficiency. Especially in complex assembly tasks, the humanoid robot needs to understand a large amount of detailed information, but the related technologies are difficult to automatically identify and prioritize the key information, thus affecting the overall task execution effect.

[0065] (4) Weak multi-task parallel processing ability and low task scheduling efficiency.

[0066] In a flexible manufacturing environment, the production line needs to process multiple assembly tasks simultaneously. The current humanoid robot system has limited capabilities in task scheduling and priority allocation. Due to the lack of the ability to analyze the importance and urgency of tasks, the humanoid robot often cannot reasonably arrange the execution order of tasks, resulting in waste of resources and a decrease in production efficiency.

[0067] In view of this, in the embodiments of the present application, a flexible assembly method, system and medium for a humanoid robot based on a chain of thought are provided. This solution obtains the assembly task instructions provided by the user, parses the assembly task instructions through a chain of thought model, and calculates the weight of each word in the assembly task instructions through an attention mechanism during the parsing process. Dynamically adjust the parsing target related to the current task according to the weight, decompose the task layer by layer into multiple subtasks, and obtain a task tree structure. It can automatically and accurately extract key information from complex instructions, ignore irrelevant content, adapt to the changing expressions in natural language, and improve the flexibility and accuracy of the autonomous task parsing ability without relying on a preset fixed instruction template or rule.

[0068] Determine the assembly plan by traversing the task tree structure. The assembly plan includes the execution arrangement order, task path planning, and assembly action parameters. Control the humanoid robot to perform the assembly operation according to the assembly plan, which can convert the task into an assembly operation, generate a clear execution plan, reduce the development and operation costs of the flexible assembly system, and realize the intelligence of flexible assembly.

[0069] The flexible assembly method for humanoid robots based on the chain of thought provided by the embodiments of this application relates to the field of natural language processing technology. The flexible assembly method for humanoid robots based on the chain of thought provided by the embodiments of this application can be applied to a terminal, a server, or software running on a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a vehicle-mounted terminal, etc., but is not limited thereto; the server side can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application implementing the flexible assembly method for humanoid robots based on the chain of thought, etc., but is not limited to the above forms.

[0070] This application can be used in many general or specific computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and so on. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This application can also be practiced in a distributed computing environment where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0071] Figure 1 is an optional flowchart of the flexible assembly method for humanoid robots based on the chain of thought provided by the embodiments of this application. Figure 1 The method in can include but is not limited to steps S101 to S104.

[0072] Step S101, obtain the assembly task instruction provided by the user.

[0073] It should be noted that in the flexible assembly of humanoid robots, the humanoid robot needs to receive the assembly task instruction provided by the user. The assembly task instruction is multimodal information provided by the operator to the humanoid robot, which is a series of instructions used to guide the humanoid robot to complete a specific assembly task. These instructions usually include tasks such as the target object of the assembly, the required components, the assembly steps, and the accuracy requirements.

[0074] In step S102, the assembly task instructions are parsed through a thought chain model, and during the parsing process, the weight of each word in the assembly task instructions is calculated through an attention mechanism. The parsing target related to the current task is dynamically adjusted according to the weight, and the task is decomposed layer by layer into multiple subtasks according to the parsing target, obtaining a task tree structure.

[0075] In some embodiments, after obtaining the assembly task instructions, since the instructions often include a large amount of complex information, including information related to the current task as well as redundant and irrelevant information, it is also necessary to parse the assembly task instructions through a pre-trained thought chain model. The hierarchical thought chain decomposes complex tasks into subtasks layer by layer, forming a tree-like structure that is convenient for the humanoid robot to gradually understand and execute. However, traditional thought chain technologies lack a flexible attention mechanism when dealing with diverse assembly tasks and are difficult to cope with the dynamic changes of information during the task process.

[0076] To solve this problem, this embodiment introduces an attention mechanism, enabling the humanoid robot to focus on key information during the task parsing process and ignore redundant data, thereby improving the accuracy and execution efficiency of task decomposition.

[0077] The assembly task instructions are parsed through a thought chain model, and during the parsing process, the weight of each word in the assembly task instructions is calculated through an attention mechanism. The parsing target related to the current task is dynamically adjusted according to the weight, enabling the humanoid robot to accurately extract key information from the operator's instructions and dynamically adjust the model's attention degree to different task details, where the current task refers to a specific assembly task that the humanoid robot is processing or about to process. Then, the task is decomposed layer by layer into multiple subtasks according to the parsing target, obtaining a task tree structure.

[0078] Specifically, first, the assembly task instructions are input into the thought chain model for preliminary parsing. During the parsing process, the attention mechanism helps the humanoid robot extract the main task objectives from the complex instructions, obtaining multiple high-level tasks, which are used as the high-level nodes of the thought chain. Since the instructions may contain multiple task objectives, the attention mechanism calculates the weight of each word in the instructions, and determines the key steps in the task according to the weight distribution to obtain the parsing target. The higher the weight, the more likely it is a key step in the current task. For example, if the instruction is "Please install part A onto component B and then perform precision calibration", the high-level tasks include two tasks: "Install part A onto component B" and "Precision calibration".

[0079] It should be noted that the attention mechanism can pre-train and learn the large language model through a large amount of sample data related to assembly tasks, enabling the large language model to identify which steps are key in a specific scenario or context.

[0080] It can be understood that the attention mechanism does not directly understand which information is important from the assembly task instructions. Instead, through the use of a pre-trained model and combined with training data in a specific task domain, it automatically learns to identify common key steps in the task. For example, after being trained with a large amount of assembly task data, the model can learn to automatically assign weights and dynamically determine the focus of the current task. The attention mechanism adjusts the model's attention to the input information according to these weights, thus ensuring higher weights and attention are given to the key content of the task. For example, the trained chain of thought model can identify that operations such as "installing parts" and "precision calibration" are usually key steps in the assembly task.

[0081] Through the pre-trained chain of thought model and training data in the task domain, the model can flexibly handle different task instructions. The attention mechanism automatically determines which information in the instruction is the key to the current task based on the task input (i.e., the assembly task instruction). Therefore, there is no need to preset fixed template rules.

[0082] Secondly, according to the multiple parsing targets obtained by parsing, each parsing target is further recursively decomposed, decomposing the high-level task into more detailed subtasks to form middle-level nodes. During the subtask decomposition process, the attention mechanism guides the humanoid robot to focus on the details related to the current task, avoiding interference from irrelevant information, and refining the goal of the high-level task. For example, the high-level task of "installing part A to component B" can be decomposed into "obtaining part A", "locating component B", and "installing part A to component B".

[0083] Finally, after gradual decomposition, a set including high-level tasks and subtasks is obtained, and each node is connected to each other through the dependency relationship of the tasks, forming a task tree structure. In the task tree, each node represents a high-level task or subtask, and the connection between nodes represents the dependency relationship of the tasks.

[0084] Step S103, traverse the task tree structure to determine the assembly plan, where the assembly plan includes the execution arrangement order, task path planning, and assembly action parameters.

[0085] Specifically, the task tree structure is a multi-way tree. In this structure, each node can have any number of child nodes. By traversing the task tree, according to the sequence and dependency relationship of the tasks, the assembly plan is determined, where the assembly plan includes the execution arrangement order, task path planning, and assembly action parameters.

[0086] Furthermore, during the process of generating the assembly plan, kinematic and dynamic constraints of the humanoid robot, as well as various limiting conditions in the environment, can also be considered. For example, during the process of generating the task path planning, motion planning algorithms are used to generate feasible motion paths to avoid collisions with obstacles; when generating the assembly action parameters for grasping parts, appropriate grasping forces and postures are calculated to avoid damaging the parts or generating assembly errors.

[0087] Step S104, control the humanoid robot to perform the assembly operation according to the assembly plan.

[0088] Exemplarily, the humanoid robot sequentially executes each task step according to the generated assembly plan and the execution arrangement order. For example, in the order of "move to the position of part A", "grasp part A", and "install part A into component B", according to the task path planning, it moves from the part storage area to the workbench, uses the vision sensor to confirm the part position according to the assembly action parameters, and uses the force sensor to monitor the force condition during the assembly process to perform the assembly operation.

[0089] In this embodiment, by introducing a hierarchical thought chain structure, complex tasks are refined layer by layer, and combined with the attention mechanism, the humanoid robot can parse the assembly task instructions input by the operator and dynamically identify the key information therein, automatically generating a detailed task decomposition and assembly plan, which not only improves the flexibility of task parsing, but also greatly improves the understanding accuracy of task details, enabling the humanoid robot to accurately execute various complex assembly tasks. Introducing the attention mechanism during the task parsing process enables the humanoid robot to have an adaptive adjustment ability, without relying on fixed templates or predefined rules, solving the problem that the current robot assembly system has difficulty adapting to the variable expression ways in natural language when dealing with diverse and complex task instructions.

[0090] In addition, compared with the current robot assembly system, when facing new tasks, it often needs to reprogram or adjust the task process, which not only increases the system development time, but also raises the usage threshold and cost. In this embodiment, by combining the large language model and multi-modal parsing ability, the humanoid robot can understand the natural language description of the operator and automatically generate a task plan, greatly simplifying the process of task writing and adjustment. The operator only needs to issue a simple instruction to complete the task assignment to the humanoid robot, without writing complex scripts, thus reducing the system development and operation costs.

[0091] In some embodiments, step S101 may include but is not limited to steps S201 to S204.

[0092] Step S201, obtain multi-modal data provided by the user, where the multi-modal data includes text, voice, and visual information.

[0093] It is understandable that user multimodal data is data composed of the user using various information forms when providing assembly task instructions, including but not limited to describing the specific content and requirements of the assembly task in natural language (such as Chinese, English, etc.), or issuing instructions through voice, and expressing intentions through multimodal information such as gestures and schematic diagrams. In order for the humanoid robot to comprehensively understand this information, it is first necessary to obtain the multimodal data provided by the user for subsequent unified encoding of the task input.

[0094] It should be noted that the user can provide multimodal data in one or more ways according to actual needs and preferences, and the embodiments of the present application do not make specific restrictions.

[0095] Step S202, using natural language processing technology to perform word segmentation, part-of-speech tagging, and semantic parsing on the text and speech, and extracting the main content and intention of the task to obtain text information.

[0096] Optionally, when the multimodal data includes text and speech, natural language processing technology is used to perform word segmentation, part-of-speech tagging, and semantic parsing on the text and speech to extract the main content and intention of the task to obtain text information.

[0097] Exemplarily, the voice command is recognized, the voice signal is converted into text, and then natural language processing technology is used to perform word segmentation and part-of-speech tagging on the text to identify which words in the command are nouns, verbs, etc. Then, semantic parsing is performed to understand the overall meaning and intention of the command, which usually involves mapping the segmented text to a semantic framework or using a deep learning model for semantic understanding. For example, in the command "Please install part A onto component B", after word segmentation, we get "Please", "install", "part A", "onto", "component B". Through semantic parsing, it is recognized that the main content of the task is "install", and the key information "part A" and "component B" are extracted.

[0098] Step S203, using computer vision technology to perform feature extraction and recognition on the visual information to obtain symbolic information.

[0099] Optionally, when the multimodal data includes visual information such as gestures and schematic diagrams, computer vision technology is used to perform feature extraction on the visual information to convert the visual information into symbolic information that can be processed by a computer.

[0100] Exemplarily, first, preprocess the gesture image (such as denoising, enhancing, etc.), extract feature information in the gesture image such as shape, movement trajectory, etc., and then use a recurrent neural network to classify and recognize the extracted features to convert the features into symbolic information.

[0101] Step S204: Map the text information and symbol information into a vector space to obtain the assembly task instruction.

[0102] Specifically, map the text information and symbol information obtained by converting all multimodal data through the embedding layer into a unified high-dimensional vector space to form a multimodal feature representation of the task input, obtain the assembly task instruction, and provide a basis for subsequent task parsing.

[0103] In step S204 of another embodiment, when the multimodal data consists only of text or language, natural language processing techniques are used to perform word segmentation, part-of-speech tagging, and semantic parsing on the text or speech to extract the main content and intention of the task to obtain the text information, and map the text information into the vector space to obtain the assembly task instruction.

[0104] Alternatively, when the multimodal data consists only of visual information, computer vision techniques are used to extract features and recognize the visual information to obtain the symbol information, and map the symbol information into the vector space to obtain the assembly task instruction.

[0105] In step S102 of some embodiments, calculate the weight of each word in the assembly task instruction through the attention mechanism, and dynamically adjust the parsing target related to the current task according to the weight, which may include but is not limited to steps S301 to S303.

[0106] Step S301: Calculate the attention score of each word in the assembly task instruction based on the context information through the attention mechanism, where the context information includes the task history state and domain knowledge.

[0107] Specifically, the attention mechanism was initially used in the field of neural machine translation. When processing sequence data, it dynamically focuses on different parts of the input sequence. Introducing the attention mechanism allows the model to automatically identify and focus on the key information in the instruction during task parsing.

[0108] However, traditional scoring functions usually only adopt a simple matching method. To improve the understanding of complex instructions, in this embodiment, the scoring function is improved, context information is introduced during the attention scoring process, and the attention score of each word in the assembly instruction is calculated based on the context information through the attention mechanism, where the context information includes the task history state at the previous time step and domain knowledge related to the task.

[0109] The scoring function ei is expressed as formula (1):

[0110] e i =Score(h i ,S t-1 )=v T tanh(W h h i+W s S t-1 +W k k i ) (1),

[0111] In formula (1), h i is the hidden state of the i-th word in the input sequence, usually the representation obtained through the encoder; S t-1 represents the decoder state of the previous time step, reflecting the historical context of the model, k i is the domain knowledge vector, representing the domain knowledge related to the task; W h 、W s 、W k represent weight matrices, which are parameters that need to be optimized through learning; v is a learnable weight vector used to perform weighted aggregation on the output of the tanh function.

[0112] Exemplarily, first, receive the assembly task instruction input by the task and obtain the task state of the previous step, and at the same time call the domain knowledge base to obtain the background information related to the current task. Then, use the scoring function to score each word or each segment of the input information to evaluate the importance of each word.

[0113] It should be noted that in the case of parsing the instruction for the first time, an initial value or default value can be set for the decoder as the decoder state of the previous time step, and it is scored together with the current input word through the scoring function.

[0114] Step S302, normalize the attention scores to obtain the weights of the words.

[0115] Specifically, in the process of task decomposition at each level, the humanoid robot normalizes the attention scores of each word and calculates the weights of each word.

[0116] The weight α i is calculated by the formula as shown in formula (2):

[0117]

[0118] In formula (2), α i represents the attention weight of the i-th word, e i is the scoring function used to evaluate the importance of the current word for task parsing, n is the length of the input sequence, and the scoring function e j calculates the importance of the j-th word in the input sequence for task parsing.

[0119] Exemplarily, for the instruction "Please install part A onto component B and then perform precision calibration", the attention mechanism will assign higher weights to "install part A onto component B" and "perform precision calibration".

[0120] Step S303: Determine the words with weights greater than the preset threshold as the parsing targets related to the current task.

[0121] Optionally, according to the calculated weights, determine the words with weights greater than the preset threshold in the assembly task instructions as the parsing targets related to the current task. For example, "Install part A" has a higher weight than other irrelevant descriptions because it is related to "assembly". The attention mechanism determines that this is a key step in the current task based on this weight.

[0122] In step S303 of another embodiment, it is also possible to sort according to the calculated weights and determine the words within the preset quantity as the parsing targets of the current task.

[0123] In this embodiment, the system does not need to establish a fixed subtask library for matching. Instead, by introducing the attention mechanism, the model can flexibly handle different task instructions during task parsing, automatically identify and focus on the key information in the instructions according to the task input and context, so as to assign weights to each step. The higher the weight of a step, the more likely it is a key step in the current task, thereby improving the humanoid robot's understanding of complex assembly tasks.

[0124] In some embodiments, step S103 may include but is not limited to steps S401 to S405.

[0125] Step S401: Traverse the task tree structure to determine the sequence and dependencies of subtasks.

[0126] Specifically, common traversal methods for multi-way trees include depth-first traversal and breadth-first traversal. Depth-first traversal means starting from the root node and traversing each branch of the tree to the child nodes, and then traversing other branches. Depth-first traversal is further divided into pre-order traversal and post-order traversal. Among them, pre-order traversal means that the root node is visited before the child nodes, and post-order traversal means that the child nodes are processed preferentially during the visit; breadth-first traversal starts from the root node and traverses layer by layer from top to bottom.

[0127] Refer to Figure 2 , given an N-way tree, for the whole tree, node 1 is the root of node 3, node 2, and node 4, and node 3 is the root of node 5 and node 6.

[0128] When the pre-order traversal is used to traverse the task tree, the access order is node 1, node 3, node 5, node 6, node 2, node 4.

[0129] When the post-order traversal is used to traverse the task tree, the access order is node 5, node 6, node 3, node 2, node 4, node 1.

[0130] When performing a breadth - first traversal of the task tree, the access order is node 1, node 3, node 2, node 4, node 5, and node 6.

[0131] Determine the sequence and dependency relationship of subtasks by traversing all root nodes and child nodes of the task tree structure.

[0132] Step S402: According to the sequence and dependency relationship of subtasks, perform logical verification on the task tree structure, and optimize the arrangement of subtasks in the task tree structure based on the attention mechanism to obtain the execution arrangement order.

[0133] Specifically, according to the sequence and dependency relationship of subtasks, perform logical verification on the task tree structure. By analyzing the domain knowledge base to compare the generated task steps, eliminate unreasonable operation steps, ensure the rationality of task dependency relationships and execution conditions during the task decomposition process, and optimize the arrangement of subtasks in the task tree structure based on the attention mechanism, adjust the node order in the task tree to optimize the execution efficiency, and obtain the execution arrangement order.

[0134] Step S403: According to the execution arrangement order, use a motion planning algorithm to determine the task path planning.

[0135] Specifically, the system uses a motion planning algorithm (such as the Rapidly - exploring Random Tree algorithm or the A* algorithm) according to the execution arrangement order, combined with the joint constraints of the humanoid robot and the environmental map, to generate a feasible motion path to determine the task path planning.

[0136] Step S404: Decompose the subtasks until they are decomposed into multiple basic actions that the humanoid robot can directly execute, and determine the required assembly action parameters based on the basic actions.

[0137] Specifically, after determining the task execution order and task path, continue to decompose the subtasks using the attention mechanism. Decompose each subtask into specific operations to be performed at each task node until they are decomposed into multiple basic actions that the humanoid robot can directly execute, and determine the required assembly action parameters such as position information, force control parameters, etc. based on the basic actions.

[0138] Exemplarily, use a force control algorithm to ensure that the humanoid robot applies appropriate force during the assembly process to avoid damaging parts or generating assembly errors.

[0139] Step S405: Obtain the assembly plan according to the execution arrangement order, task path planning, and assembly action parameters.

[0140] Specifically, the execution arrangement order, task path planning, and assembly action parameters are integrated to form a complete assembly plan, which details all the steps and actions that the humanoid robot needs to perform during the entire assembly process, as well as the specific parameters of each step and action, so that during the execution process, the humanoid robot can be controlled to gradually complete each subtask according to the planned path and action sequence.

[0141] In some embodiments, step S402 may include, but is not limited to, steps S501 to S502.

[0142] Step S501, when there are multiple subtasks to be executed simultaneously, evaluate the priority and resource requirements of each subtask through an attention mechanism to obtain an evaluation result.

[0143] Specifically, when there are multiple subtasks to be executed simultaneously, all the subtasks to be executed simultaneously are analyzed as a set, and the priority and resource requirements of each subtask are evaluated through an attention mechanism to obtain an evaluation result.

[0144] Step S502, determine the task scheduling strategy through a reinforcement learning algorithm according to the evaluation result, and adjust the arrangement of subtasks in the task tree structure according to the task scheduling strategy to obtain the execution arrangement order.

[0145] Furthermore, determine the task scheduling strategy through a reinforcement learning algorithm according to the evaluation result. For example, when two subtasks both need to use the same tool, their execution order needs to be arranged, or an alternative solution needs to be found. Dynamically adjust the arrangement of subtasks in the task tree structure according to the task scheduling strategy, adaptively add new tasks or change the order of original subtasks to obtain the execution arrangement order.

[0146] Exemplarily, the reinforcement learning algorithm (such as Q-learning or deep reinforcement learning DQN) dynamically adjusts and optimizes the multi-task scheduling strategy through continuous learning and feedback loops according to the priority, resource requirements, task dependencies, etc. of the tasks in the evaluation result to ensure reasonable resource allocation and maximize the overall efficiency.

[0147] First, encode the current task execution status, resource usage, and task dependencies into a state vector as the input of the reinforcement learning. These state information include the current progress of each task, the resource requirements of the task, the availability of resources, etc.

[0148] Secondly, determine the action space, which is defined as the scheduling decisions that can be taken in each state. For example, one can choose which task to execute first, pause a certain task, or change the task order, etc. Since the choice of each action will affect the smooth progress of task execution, reinforcement learning evaluates the effect of each scheduling decision through a reward mechanism, sets a reward value for each action, and the higher the scheduling efficiency, the greater the reward. The reward can be based on multiple metrics, such as the completion time of tasks, resource utilization rate, reduction of conflicts, etc. If the scheduling is successfully optimized, the task parallelism is increased, and the waiting time is reduced, a higher reward will be obtained.

[0149] The reinforcement learning algorithm tries different scheduling strategies and learns which scheduling decisions are the optimal solutions based on the reward feedback in different states. By repeatedly executing and updating the scheduling strategy, it can efficiently schedule tasks when encountering the same or similar tasks in the future.

[0150] It can be understood that the reinforcement learning algorithm allows the system to make real-time adjustments according to the feedback during task execution. If a new task is added or an existing task changes, the system will dynamically evaluate the new state and select an appropriate scheduling strategy. Through continuous exploration and adjustment, the system gradually learns the optimal multi-task scheduling method.

[0151] In this embodiment, by combining the result of the chain of thought parsing and the reinforcement learning algorithm, an intelligent multi-task scheduling mechanism is established. The attention mechanism is used to evaluate the priority and resource requirements of each task, and the execution order of tasks is optimized so that high-priority tasks can be processed in a timely manner while reasonably arranging resources. This multi-task scheduling strategy improves the parallel processing ability of the humanoid robot and enhances the overall working efficiency of the flexible assembly production line, and can effectively solve the problems of resource waste and production bottlenecks in the current robot system in a multi-task environment due to the lack of the ability to evaluate and schedule task priorities.

[0152] In some embodiments, the flexible assembly method of the humanoid robot based on the chain of thought may further include, but is not limited to, steps S601 to S603.

[0153] Step S601, obtain monitoring data, where the monitoring data includes environmental information and assembly status data.

[0154] Specifically, assembly tasks are often complex and uncertain, and changes in the environment may affect the smooth execution of tasks. Therefore, it is necessary to obtain monitoring data through multiple sensors in real time to enable the system to have the ability of real-time feedback and adaptive adjustment. The detection data includes environmental information and assembly status information. For example, a vision sensor can detect the position of parts to obtain visual data, and a force sensor can sense the force during the assembly process to obtain tactile data.

[0155] Step S602: Determine whether there is an abnormal situation in the currently executing subtask according to the monitoring data.

[0156] Optionally, determine whether there is an abnormal situation in the currently executing subtask according to the monitoring data obtained by multiple sensors.

[0157] Exemplarily, determine whether there is a position deviation according to the position of the part monitored by the vision sensor, and determine whether the force exceeds or is lower than the preset safety range according to the force perception of the force sensor.

[0158] Step S603: When there is an abnormal situation in the subtask, re-parse the subtask through the attention mechanism and dynamically adjust the assembly plan corresponding to the subtask.

[0159] Specifically, when it is detected that there is an abnormal situation in the subtask, trigger the adaptive adjustment mechanism, including re-parsing the subtask through the attention mechanism and dynamically adjusting the assembly plan corresponding to the subtask. Among them, the adaptive adjustment is mainly reflected in two aspects: on the one hand, use the attention mechanism to re-evaluate the key points of the task, adjust the task parsing and planning; on the other hand, modify the execution plan in real time and take corrective measures. For example, if a position deviation of the part is found during the assembly process, the system will temporarily suspend the current action, reposition the part, and update the subsequent assembly path and action parameters. This real-time feedback and adjustment mechanism ensures that the humanoid robot can still complete the assembly task efficiently and accurately in a dynamic environment, improving the robustness of the system.

[0160] It can be understood that when an abnormality is detected, the problem is usually only limited to the currently executing subtask. At this time, only the attention mechanism needs to be used to re-evaluate the currently executing subtask and make corresponding adjustments. The adaptive adjustment mechanism usually only re-evaluates the subtask with detected abnormalities, which can improve efficiency and avoid unnecessary replanning of the entire task. Only when the abnormality affects the entire task chain (such as a key equipment failure), trigger a global evaluation, consider re-evaluating the task globally and making comprehensive adjustments.

[0161] It should be noted that during the task execution of some embodiments, the humanoid robot can dynamically adjust the attention weights according to real-time feedback and re-evaluate the key parts in the task. For example, when an assembly error is detected, the attention mechanism will assign more weights to the relevant assembly steps, dynamically adjust the focus of attention, and prompt the humanoid robot to focus on inspection and correction.

[0162] Furthermore, in step S603 of some embodiments, in order to make full use of multi-modal information such as vision and touch, the scoring function is further improved. The improved attention mechanism fuses multi-modal data to obtain a new scoring function as shown in Equation (3):

[0163] e i = v T anh(W h h i + W s S t-1 + W v v i + W t t i ) (3),

[0164] In Equation (3), W v and W t represent weight matrices, which are parameters that need to be optimized through learning; v i is the feature vector of visual data, and t i is the feature vector of tactile data.

[0165] Optionally, the data of sensors such as vision and touch are integrated through a multi-modal data fusion module to generate a unified feature representation, so as to help the humanoid robot better perceive the environment, support the humanoid robot to accurately locate components and adjust the assembly force during task parsing and assembly processes, thereby improving the stability and accuracy of the assembly process.

[0166] In this embodiment, considering that when current humanoid robots encounter problems such as position deviation and component specification changes during the assembly process, manual intervention is often required for adjustment, resulting in reduced production efficiency. In this embodiment, during the assembly process, according to the monitored data (such as vision and force sense information) fed back in real time, the attention weights are dynamically adjusted, the key points of the task are re-evaluated, and the execution steps are automatically optimized, so that the humanoid robot can quickly make self-adjustments when facing environmental changes during the assembly process, thereby realizing the adaptive ability of the humanoid robot to a changing environment, reducing human intervention, and improving the stability and continuity of the production line.

[0167] In some embodiments, the flexible assembly method of the humanoid robot based on the chain of thought may further include but is not limited to steps S701 to S702.

[0168] Step S701, record the task tree structure and the assembly state data to obtain the assembly process result.

[0169] Specifically, a tree structure or a linked list is used to record all nodes of the task tree structure obtained in the task parsing process, including the task objectives, execution steps and their dependencies of each node, and the collected assembly state data is associated with the corresponding nodes in the task tree structure, and the assembly process result is obtained by summarization.

[0170] Optionally, the results of the assembly process can be generated in the form of an assembly report, which lists the execution status, elapsed time, execution progress of each step, and information on any relevant exceptions or errors.

[0171] Step S702, display the results of the assembly process through the interaction interface.

[0172] It can be understood that, in order to facilitate the operator to view the parsing results and execution status of the task in real time, the results of the assembly process can also be displayed through the interaction interface, and complex assembly processes can be presented in an intuitive and understandable way using multimedia elements such as charts and animations. For each step, the interaction interface can also provide functions such as zooming in, zooming out, dragging, filtering, and searching, so that the operator can view and analyze the results of the assembly process as needed.

[0173] Next, in combination with specific application examples, the solutions of the embodiments of the present invention will be introduced and described in detail.

[0174] In order to implement the flexible assembly method for humanoid robots based on the chain of thought provided by the embodiments of the present application and solve the problems of insufficient flexibility and adaptability of humanoid robots in complex task parsing and assembly processes in the related art, refer to Figure 3 , the embodiments of the present application provide a software system for a flexible assembly method for humanoid robots based on the chain of thought, which adopts a modular design and includes a multi-modal task input and encoding module, a hierarchical chain of thought task parsing module with an attention mechanism, a task planning and execution module, a real-time feedback and adaptive adjustment module, a multi-task scheduling and optimization module, and a system control and human-machine interaction interface module.

[0175] Each functional module communicates through a standard interface. For example, the task parsing module, the task planning and execution module, and the real-time feedback and adaptive adjustment module work together through message passing and data sharing.

[0176] The multi-modal task input and encoding module is used to receive the assembly task instructions issued by the operator, and these instructions may contain various information forms, such as text descriptions, voice instructions, or even gestures or schematic diagrams. For the convenience of humanoid robots to process, the task input can be uniformly encoded through the multi-modal task input and encoding module.

[0177] In the implementation process, the system uses natural language processing technology to perform word segmentation, part-of-speech tagging, and semantic parsing on instructions in text or voice form, and extracts the main content and intention of the task. At the same time, for visual inputs such as gestures or schematic diagrams, computer vision technology is used for feature extraction and recognition, and they are converted into symbolic information that can be processed. Finally, all modal data are mapped to a unified high-dimensional vector space through an embedding layer to form a multi-modal feature representation of the task input, providing a basis for subsequent task parsing.

[0178] The hierarchical thinking chain task parsing module of the attention mechanism is used to use the encoded assembly task instructions to carry out the high-level task parsing stage. The task instructions are parsed word by word or fragment by fragment through the attention mechanism, so as to automatically identify the key information in the assembly task instructions and obtain multiple high-level tasks.

[0179] Next, each high-level task is recursively decomposed into subtasks. During the subtask decomposition process, the humanoid robot is guided to focus on the details related to the current subtask to avoid being disturbed by irrelevant information. The attention mechanism continues to play a role, guiding the model to focus on the detailed information related to the current subtask. For example, when decomposing "installing part A to component B", the attention mechanism will make the model focus on the necessary information such as the attributes of "part A" and the location of "component B".

[0180] When implementing task decomposition, the model uses a recurrent neural network (such as LSTM or GRU) or Transformer architecture, combined with an attention mechanism, to parse the task instructions word by word or fragment by fragment. In each parsing step, the model dynamically calculates the attention weight based on the task status of the previous step and the current input, and determines the next content to focus on, thereby flexibly adjusting the parsing path and ensuring a deep understanding of the task.

[0181] Finally, after multi-level task decomposition, the system generates specific task execution steps based on key information and obtains a complete task tree structure. The task tree contains detailed information from high-level task objectives to specific execution steps, laying the foundation for subsequent task planning and execution.

[0182] After the task planning and execution module obtains the task tree generated by the task parsing module, it traverses the task tree and generates a specific assembly plan according to the order and dependency of the tasks, including the execution arrangement sequence, task path planning, and the generation of assembly action parameters. The task tree structure is used to generate detailed assembly steps, and the motion planning algorithm is used to guide the humanoid robot to perform operations such as moving, grasping, and installing, ensuring that the assembly task is completed efficiently according to the predetermined process.

[0183] During the execution process, in order to improve the accuracy of execution, sensor feedback control can also be added at key steps. For example, visual sensors can be used to confirm the position and posture of parts, and force sensors can be used to monitor the force conditions during the assembly process, so as to adjust the movements of the humanoid robot in real time.

[0184] The real-time feedback and adaptive adjustment module is used to monitor the status and environmental information of the humanoid robot in real time through multiple sensors during the execution phase, and adjust the task planning according to the feedback. By monitoring information such as position deviation and force feedback during the assembly process in real time, the system can re-adjust the task execution plan when detecting abnormalities, ensure the smooth completion of tasks, and thus achieve the adaptive ability of the humanoid robot to the changing environment.

[0185] The multi-task scheduling and optimization module is used to reasonably schedule and optimize multiple tasks when the humanoid robot needs to process multiple assembly tasks simultaneously. The implementation details include: first, analyze the set of tasks to be executed, use the attention mechanism to evaluate the priority and resource requirements of each task, and then, combined with the reinforcement learning algorithm, learn the optimal task scheduling strategy to reasonably allocate the resources of the humanoid robot and maximize the overall assembly efficiency.

[0186] During the scheduling process, the system needs to consider the dependencies between tasks, avoid resource conflicts, ensure that high-priority tasks are processed in a timely manner, achieve the reasonable parallel execution of multiple tasks, optimize resource allocation, and improve the overall assembly efficiency.

[0187] The system control and human-machine interaction interface module is used to provide human-machine interaction, support the input of task instructions and real-time monitoring of the assembly process. The operator can input task instructions through natural language in the user interface and view the parsing results, execution status, progress, and assembly results of the tasks in real time, which is convenient for the operator to monitor in real time. The interface also provides a visual display of the assembly process to help the operator understand the working conditions of the humanoid robot. In necessary cases, the operator can intervene in the humanoid robot through the interface to provide additional instructions or adjust task parameters.

[0188] In this embodiment, by using the hierarchical thinking chain technology and combining the attention mechanism, the humanoid robot can automatically parse complex natural language assembly instructions, convert them into specific operation steps, and achieve efficient and accurate task parsing. With the rapid development of large models (such as GPT-4, etc.) and natural language processing technologies, the task parsing method based on pre-trained language models demonstrates powerful understanding and reasoning capabilities. Using large models, the humanoid robot can extract key task information from complex natural language instructions, achieve automatic task parsing and execution, greatly reduce the need for manual intervention, and improve the humanoid robot's understanding ability for diverse assembly tasks.

[0189] By introducing the attention mechanism, the humanoid robot can give priority to key information during the task parsing process, ignore irrelevant content, and improve the accuracy and speed of task execution. This enables the humanoid robot to more efficiently utilize the useful information in the task description when facing complex tasks, optimize information processing, and achieve task key point recognition.

[0190] According to the priority of tasks and the availability of resources, the humanoid robot reasonably schedules and executes multiple assembly tasks, optimizes the task execution order, reduces resource waste in the production process, and thus improves the overall working efficiency of the flexible assembly production line.

[0191] In addition, during the task execution process, the humanoid robot can dynamically adjust the operation steps according to real-time feedback, adapt to changes in the assembly environment and task requirements, and ensure efficient operation in a complex production environment. This can significantly enhance the flexibility and adaptability of the production line and strengthen the self-adaptability of task planning and execution.

[0192] Refer to Figure 4 , Figure 4 It provides a flowchart of the overall operation of the chain of thought. Using a hierarchical chain of thought structure, complex tasks are refined layer by layer, and combined with multi-modal perception data for logical verification to ensure the rationality and accuracy of the task decomposition process.

[0193] Exemplarily, it receives a natural language instruction input by the operator, and preliminarily parses the natural language instruction through the task input and preliminary parsing module to generate a preliminary semantic representation. Then, using the hierarchical chain of thought structure, the task is parsed layer by layer. When parsing the task, the system combines the input multi-modal data (such as natural language description, visual information) and context information, and dynamically calculates the weight of each input through the attention mechanism, giving higher weight to the information related to the current task node to optimize the task parsing result.

[0194] Specifically, in the task parsing stage, the system adopts an improved hierarchical chain of thought, combines the attention mechanism, and decomposes and understands the task layer by layer. The implementation details are as follows: The main task objectives in the instruction are extracted through the high-level task parsing module, and corresponding high-level task nodes are generated. Among them, the high-level task nodes include assembly objectives, operation requirements, and execution conditions. Then, the high-level task nodes are further parsed through the sub-task recursive parsing module to generate multiple middle-level tasks and basic operation steps, forming a complete task tree structure. Each node in the task tree contains the task objective, execution steps, and their dependencies.

[0195] Optionally, after the task tree is generated, the system displays the results of task parsing in real time through the human-computer interaction interface. The operator can view the decomposition steps and execution status of the task on the interface, and confirm or modify the adjustment suggestions proposed by the system.

[0196] It should be noted that in the aforementioned parsing process, the dynamic weight adjustment of the key parts in the instruction is combined with the attention mechanism, and higher weight is given to the information related to the current task node, so that the task parsing process gives priority to the features related to the current task node to optimize the task parsing result.

[0197] Furthermore, during the task parsing process of some embodiments, the weighted feature vectors generated by the multi-modal attention mechanism can also be combined, enabling the humanoid robot to simultaneously process natural language instructions, visual information, and tactile information, achieving the fusion parsing of multi-modal data, thereby improving the accuracy of task parsing and the reliability of execution.

[0198] Next, the generated task tree is logically verified by the task verification and optimization module, and the generated task steps are compared by analyzing the domain knowledge base to exclude unreasonable operation steps, ensuring the rationality of task dependencies and execution conditions, and using the attention mechanism to optimize the task execution plan. The arrangement of nodes in the task tree is optimized according to task priorities and resource requirements to optimize the execution efficiency and generate the final assembly plan.

[0199] The assembly plan is sent to the task execution module to drive the humanoid robot to perform assembly according to the assembly plan.

[0200] It can be understood that during the task execution process, the environmental changes during the assembly process of the humanoid robot can also be monitored in real time, the focus of task parsing can be readjusted using the attention mechanism, and the task tree structure can be dynamically updated according to the detected abnormal situations.

[0201] In this embodiment, different from the existing fixed rule or template-driven flexible assembly methods that can only adapt to specific production environments and are difficult to cope with diverse production requirements, this embodiment uses the thought chain task parsing method to endow the humanoid robot with broader task understanding and execution capabilities. It can flexibly handle different types of assembly tasks in a variety of production scenarios. Whether it is a small-batch, multi-variety production environment or a production line that requires real-time adjustment of task strategies, the intelligent task parsing and assembly execution can be achieved based on the method of this embodiment. During the assembly process, the attention mechanism can focus on key steps, such as the grasping and installation of components, thereby reducing operation errors caused by external interference. Compared with traditional fixed process systems, the method of this embodiment has strong adaptability, significantly improves the application breadth of flexible manufacturing, and can perform well in assembly tasks with high-precision requirements, ensuring the assembly quality and reliability.

[0202] In some embodiments, when faced with scheduling multiple tasks to be executed in a multi-task environment, an optimal task scheduling strategy is generated through a reinforcement learning algorithm according to the priorities, urgency levels, and required resources of the tasks to ensure the efficient utilization of the humanoid robot's resources. During the task scheduling process, the reinforcement learning algorithm continuously optimizes the task priority sorting and resource allocation strategies by learning from the feedback of multi-task execution, enabling the humanoid robot to operate in parallel in a multi-task environment and improving the overall assembly efficiency.

[0203] In some embodiments, to verify the humanoid robot flexible assembly system and method based on the chain of thought of the present solution, appropriate software and hardware support can also be provided in the experimental environment for verification experiments.

[0204] Specifically, the hardware environment includes:

[0205] (1) Humanoid robot: A robotic arm with more than 6 degrees of freedom, used to simulate human grasping, assembly, etc. operations, and equipped with vision and force sensors to perceive the environment.

[0206] (2) Control host: Configured with a high-performance CPU and a GPU supporting deep learning computing (such as NVIDIA RTX4090), used to run the task parsing model and control algorithms.

[0207] (3) Workbench and components: Set up an experimental workbench to store the components and tools to be assembled, simulating the actual production environment.

[0208] The software environment includes:

[0209] (1) Operating system: Use the Linux system (such as Ubuntu 20.04) to ensure hardware compatibility and deep learning support.

[0210] (2) Development platform: Use PyTorch or TensorFlow for model training, combined with the Robot Operating System (ROS) for robot control and sensor data processing.

[0211] (3) Large model: Select a pre-trained language model (such as GPT-4) and fine-tune it for parsing task instructions.

[0212] Before the verification experiment starts, prepare the task instruction data and multi-modal data in advance. Among them, the task instruction data contains the natural language descriptions of different assembly tasks and their parsing results, used to train the model; the multi-modal data contains the visual and force data collected during the operation of the humanoid robot, used to enhance the model's perception ability during the assembly process.

[0213] After the model training is completed, according to the space requirements of the experimental site, prepare a 3x3-meter experimental area to ensure that there is enough space for the humanoid robot to operate, and prepare corresponding safety measures, set up an emergency stop button and an isolation area to prevent safety risks caused by misoperation of the humanoid robot during the experiment.

[0214] In the experimental site, including the robot work area, workbench, and component storage area, after confirming that the experimental area is clean and has enough space for movement, start the humanoid robot and the control host, connect devices such as vision sensors and force sensors, and check the working status of each device to ensure the normal operation of the hardware.

[0215] When it is confirmed that the working status of each device is normal and the hardware is running properly, load the pre-trained and fine-tuned large language model (such as GPT-4) and multi-modal fusion model on the control host, start ROS, and ensure smooth communication between modules.

[0216] First, perform data input and task instruction issuing operations. The operator inputs the assembly task instructions through natural language, such as "Install part A onto component B and perform precision calibration". The control host parses the natural language instructions through a multi-modal encoder and generates a high-dimensional feature representation.

[0217] Next, use the improved chain of thought structure and attention mechanism to parse the input task instructions and generate a complete task tree structure, including high-level task objectives and specific operation steps.

[0218] Then, based on the parsed task tree, the system conducts logical verification and path planning for the task, generating a specific assembly plan and a sequence of robot actions. Utilize the path planning function in ROS to plan the movement trajectory of the humanoid robot from the part storage area to the workbench and generate specific assembly action instructions, such as "Move to the position of part A", "Grasp part A", "Install part A onto component B", etc.

[0219] Finally, the humanoid robot gradually executes each task step according to the generated assembly plan, uses a vision sensor to confirm the part position, and uses a force sensor to monitor the force during the assembly process. At the same time, during the execution of the assembly task, real-time feedback of assembly status data (such as force perception data, position data) is sent to the control host. If an abnormality is detected during the assembly process (such as position deviation or abnormal assembly force), the system re-evaluates the task focus through the attention mechanism, adjusts the assembly actions, and continues to execute the remaining steps.

[0220] During the entire experiment process, record the execution time of each task step, the time-consuming of path planning, the assembly error situation, sensor data, etc., and record the task tree structure automatically generated by the system during the task parsing process, as well as the weight change situation adjusted through the attention mechanism, and analyze the accuracy of the parsing. If assembly failure or task interruption occurs, record the cause of the failure and analyze the effect of adaptive adjustment.

[0221] After the above verification experiment, compare the consistency between the actual completion of the assembly task by the humanoid robot and the expected task plan, so as to evaluate the accuracy of task parsing and the stability of execution. Analyze the scheduling efficiency and assembly quality during multi-task execution, including the priority assignment and resource usage of each task. Based on the experimental data, evaluate the effect of the system's adaptive adjustment ability in a dynamic environment, such as the adjustment time and adjustment success rate after detecting an abnormality.

[0222] After the experiment is completed, stop the operation of the humanoid robot, close all experimental processes on the control host, and organize and save the data files and logs generated during the experiment. Inspect and maintain the experimental equipment to ensure that the next experiment can proceed normally.

[0223] According to the experimental results, fine-tune and improve the model to enhance the application effect of the system in the actual flexible manufacturing scenario.

[0224] Refer to Figure 5 , this application embodiment also provides a virtual system for flexible assembly of humanoid robots based on the chain of thought, which can implement the above-mentioned method for flexible assembly of humanoid robots based on the chain of thought. The system includes:

[0225] The first module is used to obtain the assembly task instructions provided by the user.

[0226] The second module is used to perform task parsing on the assembly task instructions through the chain of thought model, calculate the weight of each word in the assembly task instructions through the attention mechanism during the parsing process, dynamically adjust the parsing target related to the current task according to the weight, and decompose the task layer by layer into multiple subtasks according to the parsing target to obtain a task tree structure.

[0227] The third module is used to traverse the task tree structure to determine the assembly plan, where the assembly plan includes the execution arrangement sequence, task path planning, and assembly action parameters.

[0228] The fourth module is used to control the humanoid robot to perform assembly operations according to the assembly plan.

[0229] It can be understood that the content in the above method embodiments is applicable to this virtual system embodiment. The functions specifically implemented by this virtual system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those in the above method embodiments.

[0230] This application embodiment also provides a hardware system for flexible assembly of humanoid robots based on the chain of thought. The hardware system includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the above-mentioned method for flexible assembly of humanoid robots based on the chain of thought. The electronic device can be any intelligent terminal including a tablet computer, in-vehicle computer, etc.

[0231] It can be understood that the content in the above method embodiments is applicable to this hardware system embodiment. The functions specifically implemented by this hardware system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those in the above method embodiments.

[0232] Refer to Figure 6 , Figure 6Schematically shows the hardware structure of a thought-chain-based humanoid robot flexible assembly hardware system according to another embodiment. The hardware system includes:

[0233] A processor 901, which can be implemented in ways such as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0234] A memory 902, which can be implemented in forms such as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When implementing the technical solutions provided in the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 902 and are called by the processor 901 to execute the thought-chain-based humanoid robot flexible assembly method of the embodiments of the present application.

[0235] An input / output interface 903, which is used to implement information input and output.

[0236] A communication interface 904, which is used to implement communication interaction between this device and other devices, and can achieve communication through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0237] A bus 905, which transmits information between various components of the device (such as the processor 901, the memory 902, the input / output interface 903, and the communication interface 904).

[0238] Among them, the processor 901, the memory 902, the input / output interface 903, and the communication interface 904 achieve communication connections with each other inside the device through the bus 905.

[0239] The embodiments of the present application also provide a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned thought-chain-based humanoid robot flexible assembly method.

[0240] It can be understood that the content in the above method embodiments is applicable to the embodiments of this storage medium. The functions specifically implemented by the embodiments of this storage medium are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those of the above method embodiments.

[0241] As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory optionally includes a memory remotely disposed relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above networks include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0242] The embodiments described in the embodiments of the present application are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art will know that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present application are equally applicable to similar technical problems.

[0243] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than those shown in the figures, or combine certain steps, or different steps.

[0244] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0245] Those of ordinary skill in the art can understand that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices can be implemented as software, firmware, hardware, and appropriate combinations thereof.

[0246] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present application described here can be implemented in an order other than those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units does not have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0247] The preferred embodiments of the embodiments of the present application have been described above with reference to the accompanying drawings. This does not limit the scope of the rights of the embodiments of the present application. Any modifications, equivalent replacements, and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall fall within the scope of the rights of the embodiments of the present application.

Claims

1. A flexible assembly method for humanoid robots based on the chain of thought, characterized in that The method includes the following steps: Obtain the assembly task instructions provided by the user; Perform task parsing on the assembly task instructions through a chain of thought model, and calculate the weight of each word in the assembly task instructions through an attention mechanism during the parsing process. Dynamically adjust the parsing objectives related to the current task according to the weights, and layer by layer decompose the task into multiple subtasks based on the parsing objectives to obtain a task tree structure; Traverse the task tree structure to determine the assembly plan, where the assembly plan includes the execution arrangement order, task path planning, and assembly action parameters; Control the humanoid robot to perform assembly operations according to the assembly plan; The obtaining of the assembly task instructions provided by the user includes the following steps: Obtain the multimodal data provided by the user, where the multimodal data includes text, speech, and visual information, and the visual information includes gestures; Use natural language processing techniques to perform word segmentation, part-of-speech tagging, and semantic parsing on the text and speech, and extract the main content and intention of the task to obtain text information; Extract the shape and motion trajectory of the gesture to obtain gesture features, classify and recognize the gesture features, and convert the gesture features into symbolic information; Map the text information and the symbolic information to a vector space to obtain the assembly task instructions; The calculating of the weight of each word in the assembly task instructions through the attention mechanism and dynamically adjusting the parsing objectives related to the current task according to the weights includes the following steps: Calculate the attention score of each word in the assembly task instructions based on the context information through the attention mechanism, where the context information includes the task historical state of the previous time step and the domain knowledge related to the task; Perform normalization processing on the attention scores to obtain the weights of the words; Determine the words with weights greater than a preset threshold as the parsing objectives related to the current task; Among them, the scoring function e of the attention mechanism i The expression is: e i = Score(h i , S t-1 ) = v T tanh(W h h i + W s S t-1 + W k k i ); where h i is the hidden state of the i-th word in the input sequence; S t-1 represents the decoder state at the previous time step, reflecting the historical context of the model, and k i is the domain knowledge vector, representing the domain knowledge related to the task; W h , W s , W k represent weight matrices, which are parameters to be optimized through learning; v is a learnable weight vector used to perform weighted aggregation on the output of the tanh function.

2. The method according to claim 1, wherein The traversing of the task tree structure to determine the assembly plan includes the following steps: Traverse the task tree structure to determine the sequence and dependency of the subtasks; According to the sequence and dependency of the subtasks, perform logical verification on the task tree structure, and optimize the arrangement of the subtasks in the task tree structure based on the attention mechanism to obtain the execution arrangement order; Determine the task path planning according to the execution arrangement order using a motion planning algorithm; Decompose the subtasks until they are decomposed into multiple basic actions that the humanoid robot can directly execute, and determine the required assembly action parameters according to the basic actions; Obtain the assembly plan according to the execution arrangement order, the task path planning, and the assembly action parameters.

3. The method according to claim 2, wherein The performing of logical verification on the task tree structure according to the sequence and dependency of the subtasks, and optimizing the arrangement of the subtasks in the task tree structure based on the attention mechanism to obtain the execution arrangement order includes the following steps: When there are multiple subtasks to be executed simultaneously, evaluate the priority and resource requirements of each subtask through the attention mechanism to obtain an evaluation result; Determine a task scheduling policy through a reinforcement learning algorithm according to the evaluation result, and adjust the arrangement of subtasks in the task tree structure according to the task scheduling policy to obtain the execution arrangement order.

4. The method according to claim 1, wherein The flexible assembly method for a humanoid robot based on the chain of thought further includes the following steps: Obtain monitoring data, where the monitoring data includes environmental information and assembly status data; Judge whether there is an abnormal situation in the currently executed subtask according to the monitoring data; When there is an abnormal situation in the subtask, re-parse the subtask through an attention mechanism and dynamically adjust the corresponding assembly plan of the subtask.

5. The method according to claim 4, wherein The flexible assembly method for a humanoid robot based on the chain of thought further includes the following steps: Record the task tree structure and the assembly status data to obtain an assembly process result; Display the assembly process result through an interaction interface.

6. A flexible assembly virtual system for humanoid robots based on the chain of thought, characterized in that, The virtual system includes: A first module for obtaining an assembly task instruction provided by a user; A second module for parsing the assembly task instruction through a chain of thought model, calculating the weight of each word in the assembly task instruction through an attention mechanism during the parsing process, dynamically adjusting the parsing target related to the current task according to the weight, and decomposing the task layer by layer into multiple subtasks according to the parsing target to obtain a task tree structure; A third module for traversing the task tree structure to determine an assembly plan, where the assembly plan includes an execution arrangement order, a task path planning, and assembly action parameters; A fourth module for controlling the humanoid robot to perform an assembly operation according to the assembly plan; The obtaining of the assembly task instruction provided by the user includes the following steps: Obtain multi-modal data provided by the user, where the multi-modal data includes text, voice, and visual information, and the visual information includes gestures; Use natural language processing technology to perform word segmentation, part-of-speech tagging, and semantic parsing on the text and voice, extract the main content and intention of the task to obtain text information; Extract the shape and motion trajectory of the gesture to obtain gesture features, classify and recognize the gesture features, and convert the gesture features into symbolic information; Map the text information and the symbolic information to a vector space to obtain the assembly task instruction; The calculating of the weight of each word in the assembly task instruction through an attention mechanism and dynamically adjusting the parsing target related to the current task according to the weight includes the following steps: Calculate the attention score of each word in the assembly task instruction based on context information through an attention mechanism, where the context information includes the task historical state of the previous time step and domain knowledge related to the task; Perform normalization processing on the attention score to obtain the weight of the word; Determine the words with weights greater than a preset threshold as the parsing targets related to the current task; Among them, the scoring function e of the attention mechanism i The expression is: e i = Score(h i , S t-1 ) = v T tanh(W h h i + W s S t-1 + W k k i ); where h i is the hidden state of the i-th word in the input sequence; S t-1 represents the decoder state at the previous time step, reflecting the historical context of the model, k i is the domain knowledge vector, representing the domain knowledge related to the task; W h , W s , W k represent weight matrices, which are parameters to be optimized through learning; v is a learnable weight vector used to weighted sum the output of the tanh function.

7. A flexible assembly hardware system for a humanoid robot based on the chain of thought, characterized in that, The hardware system includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the method according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, the method according to any one of claims 1 to 5 is implemented.

Images