Humanoid robot flexible assembly method and system based on thinking chain and medium

Abstract

The embodiment of the invention provides a humanoid robot flexible assembly method and system based on a thinking chain and a medium, and belongs to the technical field of natural language processing. According to the scheme, the assembly task instruction provided by the user is obtained, the assembly task instruction is subjected to task analysis through the thinking chain model, the weight of each word in the assembly task instruction is calculated through the attention mechanism in the analysis process, and the analysis target related to the current task is dynamically adjusted according to the weight; the task is decomposed into a plurality of subtasks layer by layer according to an analysis target, a task tree structure is obtained, the task tree structure is traversed to determine an assembly plan, the assembly plan comprises an execution arrangement sequence, task path planning and assembly action parameters, and the humanoid robot is controlled to execute assembly operation according to the assembly plan. The task information is automatically extracted by combining a hierarchical thinking chain and an attention mechanism and is converted into assembly operation, a clear execution plan is generated, and the flexibility of task analysis is improved.

Description

Technical Field

[0001] The present application relates to the field of natural language processing technology, and in particular to a method, system and medium for flexible assembly of a humanoid robot based on a thought chain. Background Art

[0002] As the manufacturing industry develops towards intelligence and flexibility, the traditional fixed process assembly system can no longer meet the needs of diversified and customized production. As a solution to complex and changing production environments, flexible manufacturing systems can quickly adapt to different types of assembly tasks and product changes, and improve the efficiency and flexibility of production lines. In this process, humanoid robots have gradually become an important role in the field of flexible manufacturing due to their human-like operation methods and flexible working capabilities.

[0003] However, current humanoid robots still face multiple challenges in complex task parsing and autonomous assembly. For example, when faced with new tasks, the task process often needs to be reprogrammed or adjusted, which not only increases the system development time, but also increases the threshold and cost of use. Traditional robot task allocation and planning mostly rely on pre-written programs or fixed instruction templates, with insufficient task parsing capabilities and difficulty in understanding complex instructions. This method is too rigid and inflexible when dealing with changing assembly tasks, and cannot meet the requirements of rapid adjustment and real-time response. At the same time, when planning and executing tasks, there is often a lack of in-depth understanding and adaptive capabilities of complex assembly processes, which requires operators to spend a lot of time on task guidance and parameter adjustment, greatly limiting the level of automation of the production line. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a humanoid robot flexible assembly method, system and medium based on thought chain, aiming to reduce development and operation costs without relying on pre-written programs or fixed instruction templates, automatically and accurately extract key information from complex instructions, generate clear execution plans, and realize intelligent flexible assembly.

[0005] To achieve the above-mentioned purpose, one aspect of an embodiment of the present application proposes a humanoid robot flexible assembly method based on a thought chain, the method comprising the following steps:

[0006] Obtaining assembly task instructions provided by the user;

[0007] The assembly task instruction is parsed through the thought chain model, and the weight of each word in the assembly task instruction is calculated through the attention mechanism during the parsing process, and the parsing target related to the current task is dynamically adjusted according to the weight, and the task is decomposed into multiple subtasks layer by layer according to the parsing target to obtain a task tree structure;

[0008] Traversing the task tree structure to determine an assembly plan, wherein the assembly plan includes an execution arrangement sequence, task path planning, and assembly action parameters;

[0009] The humanoid robot is controlled to perform assembly operations according to the assembly plan.

[0010] In some embodiments, the step of obtaining the assembly task instruction provided by the user comprises the following steps:

[0011] Acquiring multimodal data provided by a user, wherein the multimodal data includes text, voice, and visual information;

[0012] Use natural language processing technology to segment text and speech, perform part-of-speech tagging and semantic analysis, extract the main content and intention of the task to obtain text information;

[0013] Using computer vision technology to extract and identify features of the visual information to obtain symbol information;

[0014] The text information and the symbol information are mapped to a vector space to obtain the assembly task instruction.

[0015] In some embodiments, the weight of each word in the assembly task instruction is calculated by the attention mechanism, and the parsing target related to the current task is dynamically adjusted according to the weight, including the following steps:

[0016] Calculating an attention score for each word in the assembly task instruction based on context information through an attention mechanism, wherein the context information includes task history status and domain knowledge;

[0017] Normalizing the attention score to obtain the weight of the word;

[0018] The words whose weights are greater than a preset threshold are determined as the parsing targets related to the current task.

[0019] In some embodiments, traversing the task tree structure to determine the assembly plan includes the following steps:

[0020] Traversing the task tree structure to determine the order and dependency of subtasks;

[0021] According to the sequence and dependency relationship of the subtasks, the task tree structure is logically verified, and the arrangement of the subtasks in the task tree structure is optimized based on the attention mechanism to obtain the execution arrangement order;

[0022] Determine the task path planning using a motion planning algorithm according to the execution arrangement sequence;

[0023] Decomposing the subtasks into a plurality of basic actions that can be directly executed by the humanoid robot, and determining the required assembly action parameters according to the basic actions;

[0024] An assembly plan is obtained according to the execution arrangement sequence, the task path planning and the assembly action parameters.

[0025] In some embodiments, the task tree structure is logically verified according to the sequence and dependency of the subtasks, and the arrangement of the subtasks in the task tree structure is optimized based on the attention mechanism to obtain the execution arrangement sequence, including the following steps:

[0026] When there are multiple subtasks to be executed at the same time, the priority and resource requirements of each subtask are evaluated through the attention mechanism to obtain an evaluation result;

[0027] A task scheduling strategy is determined by a reinforcement learning algorithm according to the evaluation result, and the arrangement of subtasks in the task tree structure is adjusted according to the task scheduling strategy to obtain the execution arrangement order.

[0028] In some embodiments, the humanoid robot flexible assembly method based on thought chain further includes the following steps:

[0029] Acquiring monitoring data, wherein the monitoring data includes environmental information and assembly status data;

[0030] Determine whether there is any abnormality in the subtask currently being executed according to the monitoring data;

[0031] When an abnormal situation occurs in the subtask, the subtask is re-analyzed through the attention mechanism, and the assembly plan corresponding to the subtask is dynamically adjusted.

[0032] In some embodiments, the humanoid robot flexible assembly method based on thought chain further includes the following steps:

[0033] Recording the task tree structure and the assembly status data to obtain an assembly process result;

[0034] The assembly process results are displayed through an interactive interface.

[0035] To achieve the above-mentioned purpose, another aspect of the embodiment of the present application proposes a humanoid robot flexible assembly virtual system based on thought chain, the virtual system comprising:

[0036] The first module is used to obtain the assembly task instructions provided by the user;

[0037] The second module is used to perform task parsing on the assembly task instruction through a thought chain model, and calculate the weight of each word in the assembly task instruction through an attention mechanism during the parsing process, dynamically adjust the parsing target related to the current task according to the weight, and decompose the task layer by layer into multiple subtasks according to the parsing target to obtain a task tree structure;

[0038] A third module is used to traverse the task tree structure to determine an assembly plan, wherein the assembly plan includes an execution arrangement sequence, task path planning and assembly action parameters;

[0039] The fourth module is used to control the humanoid robot to perform assembly operations according to the assembly plan.

[0040] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application proposes a humanoid robot flexible assembly hardware system based on a thinking chain, wherein the hardware system includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned method when executing the computer program.

[0041] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0042] The embodiments of the present application include at least the following beneficial effects: The present application provides a humanoid robot flexible assembly method, system and medium based on thought chain. The scheme obtains assembly task instructions provided by the user, performs task parsing on the assembly task instructions through the thought chain model, and calculates the weight of each word in the assembly task instruction through the attention mechanism during the parsing process, dynamically adjusts the parsing target related to the current task according to the weight, decomposes the task layer by layer into multiple subtasks according to the parsing target, and obtains a task tree structure. It can automatically and accurately extract key information from complex instructions during the task parsing process by combining hierarchical thought chain and attention mechanism without relying on preset fixed instruction templates or rules, ignoring irrelevant content, adapting to the changing expressions in natural language, and improving the flexibility and accuracy of autonomous task parsing capabilities.

[0043] The assembly plan is determined by traversing the task tree structure, where the assembly plan includes the execution arrangement sequence, task path planning and assembly action parameters. The humanoid robot is controlled to perform assembly operations according to the assembly plan. The task can be converted into assembly operations, a clear execution plan can be generated, the development and operation costs of the flexible assembly system can be reduced, and the intelligent flexible assembly can be realized. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1is a flow chart of a humanoid robot flexible assembly method based on a thought chain provided in an embodiment of the present application;

[0045] Figure 2 It is a schematic diagram of the structure of the task tree provided in the embodiment of the present application;

[0046] Figure 3 It is a structural schematic diagram of a humanoid robot flexible assembly software system based on a thinking chain provided in an embodiment of the present application;

[0047] Figure 4 It is a flow chart of the overall operation of the thinking chain provided in the embodiment of the present application;

[0048] Figure 5 is a structural schematic diagram of a humanoid robot flexible assembly virtual system based on a thinking chain provided by another embodiment of the present application;

[0049] Figure 6 It is a schematic diagram of the hardware structure of the humanoid robot flexible assembly hardware system based on the thinking chain provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0051] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0052] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0054] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first explained.

[0055] (1) Hierarchical Chain of Thought (H-CoT): Hierarchical Chain of Thought technology breaks down complex tasks into sub-tasks at multiple levels. The chain structure enables the robot to gradually understand the overall framework and specific steps of the task.

[0056] (2) Attention Mechanism: It refers to a mechanism in which a system or model can selectively focus on part of the input information while ignoring other irrelevant or minor information when processing information. This mechanism simulates the way humans allocate attention when processing complex perceptual information, allowing the system or model to process and understand information more efficiently. In deep learning, the attention mechanism is usually implemented as a weight parameter allocation mechanism, which calculates the importance of different parts of information and assigns different weights accordingly, thereby helping the model capture relatively important information.

[0057] Next, some related technologies involved in the embodiments of the present application are explained.

[0058] In the current flexible manufacturing system, humanoid robots have the following key problems in task analysis and autonomous assembly, which seriously limit the efficiency and adaptability of flexible manufacturing systems.

[0059] (1) Insufficient task analysis ability, making it difficult to understand complex instructions.

[0060] At present, when humanoid robots receive assembly tasks, they mostly rely on pre-written instruction scripts or fixed template rules. Although this method can perform well in highly standardized assembly tasks, it cannot flexibly understand and parse new task instructions when faced with a changing and complex production environment, and it is difficult to cope with sudden changes and customized needs in production. This leads to the poor adaptability of humanoid robots to new tasks, requiring a lot of manual intervention, and it is difficult to achieve truly autonomous assembly.

[0061] (2) Task planning and execution lack flexibility and adaptability.

[0062] In related technologies, humanoid robots usually use fixed task planning processes and lack the ability to dynamically adjust tasks during the process. When the production environment or assembly tasks change, humanoid robots cannot optimize and adjust the assembly process according to real-time conditions. This leads to problems such as rigid movements and low efficiency in the execution of tasks, and they are unable to adapt to the requirements of flexible manufacturing for efficiency and flexibility.

[0063] (3) Insufficient use of information, unable to effectively distinguish task priorities.

[0064] Humanoid robot task planning methods usually fail to fully utilize the key information in the task description and are easily disturbed by irrelevant information, resulting in low task parsing efficiency. Especially in complex assembly tasks, humanoid robots need to understand a large amount of detailed information, but related technologies have difficulty in automatically identifying and prioritizing key information, thus affecting the overall task execution effect.

[0065] (4) The multi-task parallel processing capability is weak and the task scheduling efficiency is low.

[0066] In a flexible manufacturing environment, production lines need to handle multiple assembly tasks at the same time. Current humanoid robot systems have limited capabilities in task scheduling and priority allocation. Due to the lack of ability to analyze the importance and urgency of tasks, humanoid robots are often unable to reasonably arrange the execution order of tasks, resulting in waste of resources and reduced production efficiency.

[0067] In view of this, a humanoid robot flexible assembly method, system and medium based on thought chain are provided in the embodiments of the present application. The scheme obtains the assembly task instructions provided by the user, performs task analysis on the assembly task instructions through the thought chain model, and calculates the weight of each word in the assembly task instructions through the attention mechanism during the analysis process, dynamically adjusts the analysis target related to the current task according to the weight, decomposes the task layer by layer into multiple subtasks according to the analysis target, and obtains a task tree structure. It can automatically and accurately extract key information from complex instructions during the task analysis process by utilizing the combination of hierarchical thought chain and attention mechanism without relying on preset fixed instruction templates or rules, ignoring irrelevant content, adapting to the changing expressions in natural language, and improving the flexibility and accuracy of autonomous task analysis capabilities.

[0068] The assembly plan is determined by traversing the task tree structure. The assembly plan includes the execution arrangement sequence, task path planning and assembly action parameters. The humanoid robot is controlled to perform assembly operations according to the assembly plan. The task can be converted into assembly operations, and a clear execution plan can be generated, thereby reducing the development and operation costs of the flexible assembly system and realizing the intelligence of flexible assembly.

[0069] The flexible assembly method of a humanoid robot based on a thinking chain provided in an embodiment of the present application relates to the field of natural language processing technology. The flexible assembly method of a humanoid robot based on a thinking chain provided in an embodiment of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, and a vehicle terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and cloud servers for basic cloud computing services such as big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements a flexible assembly method of a humanoid robot based on a thinking chain, etc., but is not limited to the above forms.

[0070] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0071] Figure 1 is an optional flow chart of a humanoid robot flexible assembly method based on a thought chain provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S104.

[0072] Step S101, obtaining an assembly task instruction provided by a user.

[0073] It should be noted that in the flexible assembly of humanoid robots, the humanoid robots need to receive assembly task instructions provided by the user. Assembly task instructions are multimodal information provided by the operator to the humanoid robot, which are a series of instructions used to guide the humanoid robot to complete a specific assembly task. These instructions usually include the target object of assembly, the required parts, the assembly steps, and the accuracy requirements.

[0074] Step S102, perform task analysis on the assembly task instructions through the thought chain model, and calculate the weight of each word in the assembly task instructions through the attention mechanism during the analysis process, dynamically adjust the analysis target related to the current task according to the weight, and decompose the task layer by layer into multiple subtasks according to the analysis target to obtain a task tree structure.

[0075] In some embodiments, after obtaining the assembly task instructions, since the instructions often include a large amount of complex information, including information related to the current task, as well as redundant and irrelevant information, it is also necessary to perform task analysis on the assembly task instructions through a pre-trained thought chain model. The hierarchical thought chain decomposes complex tasks into subtasks layer by layer to form a tree structure that is easy for humanoid robots to gradually understand and execute. However, traditional thought chain technology lacks a flexible attention mechanism when dealing with diverse assembly tasks, and it is difficult to cope with the dynamic changes of information during the task process.

[0076] To solve this problem, this embodiment introduces an attention mechanism, which enables the humanoid robot to focus on key information and ignore redundant data during task parsing, thereby improving the accuracy and execution efficiency of task decomposition.

[0077] The assembly task instructions are parsed through the thought chain model, and the weight of each word in the assembly task instructions is calculated through the attention mechanism during the parsing process. The parsing target related to the current task is dynamically adjusted according to the weight, so that the humanoid robot can accurately extract key information from the operator's instructions and dynamically adjust the model's attention to the details of different tasks. The current task refers to the specific assembly task that the humanoid robot is processing or about to process. Then, the task is decomposed layer by layer into multiple subtasks according to the parsing target to obtain the task tree structure.

[0078] Specifically, first, the assembly task instructions are input into the thinking chain model for preliminary parsing of the instructions. During the parsing process, the attention mechanism helps the humanoid robot extract the main task objectives from complex instructions, obtain multiple high-level tasks, and use the high-level tasks as high-level nodes of the thinking chain. Since the instructions may contain multiple task objectives, the attention mechanism calculates the weight of each word in the instructions, and determines the key steps in the task according to the weight distribution to obtain the parsing objectives. The higher the weight, the more likely it is to be a key step in the current task. For example, the instruction is "Please install part A on component B, and then perform precision calibration", and the high-level tasks include "install part A to component B" and "precision calibration".

[0079] It should be noted that the attention mechanism can pre-train the large language model with a large amount of sample data related to the assembly task, so that the large language model can identify which steps are critical in a specific scenario or context.

[0080] It is understandable that the attention mechanism does not directly understand which information is the key point from the assembly task instructions, but automatically learns to identify common key steps in the task through the use of pre-trained models combined with training data in specific task fields. For example, after training with a large amount of assembly task data, the model can learn to automatically assign weights and dynamically determine the focus of the current task. The attention mechanism adjusts the model's attention to the input information based on these weights, thereby ensuring that the key content of the task is given higher weight and attention. For example, the trained thought chain model can recognize that operations such as "installing parts" and "precision calibration" are usually key steps in assembly tasks.

[0081] Through the pre-trained thought chain model and training data in the task field, the model can flexibly respond to different task instructions. The attention mechanism automatically determines which information in the instructions is the key to the current task based on the task input (i.e., assembly task instructions). Therefore, there is no need to preset fixed template rules.

[0082] Secondly, according to the multiple parsing targets obtained by parsing, each parsing target is further recursively decomposed to decompose the high-level task into more detailed subtasks to form middle-level nodes. In the subtask decomposition process, the attention mechanism guides the humanoid robot to focus on the details related to the current task, avoid being disturbed by irrelevant information, and refine the goals of the high-level task. For example, the high-level task "install part A to component B" can be decomposed into "obtain part A", "locate component B", and "install part A to component B".

[0083] Finally, after step-by-step decomposition, a set of high-level tasks and subtasks is obtained. The nodes are connected to each other through the task dependency relationship to form a task tree structure. In the task tree, each node represents a high-level task or subtask, and the connection between nodes represents the task dependency relationship.

[0084] Step S103, traversing the task tree structure to determine an assembly plan, wherein the assembly plan includes an execution arrangement sequence, task path planning, and assembly action parameters.

[0085] Specifically, the task tree structure is a multi-branch tree, in which each node can have any number of child nodes. By traversing the task tree, the assembly plan is determined according to the order and dependency of the tasks, where the assembly plan includes the execution arrangement order, task path planning and assembly action parameters.

[0086] Furthermore, in the process of generating assembly plans, the kinematic and dynamic constraints of the humanoid robot and various restrictions in the environment can also be considered. For example, in the process of generating task path planning, the motion planning algorithm is used to generate a feasible motion path to avoid collisions with obstacles; when generating assembly action parameters for grasping parts, the appropriate grasping force and posture are calculated to avoid damaging parts or generating assembly errors.

[0087] Step S104, controlling the humanoid robot to perform assembly operations according to the assembly plan.

[0088] Exemplarily, the humanoid robot executes each task step by step according to the generated assembly plan and the execution arrangement sequence, for example, in the order of "moving to the position of part A", "grabbing part A", and "installing part A to component B", according to the task path planning, it moves from the part storage area to the workbench, uses visual sensors to confirm the part position according to the assembly action parameters, and uses force sensors to monitor the force conditions during the assembly process to perform the assembly operation.

[0089] In this embodiment, by introducing a hierarchical thinking chain structure, the complex tasks are refined layer by layer, and combined with the attention mechanism, the humanoid robot can parse the assembly task instructions input by the operator, dynamically identify the key information therein, and automatically generate detailed task decomposition and assembly plans, which not only improves the flexibility of task parsing, but also greatly improves the accuracy of understanding task details, so that the humanoid robot can accurately perform various complex assembly tasks. Introducing the attention mechanism in the task parsing process can enable the humanoid robot to have adaptive adjustment capabilities without relying on fixed templates or predefined rules, solving the problem that the current robot assembly system is difficult to adapt to the changing expressions in natural language when processing diverse and complex task instructions.

[0090] In addition, compared with the current robot assembly system, when facing new tasks, it often needs to reprogram or adjust the task process, which not only increases the system development time, but also increases the threshold and cost of use. This embodiment combines a large language model and multimodal analysis capabilities to enable the humanoid robot to understand the operator's natural language description and automatically generate a task plan, greatly simplifying the task writing and adjustment process. The operator only needs to use simple instructions to complete the task assignment of the humanoid robot without writing complex scripts, thereby reducing the development and operation costs of the system.

[0091] In some embodiments, step S101 may include but is not limited to steps S201 to S204.

[0092] Step S201, obtaining multimodal data provided by a user, wherein the multimodal data includes text, voice and visual information.

[0093] It is understandable that user multimodal data is data composed of multiple information forms when the user provides assembly task instructions, including but not limited to describing the specific content and requirements of the assembly task in natural language (such as Chinese, English, etc.), or expressing intentions through multimodal information such as voice instructions and gestures and schematic diagrams. In order for the humanoid robot to fully understand this information, it is first necessary to obtain the multimodal data provided by the user so that the task input can be uniformly encoded later.

[0094] It should be noted that users can provide multimodal data in one or more ways according to actual needs and preferences, and the embodiments of the present application do not impose specific limitations.

[0095] Step S202, using natural language processing technology to perform word segmentation, part-of-speech tagging and semantic analysis on text and speech, extracting the main content and intention of the task to obtain text information.

[0096] Optionally, when the multimodal data includes text and speech, natural language processing technology is used to perform word segmentation, part-of-speech tagging and semantic analysis on the text and speech to extract the main content and intention of the task to obtain text information.

[0097] For example, the voice command is recognized, the voice signal is converted into text, and then the text is segmented and POS tagged using natural language processing technology to identify which words in the command are nouns, verbs, etc. Then, semantic analysis is performed to understand the overall meaning and intention of the command, which usually involves mapping the segmented text into a semantic framework, or using a deep learning model for semantic understanding. For example, in the instruction "Please install part A on component B", after segmentation, "please", "will", "part A", "install", "to", and "component B" are obtained. After semantic analysis, it is recognized that the main content of the task is "installation", and the key information "part A" and "component B" are extracted.

[0098] Step S203, using computer vision technology to extract and recognize features of visual information to obtain symbol information.

[0099] Optionally, when the multimodal data includes visual information such as gestures and schematic diagrams, computer vision technology is used to extract features of the visual information and convert the visual information into symbolic information that can be processed by a computer.

[0100] Exemplarily, the gesture image is first preprocessed (such as denoising, enhancement, etc.) to extract feature information in the gesture image such as shape, motion trajectory, etc., and then a recurrent neural network is used to classify and identify the extracted features to convert the features into symbolic information.

[0101] Step S204, mapping the text information and the symbol information to the vector space to obtain the assembly task instruction.

[0102] Specifically, the text information and symbolic information obtained by converting all multimodal data are mapped to a unified high-dimensional vector space through an embedding layer to form a multimodal feature representation of the task input, and the assembly task instructions are obtained, providing a basis for subsequent task analysis.

[0103] In step S204 of another embodiment, when the multimodal data consists only of text or language, natural language processing technology is used to perform word segmentation, part-of-speech tagging and semantic analysis on the text or speech to extract the main content and intention of the task to obtain text information, and the text information is mapped to the vector space to obtain the assembly task instructions.

[0104] Alternatively, when the multimodal data consists only of visual information, computer vision technology is used to extract and recognize features of the visual information to obtain symbolic information, which is then mapped to a vector space to obtain assembly task instructions.

[0105] In step S102 of some embodiments, the weight of each word in the assembly task instruction is calculated through the attention mechanism, and the parsing target related to the current task is dynamically adjusted according to the weight, which may include but is not limited to steps S301 to S303.

[0106] Step S301, calculating the attention score of each word in the assembly task instruction based on context information through an attention mechanism, wherein the context information includes task history status and domain knowledge.

[0107] Specifically, the attention mechanism was originally used in the field of neural machine translation to dynamically focus on different parts of the input sequence when processing sequence data. The introduction of the attention mechanism allows the model to automatically identify and focus on key information in the instructions during task parsing.

[0108] However, traditional scoring functions usually only use a simple matching method. In order to improve the understanding of complex instructions, this embodiment improves the scoring function, introduces context information in the attention scoring process, and calculates the attention score of each word in the assembly instruction based on the context information through the attention mechanism, wherein the context information includes the task history status of the previous time step and the domain knowledge related to the task.

[0109] The scoring function ei is expressed as formula (1):

[0110] e i =Score(h i , S t-1 )=v T tanh(W h h i+W s S t-1 +W k k i ) (1),

[0111] In formula (1), h i is the hidden state of the i-th word in the input sequence, usually the representation obtained by the encoder; S t-1 represents the decoder state of the previous time step, reflecting the historical context of the model, k i is a domain knowledge vector, representing the domain knowledge related to the task; W h , W s , W k represents the weight matrix, which is a parameter that needs to be optimized through learning; v is a learnable weight vector used to perform weighted aggregation on the output of the tanh function.

[0112] For example, first, the assembly task instruction of the task input is received and the task status of the previous step is obtained, and the domain knowledge base is called to obtain the background information related to the current task. Then, the scoring function is used to score the input information word by word or fragment by fragment to evaluate the importance of each word.

[0113] It should be noted that, when the instruction is parsed for the first time, an initial value or a default value can be set for the decoder as the decoder state of the previous time step, and scored together with the currently input word through a scoring function.

[0114] Step S302, normalize the attention score to obtain the word weight.

[0115] Specifically, during the task decomposition process at each level, the humanoid robot normalizes the attention score of each word and calculates the weight of each word.

[0116] Weight α i The calculation formula is as follows:

[0117]

[0118] In formula (2), α i represents the attention weight of the i-th word, e i is a scoring function used to evaluate the importance of the current word to task parsing, n is the length of the input sequence, and the scoring function e j Calculate the importance of the jth word in the input sequence for task parsing.

[0119] For example, for the instruction "Please install part A on component B, and then perform accuracy calibration", the attention mechanism will assign higher weights to "install part A on component B" and "perform accuracy calibration".

[0120] Step S303: determine the words whose weights are greater than a preset threshold as parsing targets related to the current task.

[0121] Optionally, based on the calculated weights, the words in the assembly task instructions with weights greater than a preset threshold are determined as parsing targets related to the current task. For example, "installing part A" has a higher weight than other irrelevant descriptions because it is related to "assembly", and the attention mechanism determines that this is a key step in the current task through this weight.

[0122] In step S303 of another embodiment, the words may be sorted by size according to the calculated weights, and the words with sequence numbers within a preset number may be determined as the parsing targets of the current task.

[0123] In this embodiment, the system does not need to establish a fixed subtask library for matching. On the contrary, by introducing the attention mechanism, the model can flexibly respond to different task instructions during the task parsing process, automatically identify and focus on the key information in the instructions based on the task input and context, and assign weights to each step. The higher the weight of the step, the more likely it is to be the key step in the current task, thereby improving the humanoid robot's understanding of complex assembly tasks.

[0124] In some embodiments, step S103 may include but is not limited to steps S401 to S405.

[0125] Step S401, traverse the task tree structure to determine the sequence and dependency relationship of subtasks.

[0126] Specifically, common traversal methods for multi-branch trees include depth-first traversal and breadth-first traversal. Depth-first traversal refers to traversing from the root node along each branch of the tree to the child nodes, and then traversing other branches. Depth-first traversal is divided into pre-order traversal and post-order traversal. Pre-order traversal means visiting the root node before the child nodes, and post-order traversal means giving priority to child nodes during the access process; breadth-first traversal starts from the root node and traverses from top to bottom in layers.

[0127] Reference Figure 2 , given an N-ary tree, for the entire tree, node 1 is the root of node 3, node 2 and node 4, and node 3 is the root of node 5 and node 6.

[0128] When the task tree is traversed using pre-order traversal, the access order is node 1, node 3, node 5, node 6, node 2, and node 4.

[0129] When the task tree is traversed using post-order traversal, the access order is node 5, node 6, node 3, node 2, node 4, node 1.

[0130] When the task tree is traversed using breadth-first traversal, the access order is node 1, node 3, node 2, node 4, node 5, and node 6.

[0131] The order and dependency of subtasks are determined by traversing all root nodes and child nodes of the task tree structure.

[0132] Step S402, according to the sequence and dependency relationship of the subtasks, the task tree structure is logically verified, and the arrangement of the subtasks in the task tree structure is optimized based on the attention mechanism to obtain the execution arrangement order.

[0133] Specifically, the task tree structure is logically verified according to the sequence and dependency relationships of the subtasks. The generated task steps are compared by analyzing the domain knowledge base to eliminate unreasonable operation steps, ensure the rationality of the task dependencies and execution conditions during the task decomposition process, and optimize the arrangement of subtasks in the task tree structure based on the attention mechanism. The order of nodes in the task tree is adjusted to optimize the execution efficiency and obtain the execution arrangement order.

[0134] Step S403, determining the task path planning using a motion planning algorithm according to the execution arrangement sequence.

[0135] Specifically, the system utilizes a motion planning algorithm (such as a fast random number algorithm or an A* algorithm) in combination with the joint constraints of the humanoid robot and an environmental map according to the execution arrangement order to generate a feasible motion path to determine the task path planning.

[0136] Step S404, decomposing the subtasks until they are decomposed into a plurality of basic actions that can be directly executed by the humanoid robot, and determining the required assembly action parameters according to the basic actions.

[0137] Specifically, after determining the task execution order and task path, the attention mechanism is used to continue to decompose the subtasks, decomposing each subtask into specific operations that need to be performed on each task node, until it is decomposed into multiple basic actions that can be directly executed by the humanoid robot, and the required assembly action parameters such as position information, force control parameters, etc. are determined based on the basic actions.

[0138] For example, a force control algorithm is used to ensure that the humanoid robot applies appropriate force during the assembly process to avoid damaging parts or causing assembly errors.

[0139] Step S405, obtaining an assembly plan according to the execution arrangement sequence, task path planning and assembly action parameters.

[0140] Specifically, the execution arrangement sequence, task path planning and assembly action parameters are integrated together to form a complete assembly plan. The assembly plan describes in detail all the steps and actions that the humanoid robot needs to perform in the entire assembly process, as well as the specific parameters of each step and action, so that during the execution process, the humanoid robot can be controlled to complete each subtask step by step according to the planned path and action sequence.

[0141] In some embodiments, step S402 may include but is not limited to steps S501 to S502.

[0142] Step S501, when there are multiple subtasks to be executed at the same time, the priority and resource requirements of each subtask are evaluated through the attention mechanism to obtain an evaluation result.

[0143] Specifically, when there are multiple subtasks to be executed at the same time, all subtasks to be executed at the same time are analyzed as a set, and the priority and resource requirements of each subtask are evaluated through the attention mechanism to obtain the evaluation result.

[0144] Step S502, determining a task scheduling strategy through a reinforcement learning algorithm according to the evaluation result, and adjusting the arrangement of subtasks in the task tree structure according to the task scheduling strategy to obtain an execution arrangement order.

[0145] Furthermore, the task scheduling strategy is determined based on the evaluation results through a reinforcement learning algorithm. For example, when two subtasks both need to use the same tool, it is necessary to arrange their execution order or find alternative solutions. The arrangement of subtasks in the task tree structure is dynamically adjusted according to the task scheduling strategy, and new tasks are adaptively added or the original subtask order is changed to obtain the execution arrangement order.

[0146] Exemplarily, reinforcement learning algorithms (such as Q-learning or deep reinforcement learning DQN) dynamically adjust and optimize multi-task scheduling strategies based on task priorities, resource requirements, task dependencies, etc. in evaluation results through continuous learning and feedback loops to ensure reasonable resource allocation and maximize overall efficiency.

[0147] First, the current task execution status, resource usage and task dependencies are encoded into a state vector as the input of reinforcement learning. This state information includes the current progress of each task, the resource requirements of the task, the availability of resources, etc.

[0148] Secondly, determine the action space, which is defined as the scheduling decisions that can be taken in each state. For example, you can choose which task to execute first, suspend a task, or change the order of tasks. Since the choice of each action will affect the smooth execution of the task, reinforcement learning evaluates the effect of each scheduling decision through a reward mechanism, setting a reward value for each action. The higher the efficiency of the scheduling, the greater the reward. The reward can be based on multiple indicators, such as task completion time, resource utilization, conflict reduction, etc. If the scheduling is successfully optimized, the task parallelism is improved, and the waiting time is reduced, a higher reward will be obtained.

[0149] The reinforcement learning algorithm tries different scheduling strategies and learns which scheduling decisions are the best solutions based on the reward feedback under different states. After repeated execution, the scheduling strategy is updated so that it can be efficiently scheduled when encountering the same or similar tasks in the future.

[0150] Understandably, reinforcement learning algorithms allow the system to make real-time adjustments based on feedback during task execution. If new tasks are added or existing tasks change, the system will dynamically evaluate the new state and select an appropriate scheduling strategy. Through continuous exploration and adjustment, the system gradually learns the optimal multi-task scheduling method.

[0151] In this embodiment, by combining the thinking chain analysis results and the reinforcement learning algorithm, an intelligent multi-task scheduling mechanism is established, and the attention mechanism is used to evaluate the priority and resource requirements of each task, and the execution order of the tasks is optimized, so that high-priority tasks are processed in a timely manner and resources are arranged reasonably. This multi-task scheduling strategy improves the parallel processing capability of humanoid robots and improves the overall work efficiency of flexible assembly production lines. It can effectively solve the current robot system in a multi-task environment. Due to the lack of task priority evaluation and scheduling capabilities, it leads to resource waste and production bottlenecks.

[0152] In some embodiments, the humanoid robot flexible assembly method based on thought chain may also include but is not limited to steps S601 to S603.

[0153] Step S601: Acquire monitoring data, wherein the monitoring data includes environmental information and assembly status data.

[0154] Specifically, assembly tasks are often complex and uncertain, and changes in the environment may affect the smooth execution of the task. Therefore, it is necessary to monitor in real time through a variety of sensors to obtain monitoring data so that the system has the ability to provide real-time feedback and adaptive adjustment. The detection data includes environmental information and assembly status information. For example, a visual sensor can detect the position of a part to obtain visual data, and a force sensor can sense the force during the assembly process to obtain tactile data.

[0155] Step S602: determine whether there is any abnormality in the currently executed subtask based on the monitoring data.

[0156] Optionally, it is determined whether there is any abnormality in the subtask currently being executed based on monitoring data obtained by multiple sensors.

[0157] Exemplarily, whether there is a position deviation is determined based on the position of the part monitored by the visual sensor, and whether the force exceeds or is lower than a preset safety range is determined based on the force conditions sensed by the force sensor.

[0158] Step S603: When an abnormal situation occurs in a subtask, the subtask is re-analyzed through the attention mechanism, and the assembly plan corresponding to the subtask is dynamically adjusted.

[0159] Specifically, when an abnormal situation is detected in a subtask, the adaptive adjustment mechanism is triggered, including re-analyzing the subtask through the attention mechanism and dynamically adjusting the assembly plan corresponding to the subtask. Among them, the adaptive adjustment is mainly reflected in two aspects: on the one hand, the attention mechanism is used to re-evaluate the focus of the task and adjust the task analysis and planning; on the other hand, the execution plan is modified in real time and corrective measures are taken. For example, if a part position deviation is found during the assembly process, the system will temporarily suspend the current action, reposition the part, and update the subsequent assembly path and action parameters. This real-time feedback and adjustment mechanism ensures that the humanoid robot can still complete the assembly task efficiently and accurately in a dynamic environment, improving the robustness of the system.

[0160] Understandably, when an anomaly is detected, the problem is usually limited to the currently executed subtask. At this point, the attention mechanism is used to focus on re-evaluating the currently executed subtask and making corresponding adjustments. The adaptive adjustment mechanism usually only re-evaluates the subtask where the anomaly is detected, which can improve efficiency and avoid unnecessary re-planning of the entire task. Only when the anomaly affects the entire task chain (such as a critical equipment failure) is the global evaluation triggered to consider globally re-evaluating the task and making comprehensive adjustments.

[0161] It should be noted that during the task execution process in some embodiments, the humanoid robot can dynamically adjust the attention weight based on real-time feedback and re-evaluate the key parts of the task. For example, when an assembly error is detected, the attention mechanism will assign more weight to the relevant assembly steps and dynamically adjust the focus of attention, prompting the humanoid robot to focus on inspection and correction.

[0162] Furthermore, in step S603 of some embodiments, in order to make full use of multimodal information such as vision and touch, the scoring function is further improved. The improved attention mechanism integrates multimodal data to obtain a new scoring function such as formula (3):

[0163] e i =v T anh(W h h i +W s S t-1 +W v v i +W t t i ) (3),

[0164] In formula (3), W v , W t represents the weight matrix, which is the parameter that needs to be optimized through learning; v i is the feature vector of visual data, t i is the feature vector of tactile data.

[0165] Optionally, data from vision, touch and other sensors are integrated through a multimodal data fusion module to generate a unified feature representation to help the humanoid robot better perceive the environment and support the humanoid robot to accurately locate parts and adjust assembly force during task analysis and assembly, thereby improving the stability and accuracy of the assembly process.

[0166] In this embodiment, considering that humanoid robots currently encounter problems such as position deviation and changes in component specifications during the assembly process, manual intervention and adjustment are often required, resulting in reduced production efficiency. This embodiment dynamically adjusts the attention weight according to real-time feedback monitoring data (such as visual and force information) during the assembly process, re-evaluates the focus of the task and automatically optimizes the execution steps, so that the humanoid robot can quickly adjust itself when facing environmental changes in the assembly process, thereby realizing the humanoid robot's ability to adapt to changing environments, reducing human intervention, and improving the stability and continuity of the production line.

[0167] In some embodiments, the humanoid robot flexible assembly method based on thought chain may also include but is not limited to steps S701 to S702.

[0168] Step S701, record the task tree structure and assembly status data to obtain the assembly process result.

[0169] Specifically, a tree structure or linked list is used to record all nodes of the task tree structure obtained during the task parsing process, including the task objectives, execution steps and dependencies of each node, and the collected assembly status data is associated with the corresponding nodes in the task tree structure to summarize the assembly process results.

[0170] Optionally, the assembly process results may be generated in the form of an assembly report, which lists the execution status, time consumption, execution progress of each step, and any related exception or error information.

[0171] Step S702: display the assembly process results through an interactive interface.

[0172] It is understandable that in order to facilitate the operator to view the analysis results and execution status of the task in real time, the assembly process results can also be displayed through the interactive interface, using multimedia elements such as charts and animations to present the complex assembly process in an intuitive and easy-to-understand way. For each step, the interactive interface can also provide functions such as zooming in, zooming out, dragging, filtering, and searching, so that the operator can view and analyze the assembly process results as needed.

[0173] The following is a detailed introduction and description of the solution of the embodiment of the present invention in conjunction with a specific application example.

[0174] In order to realize the flexible assembly method of humanoid robots based on thought chain provided in the embodiment of the present application and solve the problem of insufficient flexibility and self-adaptation ability of humanoid robots in complex task analysis and assembly process in the related art, refer to Figure 3 The embodiment of the present application provides a software system for a humanoid robot flexible assembly method based on a thinking chain, which adopts a modular design, including a multimodal task input and encoding module, a hierarchical thinking chain task parsing module with an attention mechanism, a task planning and execution module, a real-time feedback and adaptive adjustment module, a multi-task scheduling and optimization module, and a system control and human-computer interaction interface module.

[0175] Each functional module communicates through a standard interface. For example, the task analysis module, task planning and execution module, and real-time feedback and adaptive adjustment module work together through message passing and data sharing.

[0176] The multimodal task input and encoding module is used to receive assembly task instructions issued by the operator, which may contain multiple forms of information, such as text descriptions, voice commands, or even gestures or schematic diagrams. In order to facilitate the processing of the humanoid robot, the task input can be uniformly encoded through the multimodal task input and encoding module.

[0177] During the implementation process, the system uses natural language processing technology to perform word segmentation, part-of-speech tagging, and semantic analysis on instructions in text or voice form, and extract the main content and intention of the task. At the same time, for visual input, such as gestures or schematic diagrams, computer vision technology is used to extract and recognize features and convert them into processable symbolic information. Finally, all modal data are mapped to a unified high-dimensional vector space through an embedding layer to form a multimodal feature representation of the task input, providing a basis for subsequent task analysis.

[0178] The hierarchical thinking chain task parsing module of the attention mechanism is used to use the encoded assembly task instructions to carry out the high-level task parsing stage. The task instructions are parsed word by word or fragment by fragment through the attention mechanism, so as to automatically identify the key information in the assembly task instructions and obtain multiple high-level tasks.

[0179] Next, each high-level task is recursively decomposed into subtasks. During the subtask decomposition process, the humanoid robot is guided to focus on the details related to the current subtask to avoid being disturbed by irrelevant information. The attention mechanism continues to play a role, guiding the model to focus on the detailed information related to the current subtask. For example, when decomposing "installing part A to component B", the attention mechanism will make the model focus on the necessary information such as the attributes of "part A" and the location of "component B".

[0180] When implementing task decomposition, the model uses a recurrent neural network (such as LSTM or GRU) or Transformer architecture, combined with an attention mechanism, to parse the task instructions word by word or fragment by fragment. In each parsing step, the model dynamically calculates the attention weight based on the task status of the previous step and the current input, and determines the next content to focus on, thereby flexibly adjusting the parsing path and ensuring a deep understanding of the task.

[0181] Finally, after multi-level task decomposition, the system generates specific task execution steps based on key information and obtains a complete task tree structure. The task tree contains detailed information from high-level task objectives to specific execution steps, laying the foundation for subsequent task planning and execution.

[0182] After the task planning and execution module obtains the task tree generated by the task parsing module, it traverses the task tree and generates a specific assembly plan according to the order and dependency of the tasks, including the execution arrangement sequence, task path planning, and the generation of assembly action parameters. The task tree structure is used to generate detailed assembly steps, and the motion planning algorithm is used to guide the humanoid robot to perform operations such as moving, grasping, and installing, ensuring that the assembly task is completed efficiently according to the predetermined process.

[0183] During the execution process, in order to improve the accuracy of execution, sensor feedback control can also be added at key steps. For example, visual sensors can be used to confirm the position and posture of parts, and force sensors can be used to monitor the force conditions during the assembly process, so as to adjust the movements of the humanoid robot in real time.

[0184] The real-time feedback and adaptive adjustment module is used to monitor the status and environmental information of the humanoid robot in real time through various sensors during the execution phase, and adjust the task plan based on the feedback. By real-time monitoring of position deviation, force feedback and other information during the assembly process, the system can readjust the task execution plan when an abnormality is detected to ensure the smooth completion of the task, thereby realizing the humanoid robot's adaptive ability to a changing environment.

[0185] The multi-task scheduling and optimization module is used to reasonably schedule and optimize multiple tasks when the humanoid robot needs to handle multiple assembly tasks at the same time. The implementation details include: firstly, analyzing the set of tasks to be executed, using the attention mechanism to evaluate the priority and resource requirements of each task, and then combining the reinforcement learning algorithm to learn the optimal task scheduling strategy, so that the humanoid robot resources are reasonably allocated and the overall assembly efficiency is maximized.

[0186] During the scheduling process, the system needs to consider the dependencies between tasks, avoid resource conflicts, ensure that high-priority tasks are processed in a timely manner, achieve reasonable parallel execution of multiple tasks, optimize resource allocation, and improve overall assembly efficiency.

[0187] The system control and human-machine interaction interface module is used to provide human-machine interaction, support the input of task instructions and real-time monitoring of the assembly process. The operator can input task instructions in the user interface through natural language, and view the task analysis results, execution status, progress and assembly results in real time for the operator to monitor in real time. The interface also provides a visual display of the assembly process to help the operator understand the working conditions of the humanoid robot. If necessary, the operator can intervene in the humanoid robot through the interface, provide additional instructions or adjust task parameters.

[0188] In this embodiment, by using hierarchical thinking chain technology and combining it with the attention mechanism, the humanoid robot can automatically parse complex natural language assembly instructions and convert them into specific operation steps to achieve efficient and accurate task parsing. With the rapid development of large models (such as GPT-4, etc.) and natural language processing technology, the task parsing method based on pre-trained language models has shown strong understanding and reasoning capabilities. Using large models, humanoid robots can extract key task information from complex natural language instructions, realize automatic parsing and execution of tasks, greatly reduce the need for manual intervention, and improve the humanoid robot's ability to understand diverse assembly tasks.

[0189] By introducing the attention mechanism, humanoid robots can prioritize key information and ignore irrelevant content during task parsing, thereby improving the accuracy and speed of task execution. This enables humanoid robots to more efficiently utilize useful information in task descriptions, optimize information processing, and achieve task focus identification when faced with complex tasks.

[0190] The humanoid robot rationally schedules and executes a variety of assembly tasks according to task priorities and resource availability, optimizes the order of task execution, and reduces resource waste in the production process, thereby improving the overall work efficiency of the flexible assembly production line.

[0191] In addition, during the execution of tasks, humanoid robots can dynamically adjust operating steps based on real-time feedback, adapt to changes in assembly environment and task requirements, and ensure efficient operation in complex production environments. This can significantly improve the flexibility and adaptability of production lines and enhance the adaptability of task planning and execution.

[0192] Reference Figure 4 , Figure 4 A flowchart of the overall operation of the thinking chain is provided. The hierarchical thinking chain structure is used to refine complex tasks layer by layer, and multimodal perception data is combined for logical verification to ensure the rationality and accuracy of the task decomposition process.

[0193] Exemplarily, the natural language instructions input by the operator are received, and the natural language instructions are preliminarily parsed through the task input and preliminary parsing module to generate a preliminary semantic representation. Then, the task is parsed layer by layer using a hierarchical thinking chain structure. When parsing tasks, the system combines the input multimodal data (such as natural language descriptions, visual information) and contextual information, dynamically calculates the weight of each input through the attention mechanism, and gives higher weight to information related to the current task node to optimize the task parsing results.

[0194] Specifically, in the task parsing stage, the system uses an improved hierarchical thinking chain, combined with an attention mechanism, to decompose and understand the task layer by layer. The implementation details are as follows: The main task objectives in the instructions are extracted through the high-level task parsing module, and the corresponding high-level task nodes are generated, where the high-level task nodes include assembly objectives, operation requirements, and execution conditions. The high-level task nodes are further parsed through the subtask recursive parsing module to generate multiple middle-level tasks and basic operation steps, forming a complete task tree structure. Each node in the task tree contains the task objectives, execution steps, and their dependencies.

[0195] Optionally, after the task tree is generated, the system displays the results of the task analysis in real time through a human-computer interaction interface. The operator can view the decomposition steps and execution status of the task on the interface and confirm or modify the adjustment suggestions made by the system.

[0196] It should be noted that in the aforementioned parsing process, the attention mechanism is combined to dynamically adjust the weights of the key parts of the instructions, and a higher weight is given to the information related to the current task node, so that the task parsing process gives priority to the features related to the current task node to optimize the task parsing results.

[0197] Furthermore, in the task parsing process of some embodiments, the weighted feature vector generated by the multimodal attention mechanism can also be combined to enable the humanoid robot to simultaneously process natural language instructions, visual information, and tactile information, and realize the fusion parsing of multimodal data, thereby improving the accuracy of task parsing and the reliability of execution.

[0198] Next, the generated task tree is logically verified through the task verification and optimization module. The generated task steps are compared by analyzing the domain knowledge base to eliminate unreasonable operation steps, ensure the rationality of task dependencies and execution conditions, and use the attention mechanism to optimize the task execution plan. The node arrangement in the task tree is optimized according to task priority and resource requirements to optimize execution efficiency and generate the final assembly plan.

[0199] The assembly plan is sent to the task execution module, which drives the humanoid robot to assemble according to the assembly plan.

[0200] It is understandable that during the task execution, it is also possible to monitor the environmental changes in the humanoid robot assembly process in real time, use the attention mechanism to readjust the focus of task parsing, and dynamically update the task tree structure based on the detected anomalies.

[0201] In this embodiment, unlike the existing fixed rule or template-driven flexible assembly methods that can only adapt to specific production environments and are difficult to cope with diverse production needs, this embodiment uses the thinking chain task parsing method to enable the humanoid robot to have a wider range of task understanding and execution capabilities, and can flexibly respond to different types of assembly tasks in a variety of production scenarios. Whether it is a small batch, multi-variety production environment or a production line that needs to adjust the task strategy in real time, intelligent task parsing and assembly execution can be achieved based on the method of this embodiment. During the assembly process, the attention mechanism can focus on key steps, such as the grasping and installation of parts, thereby reducing operational errors caused by external interference. Compared with the traditional fixed process system, the method of this embodiment is highly adaptable, significantly improves the application breadth of flexible manufacturing, and can perform well in assembly tasks with high precision requirements, ensuring assembly quality and reliability.

[0202] In some embodiments, when multiple tasks to be executed need to be scheduled in a multi-task environment, the optimal task scheduling strategy is generated by the reinforcement learning algorithm according to the priority, urgency and required resources of the task to ensure efficient use of humanoid robot resources. In the task scheduling process, the reinforcement learning algorithm continuously optimizes the task priority sorting and resource allocation strategy by learning the feedback of multi-task execution, so that the humanoid robot can operate in parallel in a multi-task environment and improve the overall assembly efficiency.

[0203] In some embodiments, in order to verify the humanoid robot flexible assembly system and method based on the thought chain of the present solution, appropriate software and hardware support can also be provided in the experimental environment to conduct verification experiments.

[0204] Specifically, the hardware environment includes:

[0205] (1) Humanoid robot: A robotic arm with more than 6 degrees of freedom, used to simulate human grasping, assembly and other operations, and equipped with vision and force sensors to perceive the environment.

[0206] (2) Control host: Equipped with a high-performance CPU and a GPU that supports deep learning computing (such as NVIDIA RTX4090) to run task analysis models and control algorithms.

[0207] (3) Workbench and parts: Set up an experimental workbench to store the parts and tools to be assembled to simulate the actual production environment.

[0208] The software environment includes:

[0209] (1) Operating system: Use Linux system (such as Ubuntu 20.04) to ensure hardware compatibility and deep learning support.

[0210] (2) Development platform: Use PyTorch or TensorFlow for model training and combine it with the Robot Operating System (ROS) for robot control and sensor data processing.

[0211] (3) Large model: Select a pre-trained language model (such as GPT-4) and fine-tune it to parse task instructions.

[0212] Before the verification experiment begins, task instruction data and multimodal data are prepared in advance. The task instruction data contains natural language descriptions of different assembly tasks and their parsing results, which are used to train the model; the multimodal data contains the visual and force data of the mobile phone during the operation of the humanoid robot, which is used to enhance the model's perception ability during the assembly process.

[0213] After the model training is completed, prepare a 3x3 meter experimental area according to the space requirements of the experimental site to ensure that the humanoid robot has enough space for movement during operation. Prepare corresponding safety measures, set up emergency stop buttons and isolation areas to prevent safety risks caused by misoperation of the humanoid robot during the experiment.

[0214] In the experimental site, including the robot work area, workbench and parts storage area, after confirming that the experimental area is clean and has sufficient space for activities, start the humanoid robot and control host, connect visual sensors, force sensors and other equipment, and check the working status of each device to ensure the normal operation of the hardware.

[0215] When it is confirmed that all devices are working normally and the hardware is running normally, the pre-trained and fine-tuned large language model (such as GPT-4) and multimodal fusion model are loaded on the control host, and ROS is started to ensure smooth communication between modules.

[0216] First, data input and task instruction issuance are performed, and the operator inputs assembly task instructions through natural language, such as "install part A on component B and perform precision calibration". The control host parses the natural language instructions through a multimodal encoder and generates a high-dimensional feature representation.

[0217] Next, the improved thought chain structure and attention mechanism are used to parse the input task instructions and generate a complete task tree structure, including high-level task objectives and specific operation steps.

[0218] Then, the system performs logic verification and path planning on the task according to the parsed task tree, and generates a specific assembly plan and robot action sequence. Using the path planning function in ROS, the motion trajectory of the humanoid robot from the parts storage area to the workbench is planned, and specific assembly action instructions are generated, such as "move to the position of part A", "grab part A", "install part A to component B", etc.

[0219] Finally, the humanoid robot executes each task step by step according to the generated assembly plan, uses visual sensors to confirm the position of parts, and uses force sensors to monitor the force during the assembly process. At the same time, during the execution of the assembly task, the assembly status data (such as force data and position data) is fed back to the control host in real time. If an abnormality is detected during the assembly process (such as position deviation or abnormal assembly force), the system re-evaluates the task focus through the attention mechanism, adjusts the assembly action, and continues to execute the remaining steps.

[0220] During the entire experiment, the execution time of each task step, the time consumed in path planning, assembly errors, sensor data, etc. were recorded. The task tree structure automatically generated by the system during task parsing and the weight changes adjusted by the attention mechanism were recorded to analyze the accuracy of the parsing. If assembly failure or task interruption occurs, the cause of the failure is recorded and the effect of the adaptive adjustment is analyzed.

[0221] Through the above verification experiments, the consistency between the actual assembly task completed by the humanoid robot and the expected task plan is compared to evaluate the accuracy of the parsing task and the stability of the execution. The scheduling efficiency and assembly quality during multi-task execution are analyzed, including the priority allocation and resource usage of each task. Based on the experimental data, the effectiveness of the system's adaptive adjustment ability in a dynamic environment is evaluated, such as the adjustment time after detecting an anomaly and the adjustment success rate.

[0222] After the experiment is over, stop the humanoid robot, close all experimental processes on the control host, organize and save the data files and logs generated during the experiment, and inspect and maintain the experimental equipment to ensure that the next experiment can proceed normally.

[0223] Based on the experimental results, the model is fine-tuned and improved to enhance the application effect of the system in actual flexible manufacturing scenarios.

[0224] Reference Figure 5 The embodiment of the present application also provides a humanoid robot flexible assembly virtual system based on thought chain, which can implement the above-mentioned humanoid robot flexible assembly method based on thought chain, and the system includes:

[0225] The first module is used to obtain the assembly task instructions provided by the user.

[0226] The second module is used to perform task analysis on the assembly task instructions through the thought chain model, and calculate the weight of each word in the assembly task instructions through the attention mechanism during the analysis process, dynamically adjust the analysis target related to the current task according to the weight, and decompose the task layer by layer into multiple subtasks according to the analysis target to obtain a task tree structure.

[0227] The third module is used to traverse the task tree structure to determine an assembly plan, wherein the assembly plan includes an execution arrangement sequence, task path planning and assembly action parameters.

[0228] The fourth module is used to control the humanoid robot to perform assembly operations according to the assembly plan.

[0229] It can be understood that the contents of the above method embodiments are all applicable to the present virtual system embodiments, the functions specifically implemented by the present virtual system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0230] The embodiment of the present application also provides a humanoid robot flexible assembly hardware system based on thought chain, the hardware system includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned humanoid robot flexible assembly method based on thought chain when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0231] It can be understood that the contents of the above method embodiments are all applicable to the present hardware system embodiments, the functions specifically implemented by the present hardware system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0232] Reference Figure 6 , Figure 6The hardware structure of a humanoid robot flexible assembly hardware system based on a thinking chain in another embodiment is illustrated. The hardware system includes:

[0233] The processor 901 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0234] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the humanoid robot flexible assembly method based on the thinking chain of the embodiment of this application.

[0235] The input / output interface 903 is used to implement information input and output.

[0236] The communication interface 904 is used to realize the communication interaction between this device and other devices. Communication can be realized through wired methods (such as USB, network cable, etc.) or wireless methods (such as mobile network, WIFI, Bluetooth, etc.).

[0237] The bus 905 transmits information between various components of the device (eg, the processor 901 , the memory 902 , the input / output interface 903 , and the communication interface 904 ).

[0238] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0239] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned humanoid robot flexible assembly method based on thought chain is implemented.

[0240] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0241] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0242] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0243] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0244] The system embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0245] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0246] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0247] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A humanoid robot flexible assembly method based on thought chain, characterized in that: The method comprises the following steps: Obtaining assembly task instructions provided by the user; The assembly task instruction is parsed through the thought chain model, and the weight of each word in the assembly task instruction is calculated through the attention mechanism during the parsing process, and the parsing target related to the current task is dynamically adjusted according to the weight, and the task is decomposed into multiple subtasks layer by layer according to the parsing target to obtain a task tree structure; Traversing the task tree structure to determine an assembly plan, wherein the assembly plan includes an execution arrangement sequence, task path planning, and assembly action parameters; The humanoid robot is controlled to perform assembly operations according to the assembly plan.

2. The method according to claim 1, characterized in that The step of obtaining the assembly task instruction provided by the user comprises the following steps: Acquiring multimodal data provided by a user, wherein the multimodal data includes text, voice, and visual information; Use natural language processing technology to segment text and speech, perform part-of-speech tagging and semantic analysis, extract the main content and intention of the task to obtain text information; Using computer vision technology to extract and identify features of the visual information to obtain symbol information; The text information and the symbol information are mapped to a vector space to obtain the assembly task instruction.

3. The method according to claim 1, characterized in that The step of calculating the weight of each word in the assembly task instruction by using the attention mechanism and dynamically adjusting the parsing target related to the current task according to the weight includes the following steps: Calculating an attention score for each word in the assembly task instruction based on context information through an attention mechanism, wherein the context information includes task history status and domain knowledge; Normalizing the attention score to obtain the weight of the word; The words whose weights are greater than a preset threshold are determined as the parsing targets related to the current task.

4. The method according to claim 1, characterized in that The step of traversing the task tree structure to determine the assembly plan includes the following steps: Traversing the task tree structure to determine the order and dependency of subtasks; According to the sequence and dependency relationship of the subtasks, the task tree structure is logically verified, and the arrangement of the subtasks in the task tree structure is optimized based on the attention mechanism to obtain the execution arrangement order; Determine the task path planning using a motion planning algorithm according to the execution arrangement sequence; Decomposing the subtasks into a plurality of basic actions that can be directly executed by the humanoid robot, and determining the required assembly action parameters according to the basic actions; An assembly plan is obtained according to the execution arrangement sequence, the task path planning and the assembly action parameters.

5. The method according to claim 4, characterized in that The step of performing logical verification on the task tree structure according to the sequence and dependency of the subtasks, and optimizing the arrangement of the subtasks in the task tree structure based on the attention mechanism to obtain the execution arrangement sequence includes the following steps: When there are multiple subtasks to be executed at the same time, the priority and resource requirements of each subtask are evaluated through the attention mechanism to obtain an evaluation result; A task scheduling strategy is determined by a reinforcement learning algorithm according to the evaluation result, and the arrangement of subtasks in the task tree structure is adjusted according to the task scheduling strategy to obtain the execution arrangement order.

6. The method according to claim 1, characterized in that The humanoid robot flexible assembly method based on thought chain also includes the following steps: Acquiring monitoring data, wherein the monitoring data includes environmental information and assembly status data; Determine whether there is any abnormality in the subtask currently being executed according to the monitoring data; When an abnormal situation occurs in the subtask, the subtask is re-analyzed through the attention mechanism, and the assembly plan corresponding to the subtask is dynamically adjusted.

7. The method according to claim 6, characterized in that The humanoid robot flexible assembly method based on thought chain also includes the following steps: Recording the task tree structure and the assembly status data to obtain an assembly process result; The assembly process results are displayed through an interactive interface.

8. A humanoid robot flexible assembly virtual system based on thought chain, characterized in that: The virtual system comprises: The first module is used to obtain the assembly task instructions provided by the user; The second module is used to perform task parsing on the assembly task instruction through a thought chain model, and calculate the weight of each word in the assembly task instruction through an attention mechanism during the parsing process, dynamically adjust the parsing target related to the current task according to the weight, and decompose the task layer by layer into multiple subtasks according to the parsing target to obtain a task tree structure; A third module is used to traverse the task tree structure to determine an assembly plan, wherein the assembly plan includes an execution arrangement sequence, task path planning and assembly action parameters; The fourth module is used to control the humanoid robot to perform assembly operations according to the assembly plan.

9. A humanoid robot flexible assembly hardware system based on thought chain, characterized in that: The hardware system includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images