A method and device for task planning of an assembly robot arm

Abstract

The present application discloses a method and device for task planning of an assembly robot arm, relating to the technical field of assembly task planning. The method comprises: obtaining multimodal data of the assembly robot arm and determining a joint feature code to determine an assembly retrieval result; determining multiple assembly subtask sequences based on the multimodal data, the assembly retrieval result, and a large language model; determining multiple feasible action sequences for each assembly subtask based on a pre-trained action library; determining the total score of the corresponding feasible action sequence based on the similarity scores and optimal feasibility scores corresponding to all actions in each feasible action sequence to obtain an optimal action sequence; determining an optimal action parameter sequence based on the pre-trained action library and the optimal action sequence; and converting the optimal action parameter sequence into the motion of the assembly robot arm based on the pre-trained action library to perform the assembly task. The present application improves the task understanding and decomposition capability, feasibility, and adaptability of task planning for an assembly robot arm.

Description

Technical Field

[0001] The present application relates to the technical field of assembly task planning, and in particular to a method and device for task planning of an assembly robot arm. Background Art

[0002] Traditional assembly robot task planning relies mainly on manually predefined rules or traditional optimization algorithms, which have three major limitations: (1) Insufficient task understanding and decomposition capabilities: It is difficult to parse multimodal task instructions (such as text and images), resulting in a mismatch between the subtask sequence and the physical scene, requiring repeated manual adjustments. (2) Planning and execution are separated, resulting in low feasibility: The abstract instructions generated by traditional planning require additional parameter calibration, and the lack of automated optimization of action parameters (such as grasping posture and force control parameters) has a high failure rate, resulting in low feasibility of assembly robot task planning. (3) Weak dynamic adaptability: In the face of changes in the assembly environment (such as part offset and loose fixtures), traditional methods cannot adjust planning strategies in real time, which can easily lead to task interruption. Summary of the Invention

[0003] The purpose of this application is to provide a task planning method and device for an assembly robot arm to solve the problems of insufficient task understanding and decomposition capabilities, low feasibility and weak adaptability.

[0004] To achieve the above objectives, this application provides the following solutions:

[0005] In a first aspect, the present application provides a method for task planning of an assembly robot arm, comprising:

[0006] Acquire multimodal data of the assembly robot arm; the multimodal data includes: an image of the assembly site and a text description of the assembly task;

[0007] Determining a joint feature code based on the image and the text description;

[0008] Determining assembly retrieval results based on the joint feature code and the assembly knowledge base;

[0009] determining a plurality of assembly subtask sequences of the assembly task based on the multimodal data, the assembly retrieval result, and a large language model;

[0010] Based on the pre-trained action library, multiple feasible action sequences of each assembly subtask in each assembly subtask sequence are determined respectively;

[0011] Determine the similarity score and optimal feasibility score corresponding to each action in each feasible action sequence;

[0012] Determine the total score of the corresponding feasible action sequence based on the similarity scores and optimal feasibility scores corresponding to all actions in each feasible action sequence;

[0013] The feasible action sequence with the largest total score is determined as the optimal action sequence for the assembly robot;

[0014] Determine the optimal action parameter sequence based on the pre-trained action library and the optimal action sequence;

[0015] Based on the pre-trained motion library, the optimal motion parameter sequence is converted into the motion of each joint of the assembly robot arm to perform the assembly task.

[0016] In one embodiment, before acquiring the multimodal data of the assembly robot arm, the method further includes: defining a pre-trained action library and an assembly knowledge library.

[0017] In one embodiment, the pre-trained action library includes: action parameters of multiple actions, strategies for selecting optimal action parameters for each action in any environment, and value functions;

[0018] The assembly knowledge base includes: a parts body library, an assembly relationship map and an assembly process knowledge tree;

[0019] The component body library is used to define the geometric properties, physical properties and functional properties of the parts;

[0020] The assembly relationship diagram is used to describe the assembly relationship between various parts;

[0021] The assembly process knowledge tree is used to represent the sequence of assembly processes.

[0022] In one embodiment, determining a joint feature code based on the image and the text description includes:

[0023] Preprocessing the image to obtain an initial image vector; the preprocessing includes: image segmentation, block embedding and position coding;

[0024] Performing feature extraction on the initial image vector to obtain a visual feature vector;

[0025] Encode the text description through the BERT model to obtain a text feature vector;

[0026] The visual feature vector and the text feature vector are weightedly integrated to obtain the joint feature code.

[0027] In one embodiment, determining an assembly search result based on the joint feature code and the assembly knowledge base includes:

[0028] Filtering out all parts whose cosine similarity with the joint feature code is greater than a first preset threshold from the part body database to obtain a plurality of related parts;

[0029] Based on the assembly relationship diagram, determine the assembly relationship between related parts;

[0030] Determine the assembly process sequence based on the assembly process knowledge tree and the assembly relationship between related parts;

[0031] The related parts, the assembly relationships between the related parts, and the assembly process sequence are determined as the assembly retrieval results.

[0032] In one embodiment, based on the pre-trained action library, multiple feasible action sequences of each assembly subtask in each assembly subtask sequence are determined, including:

[0033] Determine any assembly subtask as a subtask to be determined;

[0034] Determine the cosine similarity between the subtask to be determined and each action in the pre-trained action library;

[0035] All actions corresponding to the subtask to be determined whose cosine similarity is greater than a second preset threshold and the assembly search results are input into the large language model to obtain multiple feasible action sequences of the subtask to be determined.

[0036] In one embodiment, determining the similarity score and the optimal feasibility score corresponding to each action in each feasible action sequence includes:

[0037] Determine any assembly subtask as the current subtask, determine any feasible action sequence of the current assembly subtask as the current action sequence, and determine any action in the current action sequence as the current action;

[0038] Calculate the similarity score between the current action and the current subtask, and determine the similarity score between the current action and the current subtask as the similarity score corresponding to the current action;

[0039] Based on the strategies in the pre-trained action library, determine multiple action parameter sequences of the current action sequence;

[0040] Based on the value function in the pre-trained action library, the feasibility score of the current action under each action parameter sequence is determined;

[0041] Based on the feasibility scores of the current action under each action parameter sequence, the optimal feasibility score corresponding to the current action is determined.

[0042] In one embodiment, determining the total score of each feasible action sequence based on the similarity scores and optimal feasibility scores corresponding to all actions in each feasible action sequence includes:

[0043] Multiply the similarity score and optimal feasibility score of each action in each feasible action sequence to obtain the comprehensive score of the corresponding action;

[0044] The comprehensive scores of all actions in each action sequence are multiplied to obtain the total score of the corresponding action sequence.

[0045] In one embodiment, determining an optimal action parameter sequence based on a pre-trained action library and an optimal action sequence includes:

[0046] Determine optimal action parameters of the first action in the optimal action sequence based on the current environment of the assembly site, the first action in the optimal action sequence, and the strategy;

[0047] Based on the optimal action parameters of the first action in the optimal action sequence, the optimal action parameters of each action in the optimal action sequence are determined in sequence, thereby determining the optimal action parameter sequence.

[0048] In a second aspect, the present application provides a computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the above-described assembly robot task planning methods.

[0049] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0050] The present application discloses a method and device for task planning of an assembly robot arm. First, multimodal data of the assembly robot arm is acquired. The multimodal data includes: an image of the assembly site and a text description of the assembly task. Based on the image and text description, a joint feature code is determined. Based on the joint feature code and an assembly knowledge base, an assembly retrieval result is determined. Then, based on the multimodal data, the assembly retrieval result and a large language model, multiple assembly subtask sequences of the assembly task are determined. Secondly, based on a pre-trained action library, multiple feasible action sequences of each assembly subtask in each assembly subtask sequence are determined respectively. Subsequently, a similarity score and an optimal feasibility score corresponding to each action in each feasible action sequence are determined respectively. Based on the similarity scores and optimal feasibility scores corresponding to all actions in each feasible action sequence, a total score of the corresponding feasible action sequence is determined. The feasible action sequence with the largest total score is determined as the optimal action sequence of the assembly robot arm. Thirdly, based on the pre-trained action library and the optimal action sequence, an optimal action parameter sequence is determined. Finally, based on the pre-trained action library, the optimal action parameter sequence is converted into the motion of each joint of the assembly robot arm to perform the assembly task. This application decomposes the assembly task based on a large language model combined with RAG technology to obtain multiple assembly sub-task sequences, thereby improving the task understanding and decomposition capabilities of the assembly robot arm task planning; determines the optimal action sequence of the assembly robot arm based on the similarity score and the optimal feasibility score, thereby improving the feasibility of the assembly robot arm; the method of this application can be used to implement assembly robot arm task planning, thereby improving the adaptability of assembly robot arm task planning. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0052] Figure 1 A flowchart of a method for task planning of an assembly robot arm provided in one embodiment of the present application.

[0053] Figure 2 A schematic diagram of an assembly robot arm task planning architecture provided in one embodiment of the present application.

[0054] Figure 3 Schematic diagram of the neural network structure corresponding to the value function.

[0055] Figure 4 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0057] The purpose of this application is to provide a method and device for task planning of an assembly robot arm, aiming to improve the task understanding and decomposition capability, feasibility and adaptability of task planning of an assembly robot arm.

[0058] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0059] In an exemplary embodiment, Figure 1 and Figure 2 As shown, a method for task planning of an assembly robot arm is provided, including steps 01 to 10.

[0060] Step 01: Obtain multimodal data of the assembly robot arm; the multimodal data includes: images of the assembly site and text descriptions of the assembly task.

[0061] As an optional implementation, before step 01, the process further includes: defining a pre-trained action library and an assembly knowledge base.

[0062] As an optional implementation, the pre-trained action library includes: action parameters of multiple actions, strategies for selecting optimal action parameters for each action in any environment, and value functions.

[0063] The assembly knowledge base includes: component ontology library, assembly relationship diagram and assembly process knowledge tree.

[0064] The component library is used to define the geometric, physical and functional properties of parts.

[0065] The assembly relationship diagram is used to describe the assembly relationship between various parts.

[0066] The assembly process knowledge tree is used to represent the sequence of assembly processes.

[0067] Specifically, when defining the pre-trained action library, we must first determine the actions needed in the assembly task. , such as: grab, align, insert, drop, move, etc. And each action needs to define the following three properties:

[0068] ①Action parameters : The action parameters required for each action to execute, used by the underlying planner for each action. For example, the parameters required for grasping are the coordinates and pose of the grasped object; the parameters required for inserting are the direction and force of the insertion, etc.

[0069] ②Strategy : A strategy for selecting the optimal action parameters for each action in the current environment, which can be in the form of any reinforcement learning method, neural network or other learning-based methods.

[0070] ③A value function : Each action selects any action parameter in any environment to determine whether the final execution is successful. If the execution is successful, then If the execution fails, .

[0071] After defining the three attributes, each part needs to be trained or solved in detail until an attribute that meets the actual situation is obtained. , you also need to define a parameter for executing the action The underlying motion planner of the robot arm can use reinforcement learning, rapidly exploring random trees (RRT), probabilistic roadmap (PRM) and other robot path planning algorithms.

[0072] Secondly, the value function of each action needs to be trained , the training effect of the value function is crucial for the generation of strategies. The value of can only be 0 or 1, so you can As a binary classifier, the input is the environment of the assembly site corresponding to each moment, including the assembly image (image of the assembly site) and the part coordinates. A classifier such as Figure 3 The neural network shown is represented by The structure is as follows: image processing includes a convolutional layer, a pooling layer, a convolutional layer, a pooling layer, and three convolutional layers connected in sequence to convert the assembly image into a high-dimensional vector; the hidden layer includes two hidden layers with 256 neurons, and finally the input is binary classified through the softmax layer to obtain the value of the value function.

[0073] The training data for the value function is generated through the following steps:

[0074] ①Define the value range of action parameters.

[0075] ② Sample the data of each scene, including images and part coordinates (collected in the simulation environment).

[0076] ③ For action parameters, uniform sampling is performed within a given range.

[0077] ④Simulate the sampled environment and action parameters, and determine whether the action is successful to obtain the value function.

[0078] This sampling strategy was slightly modified to maintain a success-failure ratio of at least 40%, since for more challenging actions like pulling and pushing, uniform sampling rarely yields successful results. Of the triplets, 800,000 are used for training and the remaining 200,000 are used for validation.

[0079] Finally, train the policy for each action The strategy here is about selecting action parameters, not the strategy of selecting a path given a given goal in general path planning. Instead, it is about selecting the planning goal, that is, the strategy of selecting action parameters. At each assembly stage, a strategy is trained to select the optimal action parameters based on a specific assembly environment. Since the value function has already been trained, the learned value function model is directly used to convert it into the optimal strategy through the maximum entropy model, namely:

[0080] .

[0081] in, is the optimal strategy; is the expectation of the training set; is the training set used in the strategy training process, including sample pairs of environment and action parameters ; The entropy regularization coefficient is used to balance the two optimization objectives of the strategy, namely maximizing the value of the value function and maximizing the entropy of the strategy. Entropy regularization can improve the robustness and diversity of the strategy.

[0082] When defining the assembly knowledge base, in order to assist the large model in understanding the specified assembly environment, it is necessary to first define the parts, processes, environment, and other knowledge required in the assembly site. Assembly knowledge is mainly divided into the following categories:

[0083] ① Parts library: used to define the geometric properties (CAD model, size, etc.), physical properties (material, mass, friction coefficient, etc.), functional properties (component type, fit tolerance grade), etc. of parts.

[0084] ② Assembly relationship diagram: used to describe the assembly relationship between each part. It can be represented by a knowledge graph. Each edge can be used to represent the fit type, fit tolerance level, assembly constraint relationship (sequence constraint, geometric constraint, mechanical constraint), etc. between parts.

[0085] ③ Assembly process knowledge tree: This tree represents the order of assembly processes. Using a tree structure can better represent the order of assembly between multiple parts. The knowledge tree can also be used to define the tools, assembly sequence, and assembly methods required for each process.

[0086] After the definition is completed, it is stored in the form of a knowledge graph in graph databases such as neo4j, which makes it easy to fine-tune the retrieval capabilities of the large model in specific tasks.

[0087] Finally, a pre-trained action library containing action parameters, strategies, a value function, and a defined assembly knowledge base is defined.

[0088] Step 02: Determine the joint feature encoding based on the image and text description.

[0089] As an optional implementation, step 02 includes steps 021 to 024.

[0090] Step 021: Preprocess the image to obtain an initial image vector; the preprocessing includes: image segmentation, block embedding and position encoding.

[0091] Specifically, assuming the input image is an RGB image of size 3×224×224, preprocessing begins by first partitioning the image into non-overlapping 14×14 blocks, for a total of 256 blocks. Each 14×14 block is then flattened into a 196-dimensional vector, ultimately resulting in a 768×196 vector. Next, a learnable label with an initial value of 0 is added to each block, resulting in a 768×197 vector. Finally, to represent the position of each block, the block's position is encoded and added to the previously added learnable label. After preprocessing, the resulting image initialization vector is 768×197.

[0092] Step 022: Extract features from the initial image vector to obtain a visual feature vector.

[0093] Specifically, the ViT model, consisting of 12 stacked transformer encoders, extracts features from the initial image vector. Each transformer encoder includes a multi-head attention mechanism (12 64-dimensional heads) to capture global dependencies. The model then passes through a fully connected neural network (768→1024→768), with a normalization layer added before each to stabilize the training process. Finally, the first row of the final output labeled vector (768×1 dimension) is used as the final visual feature vector.

[0094] Step 023: Encode the text description through the BERT model to obtain a text feature vector.

[0095] Specifically, the text description is tokenized and special markers ([CLS] and [SEP]) are added at the beginning and end. A maximum text length is specified. If the text description is too long, it is truncated; if it is too short, it is padded. Finally, the words in the text description are converted to numbers from a dictionary to obtain a vector representing the text description with a dimension equal to the maximum text length. This completes the preprocessing. This vector is then fed into the pretrained BERT model. Finally, the 768-dimensional vector obtained by the BERT model for [CLS] is used as the text feature vector.

[0096] Step 024: Perform weighted integration on the visual feature vector and the text feature vector to obtain a joint feature code.

[0097] Specifically, first, the correlation between each visual feature in the visual feature vector and each text feature in the text feature vector is calculated using the dot product function, and then a softmax layer is used to obtain the attention weight, which represents the degree of correlation between the two, namely:

[0098] .

[0099] in, for and the degree of correlation between them; for and The dot product between The first visual features; is the first character in the text feature vector text features.

[0100] Then, using the obtained weights , perform weighted integration of each text feature in the text feature vector to obtain the joint feature encoding. The formula is:

[0101] .

[0102] in, is the first A joint feature.

[0103] Step 03: Determine assembly retrieval results based on the joint feature coding and assembly knowledge base.

[0104] As an optional implementation, step 03 includes steps 031 to 034.

[0105] Step 031: Filter out all parts whose cosine similarity with the joint feature code is greater than a first preset threshold from the part body database to obtain multiple related parts.

[0106] Specifically, first, the part body database is vectorized, and BERT is used to vectorize the text information in the part body database at the same time. The part body database is vectorized to obtain the vector of each part.

[0107] Based on the vector of each part and the joint feature code, the cosine similarity between each part and the joint feature code is calculated.

[0108] The first preset threshold is set to 60%.

[0109] Step 032: Based on the assembly relationship diagram, determine the assembly relationship between the relevant parts.

[0110] Step 033: Determine the assembly process sequence based on the assembly process knowledge tree and the assembly relationship between the relevant parts.

[0111] Specifically, the assembly relationships between the relevant parts are matched in the assembly process knowledge tree. By comparing the assembly order in the assembly process knowledge tree, if assembly relationship 1 and assembly relationship 2 are found under the same assembly process knowledge tree (a certain assembly process), if assembly relationship 1 is the parent node of assembly relationship 2, then assembly relationship 1 must precede assembly relationship 2, and assembly relationship 1 and assembly relationship 2 have a strong correlation. If assembly relationship 1 and assembly relationship 2 are nodes of the same level in a tree, then the assembly order of assembly relationship 1 and assembly relationship 2 is not strictly distinguished, but the correlation is strong. If assembly relationship 1 and assembly relationship 2 do not belong to the same tree, then the correlation between assembly relationship 1 and assembly relationship 2 is weak, and there is no need to distinguish the order.

[0112] Step 034: Determine the relevant parts, the assembly relationships between the relevant parts, and the assembly process sequence as the assembly search results.

[0113] For example, the related parts obtained by screening through cosine similarity include primary gears, gear shafts, secondary gears, bearings, etc. Then, in the assembly relationship map, it is found that the assembly relationship between the primary gear and the gear shaft is coaxial and interference fit. Finally, the order of each assembly relationship is sorted out. For example, the assembly of the bearing and the base should be after the gear and the gear shaft. The assembly search results are obtained and structured into the following structure:

[0114] Related parts: [“First stage gear” (radius: 30mm, number of teeth: 15...), “gear shaft” (radius: 5mm...), “bearing” (inner diameter: 5mm...)...].

[0115] Assembly relationship: [["primary gear", "gear shaft", "coaxial, H7 / h6 interference fit..."], ...].

[0116] Assembly process sequence: [[“Assembly of the primary gear and gear shaft” (first), “Assembly of the bearing and base” (last)], ...].

[0117] Step 04: Based on the multimodal data, assembly retrieval results and large language model, determine multiple assembly subtask sequences of the assembly task.

[0118] Specifically, the multimodal data, assembly search results, and prompt words are input into the large model. The example format of the input is as follows:

[0119] Assembly task description: Assembly of the secondary reducer.

[0120] Multimodal data.

[0121] Related parts: [“First stage gear” (radius: 30mm, number of teeth: 15...), “gear shaft” (radius: 5mm...), “bearing” (inner diameter: 5mm...)...].

[0122] Assembly relationship: [["primary gear", "gear shaft", "coaxial, H7 / h6 interference fit..."], ...].

[0123] Assembly process sequence: [[“Assembly of the primary gear and gear shaft” (first), “Assembly of the bearing and base” (last)], ...].

[0124] By enhancing the domain knowledge retrieval capability of large language models through Retrieval-Augmented Generation (RAG) technology (i.e., using cosine similarity to screen relevant parts before inputting into the large language model and obtaining assembly retrieval results step by step, and then inputting the assembly retrieval results and multimodal data into the large language model together), the problem of insufficient knowledge of general large language models in professional assembly scenarios is solved, and the accuracy of task decomposition and process rationality are improved.

[0125] Step 05: Based on the pre-trained action library, multiple feasible action sequences of each assembly subtask in each assembly subtask sequence are determined.

[0126] As an optional implementation, step 05 includes steps 051 to 053.

[0127] Step 051: Determine any assembly subtask as a subtask to be determined.

[0128] Step 052: Determine the cosine similarity between the subtask to be determined and each action in the pre-trained action library.

[0129] Step 053: All actions and assembly retrieval results corresponding to the subtask to be determined and having a cosine similarity greater than a second preset threshold are input into the large language model to obtain multiple feasible action sequences for the subtask to be determined.

[0130] Step 06: Determine the similarity score and optimal feasibility score corresponding to each action in each feasible action sequence.

[0131] As an optional implementation, step 06 includes steps 061 to 065.

[0132] Step 061: Determine any assembly subtask as the current subtask, determine any feasible action sequence of the current assembly subtask as the current action sequence, and determine any action in the current action sequence as the current action.

[0133] Step 062: Calculate the similarity score between the current action and the current subtask, and determine the similarity score between the current action and the current subtask as the similarity score corresponding to the current action.

[0134] Specifically, the first action in the current available action sequence of any current subtask The calculation formula for the similarity score between an action and the current subtask is:

[0135] .

[0136] in, For the Action The similarity score between and the current subtask; For and Under the conditions, the generated probability; For the 1st to an environment; For the 1st to Actions For the A vector of actions; is the vector between the current subtasks; is the magnitude of the vector; is the dot product of two vectors.

[0137] Step 063: Based on the strategies in the pre-trained action library, determine multiple action parameter sequences of the current action sequence.

[0138] Specifically, for the current action sequence, sampling is performed in the probability distribution of the action parameters in the current environment of the assembly site output by the strategy, and the first action parameters of multiple current environments are obtained. (i.e. in the environment Got the first Action parameters Then assume the action Execute, update to the environment , and again get new action parameters based on strategy sampling . ) are updated in sequence to obtain multiple action parameter sequences.

[0139] Step 064: Based on the value function in the pre-trained action library, determine the feasibility score of the current action under each action parameter sequence.

[0140] Specifically, Action The calculation formula for the feasibility score under any current action parameters is:

[0141] .

[0142] in, For the Action Feasibility score under any current action parameters; For and Under the conditions, The probability of for Action parameters; for and The value of the value function under ; For the an environment; For the an action.

[0143] Step 065: Based on the feasibility scores of the current action under each action parameter sequence, determine the optimal feasibility score corresponding to the current action.

[0144] Specifically, the maximum feasibility score among the feasibility scores of the current action under each action parameter sequence is determined as the optimal feasibility score.

[0145] Step 07: Based on the similarity scores and optimal feasibility scores corresponding to all actions in each feasible action sequence, determine the total score of the corresponding feasible action sequence.

[0146] As an optional implementation, step 07 includes steps 071 and 072.

[0147] Step 071: Multiply the similarity score and the optimal feasibility score corresponding to each action in each feasible action sequence respectively to obtain the comprehensive score of the corresponding action.

[0148] Specifically, the calculation formula for the comprehensive score of any action is:

[0149] .

[0150] in, For the Action The comprehensive score of .

[0151] Step 072: Multiply the comprehensive scores of all actions in each available action sequence to obtain the total score of the corresponding available action sequence.

[0152] Step 08: Determine the feasible action sequence with the largest total score as the optimal action sequence for the assembly robot arm.

[0153] Step 09: Determine the optimal action parameter sequence based on the pre-trained action library and the optimal action sequence.

[0154] As an optional implementation, step 09 includes steps 091 and 092.

[0155] Step 091: Based on the current environment of the assembly site, the first action in the optimal action sequence and the strategy, determine the optimal action parameters of the first action in the optimal action sequence.

[0156] Specifically, based on the current environment of the assembly site, the first action in the optimal action sequence and the strategy, multiple initial action parameters of the first action in the optimal action sequence are determined, and the initial action parameters with the highest probability are determined as the optimal action parameters of the first action in the optimal action sequence.

[0157] Step 092: Based on the optimal action parameters of the first action in the optimal action sequence, the optimal action parameters of each action in the optimal action sequence are determined in sequence, thereby determining the optimal action parameter sequence.

[0158] Specifically, according to , based on the optimal action parameters of the first action in the optimal action sequence, the optimal action parameters of each action in the optimal action sequence are determined in turn, thereby determining the optimal action parameter sequence.

[0159] Step 10: Based on the pre-trained action library, the optimal action parameter sequence is converted into the motion of each joint of the assembly robot arm to perform the assembly task.

[0160] Specifically, the optimal action parameter sequence is substituted into the pre-trained action library in step 1 to execute the underlying planner, and the optimal action parameter sequence is converted into the movement of each joint of the robotic arm, finally completing the assembly task from the complex assembly site.

[0161] Furthermore, real-time feedback of environmental information (such as part coordinates and images of the assembly site) is provided by the underlying planner, supporting dynamic adjustment of action parameters to adapt to uncertain interference in the assembly process.

[0162] In an exemplary embodiment, a computer device is provided, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a method for task planning of an assembly robot arm.

[0163] In an exemplary embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, a method for task planning of an assembly robot arm is implemented.

[0164] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for task planning of an assembly robot arm is implemented.

[0165] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0166] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0167] The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may include, but are not limited to, general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic units, data processing logic units based on quantum computing, and the like.

[0168] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0169] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0170] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A task planning method for an assembly robot arm, characterized in that: The assembly robot arm task planning method comprises: Acquire multimodal data of the assembly robot arm; the multimodal data includes: an image of the assembly site and a text description of the assembly task; Determining a joint feature code based on the image and the text description; Determining assembly retrieval results based on the joint feature code and the assembly knowledge base; determining a plurality of assembly subtask sequences of the assembly task based on the multimodal data, the assembly retrieval result, and a large language model; Based on the pre-trained action library, multiple feasible action sequences of each assembly subtask in each assembly subtask sequence are determined respectively; Determine the similarity score and optimal feasibility score corresponding to each action in each feasible action sequence; Determine the total score of the corresponding feasible action sequence based on the similarity scores and optimal feasibility scores corresponding to all actions in each feasible action sequence; The feasible action sequence with the largest total score is determined as the optimal action sequence for the assembly robot; Determine the optimal action parameter sequence based on the pre-trained action library and the optimal action sequence; Based on the pre-trained motion library, the optimal motion parameter sequence is converted into the motion of each joint of the assembly robot arm to perform the assembly task.

2. The assembly robot task planning method according to claim 1, characterized in that: Before obtaining multimodal data of the assembly robot arm, it also includes: defining a pre-trained action library and an assembly knowledge base.

3. The assembly robot task planning method according to claim 2, characterized in that: The pre-trained action library includes: action parameters of multiple actions, strategies for selecting optimal action parameters for each action in any environment, and value functions; The assembly knowledge base includes: a parts body library, an assembly relationship map and an assembly process knowledge tree; The component body library is used to define the geometric properties, physical properties and functional properties of the parts; The assembly relationship diagram is used to describe the assembly relationship between various parts; The assembly process knowledge tree is used to represent the sequence of assembly processes.

4. The assembly robot task planning method according to claim 3, characterized in that: Determining a joint feature code based on the image and the text description includes: Preprocessing the image to obtain an initial image vector; the preprocessing includes: image segmentation, block embedding and position coding; Performing feature extraction on the initial image vector to obtain a visual feature vector; Encode the text description through the BERT model to obtain a text feature vector; The visual feature vector and the text feature vector are weightedly integrated to obtain the joint feature code.

5. The assembly robot task planning method according to claim 4, characterized in that: Determining assembly retrieval results based on the combined feature code and the assembly knowledge base includes: Filtering out all parts whose cosine similarity with the joint feature code is greater than a first preset threshold from the part body database to obtain a plurality of related parts; Based on the assembly relationship diagram, determine the assembly relationship between related parts; Determine the assembly process sequence based on the assembly process knowledge tree and the assembly relationship between related parts; The related parts, the assembly relationships between the related parts, and the assembly process sequence are determined as the assembly retrieval results.

6. The assembly robot task planning method according to claim 5, characterized in that: Based on the pre-trained action library, multiple feasible action sequences for each assembly subtask in each assembly subtask sequence are determined, including: Determine any assembly subtask as a subtask to be determined; Determine the cosine similarity between the subtask to be determined and each action in the pre-trained action library; All actions corresponding to the subtask to be determined whose cosine similarity is greater than a second preset threshold and the assembly search results are input into the large language model to obtain multiple feasible action sequences of the subtask to be determined.

7. The assembly robot task planning method according to claim 6, characterized in that: Determine the similarity score and optimal feasibility score corresponding to each action in each feasible action sequence, including: Determine any assembly subtask as the current subtask, determine any feasible action sequence of the current assembly subtask as the current action sequence, and determine any action in the current action sequence as the current action; Calculate the similarity score between the current action and the current subtask, and determine the similarity score between the current action and the current subtask as the similarity score corresponding to the current action; Based on the strategies in the pre-trained action library, determine multiple action parameter sequences of the current action sequence; Based on the value function in the pre-trained action library, the feasibility score of the current action under each action parameter sequence is determined; Based on the feasibility scores of the current action under each action parameter sequence, the optimal feasibility score corresponding to the current action is determined.

8. The assembly robot task planning method according to claim 7, characterized in that: Based on the similarity scores and optimal feasibility scores of all actions in each feasible action sequence, the total score of the corresponding feasible action sequence is determined, including: Multiply the similarity score and optimal feasibility score of each action in each feasible action sequence to obtain the comprehensive score of the corresponding action; The comprehensive scores of all actions in each action sequence are multiplied to obtain the total score of the corresponding action sequence.

9. The assembly robot task planning method according to claim 8, characterized in that: Based on the pre-trained action library and the optimal action sequence, the optimal action parameter sequence is determined, including: Determine optimal action parameters of the first action in the optimal action sequence based on the current environment of the assembly site, the first action in the optimal action sequence, and the strategy; Based on the optimal action parameters of the first action in the optimal action sequence, the optimal action parameters of each action in the optimal action sequence are determined in sequence, thereby determining the optimal action parameter sequence.

10. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the assembly robot task planning method according to any one of claims 1 to 9.

Images