Robot task planning method based on multi-modal large model

Abstract

The application relates to the technical field of robot control, and discloses a robot task planning method based on a multimodal large model. The application collects multimodal data of a robot motion scene, including voice information and visual information, pre-processes the information to obtain a task instruction sequence required to be executed by the robot and an entity data set in a scene where the robot is located, inputs the pre-processed data into a multimodal large model based on a large language model, calculates, based on the pre-processed data, a semantic evaluation coefficient, a visual matching coefficient, a feasibility coefficient and a priority coefficient corresponding to each task instruction in the task instruction sequence, and obtains a task comprehensive execution coefficient corresponding to each task instruction by weighted summation, reorders the task instructions of the task instruction sequence based on the task comprehensive execution coefficient, and obtains a final task planning sequence, thereby improving the rationality and efficiency of task execution of the robot.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, specifically to a robot task planning method based on a multimodal large model. Background Technology

[0002] With the popularization of intelligent robot technology, robots are increasingly being used in complex scenarios such as home services, industrial manufacturing, and medical care. Traditional robot task planning methods often rely on single-modal input, such as recognizing objects solely through predefined visual markers or understanding voice commands only through a fixed set of instructions. However, task instructions in real-world environments often have high semantic ambiguity; for example, in the phrase "bring that thing here," "that thing" and "thing" lack clear referents. Furthermore, a single vision system struggles to understand the user's higher-level intentions. In addition, existing methods typically only consider temporal order or simple rules when evaluating task priority, neglecting the combined influence of multiple dimensions such as semantic complexity, entity matching, and spatial accessibility. This leads to unreasonable task execution order, low efficiency, and even failure. Therefore, how to construct an intelligent method that integrates speech semantics and visual spatial information and performs dynamic task planning based on multi-dimensional quantitative evaluation has become a critical problem urgently needing to be solved in the field of robot control technology. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention provides a robot task planning method based on a multimodal large model. This method involves collecting multimodal data of the robot's motion scene and constructing a multimodal large model based on a large language model. The multimodal large model analyzes and evaluates the scene data to obtain semantic evaluation coefficients, visual matching coefficients, feasibility coefficients, and optimization level coefficients for task instructions, thereby obtaining a comprehensive task execution index. Based on this comprehensive task execution index, the task instructions are reordered to obtain the final task planning sequence. This method improves the rationality and efficiency of robot task execution and solves the aforementioned technical problems.

[0004] To achieve the above objectives, the present invention provides the following technical solution: a robot task planning method based on a multimodal large model, comprising the following steps:

[0005] S1: Acquire multimodal data of the robot's motion scene;

[0006] S2: Preprocess the multimodal data to obtain preprocessed data;

[0007] S3: Construct a multimodal large model based on a large language model;

[0008] S4: Input the preprocessed data from S2 into the multimodal large model constructed in S3, and output the task planning sequence. .

[0009] As a preferred embodiment of the present invention, the S1 multimodal data includes voice information and visual information.

[0010] As a preferred embodiment of the present invention, the S2 preprocessing includes preprocessing of speech information and visual information. Specifically, the preprocessing of speech information involves converting the speech information input by the user into text, cleaning and segmenting the text to obtain clean speech instruction text, which constitutes a task instruction sequence, expressed as: ,in, Indicates a sequence of task instructions. , and They represent the first Article, No. Article and Section The first task instruction, the first A task instruction is a structured triple, expressed as: ,in, , and They represent the first The subject, predicate, and object of a task instruction.

[0011] As a preferred embodiment of the present invention, the preprocessing of visual information specifically involves: identifying entity data in the scene where the robot is located from the video frames collected by the robot, forming an entity dataset, the expression of which is: ,in, , and They represent the first The, the The and the first The first entity data is specifically the location of the entity.

[0012] As a preferred embodiment of the present invention, the S3 multimodal large model includes a data storage module, an evaluation and analysis module, and a task planning module. The data storage module is used to receive and store the preprocessed data obtained in S2, and send it to the evaluation and analysis module. The evaluation and analysis module includes a semantic understanding unit, a visual matching unit, a feasibility evaluation unit, and a priority evaluation unit. The semantic understanding unit is used to analyze each task instruction in the task instruction sequence. Semantic analysis is performed to obtain the semantic evaluation coefficient, which is expressed as follows:

[0013]

[0014] in, Indicates the first The semantic evaluation coefficient of each task instruction; Indicates the first The predicate complexity score of each task instruction is obtained by evaluation using a large language model, and the value range is [value range missing]. ; Indicates the first The specificity score of the object of each task instruction, with a value range of [value range missing]. ; and These are the weighting coefficients. ;

[0015] The first The specificity score of the object of the task instruction The expression is as follows:

[0016]

[0017] in, Indicates the number of modifiers in the object; Indicates the total number of words contained in the object.

[0018] As a preferred embodiment of the present invention, the visual matching unit is used to match the objects of task instructions in the task instruction sequence. With each entity in the entity dataset Performing matching analysis to obtain visual matching coefficients includes the following steps:

[0019] Step A1: Calculate each entity With object The matching degree coefficient constitutes the matching set. The relevant expressions are as follows:

[0020]

[0021]

[0022] in, Indicates the first The matching degree coefficient of each entity; For the first individual entities With object The semantic similarity, with values ​​ranging from 1 to 2. The value is 1 for a perfect match and 0 for a completely irrelevant match. It is an exponential function; Indicates the first individual entities Location; Indicates the robot's position; Represents Euclidean distance; This is the distance attenuation coefficient; For matching sets; , and The first The, the The and the first The matching degree coefficient corresponding to each entity;

[0023] Step A2: Based on the matching set Choose the highest matching coefficient As a visual matching coefficient Its expression is as follows:

[0024]

[0025] in, Indicates the first Visual matching coefficients for each task instruction; Represents a matching set; This indicates taking the maximum value.

[0026] As a preferred embodiment of the present invention, the feasibility assessment unit is used to assess the feasibility of task execution and obtain a feasibility coefficient, the expression of which is as follows:

[0027]

[0028] in, Indicates the first The feasibility coefficient of each task instruction; Indicates the first The object of a task instruction; This represents the set of all locations that the robot can reach. For indicator functions, when When, the value is 1, when When the value is 0; Indicates the adjustment factor; Indicates the first Score the predicate complexity of each task instruction.

[0029] As a preferred embodiment of the present invention, the priority evaluation unit determines the priority coefficient of the task based on an emergency keyword database, specifically: when the person inputs the first priority coefficient... The task instruction contains words from the emergency keyword database, the first one Priority coefficient corresponding to each task instruction ,otherwise .

[0030] As a preferred embodiment of the present invention, the evaluation and analysis module is used to calculate the overall task execution coefficient, the expression of which is as follows:

[0031]

[0032] in, Indicates the first The overall execution coefficient of each task instruction; , , and They represent the first The semantic evaluation coefficient, visual matching coefficient, feasibility coefficient, and priority coefficient of each task instruction; , , and These are the weighting coefficients, .

[0033] As a preferred embodiment of the present invention, the task planning module is based on the comprehensive task execution coefficient corresponding to each task instruction. Sequence of task instructions input by personnel Reorder the tasks to obtain the final task planning sequence. .

[0034] Compared with existing technologies, this invention provides a robot task planning method based on a multimodal large model, which has the following beneficial effects:

[0035] This invention collects multimodal data from robot motion scenarios, including voice and visual information. The data is preprocessed to obtain a sequence of task instructions the robot needs to execute and a dataset of entities in the robot's scene. The preprocessed data is then input into a large multimodal model based on a large language model. This model includes a data storage module, an evaluation and analysis module, and a task planning module. The data storage module receives and stores the preprocessed data and sends it to the evaluation and analysis module. Based on the preprocessed data, the evaluation and analysis module calculates the semantic evaluation coefficient, visual matching coefficient, feasibility coefficient, and priority coefficient for each task instruction in the task instruction sequence, and then performs a weighted summation to obtain the comprehensive task execution coefficient for each instruction. The task planning module reorders the task instructions in the task instruction sequence based on the comprehensive task execution coefficient to obtain the final task planning sequence. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the process of the present invention. Detailed Implementation

[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] Please see Figure 1 A robot task planning method based on a multimodal large model includes the following steps:

[0039] S1: Acquire multimodal data of the robot's motion scene, including voice and visual information;

[0040] S2: Preprocess the multimodal data to obtain preprocessed data;

[0041] Preprocessing includes preprocessing of speech and visual information. Specifically, speech preprocessing involves converting the input speech information into text, and then cleaning and segmenting the text. Cleaning removes modal particles, punctuation marks, and repeated words. Segmentation divides the text into multiple words to obtain clean speech instruction text, forming a task instruction sequence, the expression of which is: ,in, Indicates a sequence of task instructions. , and They represent the first Article, No. Article and Section Task instruction, number A task instruction is a structured triple, expressed as: ,in, , and They represent the first The subject, predicate, and object of each task instruction;

[0042] The preprocessing of visual information specifically involves identifying entity data in the scene where the robot is located from the video frames collected by the robot, forming an entity dataset, the expression of which is: ,in, , and They represent the first The, the The and the first There are several entity data points, where the first entity data point specifically represents the location of the entity.

[0043] S3: Construct a multimodal large model based on a large language model;

[0044] The multimodal large model includes a data storage module, an evaluation and analysis module, and a task planning module. The data storage module is used to receive and store the preprocessed data obtained by S2 and send it to the evaluation and analysis module.

[0045] The evaluation and analysis module includes a semantic understanding unit, a visual matching unit, a feasibility evaluation unit, and a priority evaluation unit; the semantic understanding unit is used to analyze each task instruction in the task instruction sequence. Semantic analysis is performed to obtain the semantic evaluation coefficient, which is expressed as follows:

[0046]

[0047] in, Indicates the first The semantic evaluation coefficient of each task instruction; Indicates the first The predicate complexity score of each task instruction is obtained by evaluation using a large language model, and the value range is [value range missing]. ; Indicates the first The specificity score of the object of each task instruction, with a value range of [value range missing]. ; and These are the weighting coefficients. Multimodal large model is a model built on a large language model. The predicate is input into the large language model, and the large language model is asked to score its complexity to obtain the result. Predicate complexity score for task instructions ;

[0048] Semantic evaluation coefficient The larger the value, the higher the semantic information and clarity of the task instruction: on the one hand, high predicate complexity means that the task itself needs more refined action decomposition, which is worth prioritizing to avoid subsequent conflicts; on the other hand, high object specificity means that the task target is more easily identified and located in the scene.

[0049] No. The specificity score of the object of the task instruction The expression is as follows:

[0050]

[0051] in, Indicates the number of modifiers in the object; Indicates the total number of words contained in the object;

[0052] The visual matching unit is used to match the objects of task instructions in a sequence of task instructions. With each entity in the entity dataset Performing matching analysis to obtain visual matching coefficients includes the following steps:

[0053] Step A1: Calculate each entity With object The matching degree coefficient constitutes the matching set. The relevant expressions are as follows:

[0054]

[0055]

[0056] in, Indicates the first The matching degree coefficient of each entity; For the first individual entities With object The semantic similarity, with values ​​ranging from 1 to 2. The value is 1 for a perfect match and 0 for a completely irrelevant match. The semantic similarity is also evaluated by the large language model. It is an exponential function; Indicates the first individual entities Location; Indicates the robot's position; Represents Euclidean distance; This is the distance attenuation coefficient, which controls the rate attenuation of the effect of distance. For matching sets; , and The first The, the The and the first The matching degree coefficient corresponding to each entity;

[0057] Matching coefficient It comprehensively reflects the degree of semantic matching between the object in the task instruction and the entity in the scene, as well as the spatial distance of the entity relative to the robot; The larger the value, the more likely it is that there is an entity in the scene that closely matches the semantics of the instruction and is close to the robot, resulting in better visual matching and higher reliability and efficiency of task execution.

[0058] Step A2: Based on the matching set Choose the highest matching coefficient As a visual matching coefficient Its expression is as follows:

[0059]

[0060] in, Indicates the first Visual matching coefficients for each task instruction; Represents a matching set; This indicates taking the maximum value;

[0061] The feasibility assessment unit is used to evaluate the feasibility of task execution and obtain the feasibility coefficient, which is expressed as follows:

[0062]

[0063] in, Indicates the first The feasibility coefficient of each task instruction; Indicates the first The object of a task instruction; This represents the set of all locations that the robot can reach. For indicator functions, when When, the value is 1, when When the value is 0; Indicates the adjustment factor; Indicates the first Score the predicate complexity of each task instruction;

[0064] Feasibility coefficient The larger the value, the lower the complexity of the current task instruction. This indicates that not only is the target object within the robot's reachable range, but the predicate action also has lower complexity (i.e., ...). If the indicator function is smaller, the robot can complete the task with a higher success rate and less energy consumption; conversely, if the indicator function is 0, the feasibility coefficient is directly 0, indicating that the task cannot be performed under the current conditions.

[0065] The priority assessment unit determines the priority coefficient of a task based on an emergency keyword database. Specifically, when the person inputs the first priority term... The task instruction contains words from the emergency keyword database, the first one Priority coefficient corresponding to each task instruction ,otherwise The emergency keyword database was obtained from the thesaurus.net and Tsinghua University Open Chinese Thesaurus, among others.

[0066] The evaluation and analysis module is used to calculate the overall task performance coefficient, and its expression is as follows:

[0067]

[0068] in, Indicates the first The overall execution coefficient of each task instruction; , , and They represent the first The semantic evaluation coefficient, visual matching coefficient, feasibility coefficient, and priority coefficient of each task instruction; , , and These are the weighting coefficients, ;

[0069] The task planning module is based on the overall task execution coefficient corresponding to each task instruction. Sequence of task instructions input by personnel Reorder the tasks to obtain the final task planning sequence. ;

[0070] S4: Input the preprocessed data from S2 into the multimodal large model constructed in S3, and output the task planning sequence. .

[0071] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A robot task planning method based on a multimodal large model, characterized by: Includes the following steps: S1: Acquire multimodal data of the robot's motion scene; S2: Preprocess the multimodal data to obtain preprocessed data; S3: Construct a multimodal large model based on a large language model; S4: Input the preprocessed data from S2 into the multimodal large model constructed in S3, and output the task planning sequence. .

2. The robot task planning method based on a multimodal large model according to claim 1, characterized in that: The S1 multimodal data includes speech information and visual information.

3. The robot task planning method based on a multimodal large model according to claim 1, characterized in that: The S2 preprocessing includes preprocessing of speech and visual information. Specifically, the speech preprocessing involves converting the user-input speech information into text, cleaning and segmenting the text to obtain clean speech instruction text, which forms a task instruction sequence. The expression for this sequence is: ,in, Indicates a sequence of task instructions. , and They represent the first Article, No. Article and Section The first task instruction, the first A task instruction is a structured triple, expressed as: ,in, , and They represent the first The subject, predicate, and object of a task instruction.

4. The robot task planning method based on a multimodal large model according to claim 3, characterized in that: The preprocessing of visual information specifically involves: identifying entity data in the scene where the robot is located from the video frames collected by the robot, forming an entity dataset, the expression of which is: ,in, , and They represent the first The, the The and the first The first entity data refers to the location of the entity.

5. The robot task planning method based on a multimodal large model according to claim 4, characterized in that: The S3 multimodal large model includes a data storage module, an evaluation and analysis module, and a task planning module. The data storage module receives and stores the preprocessed data obtained in S2 and sends it to the evaluation and analysis module. The evaluation and analysis module includes a semantic understanding unit, a visual matching unit, a feasibility evaluation unit, and a priority evaluation unit. The semantic understanding unit is used to analyze each task instruction in the task instruction sequence. Semantic analysis is performed to obtain the semantic evaluation coefficient, which is expressed as follows: ； in, Indicates the first The semantic evaluation coefficient of each task instruction; Indicates the first The predicate complexity score of each task instruction is obtained by evaluation using a large language model, and the value range is [value range missing]. ; Indicates the first The specificity score of the object of each task instruction, with a value range of [value range missing]. ; and These are the weighting coefficients. ; The first The specificity score of the object of the task instruction The expression is as follows: ； in, Indicates the number of modifiers in the object; Indicates the total number of words contained in the object.

6. The robot task planning method based on a multimodal large model according to claim 5, characterized in that: The visual matching unit is used to identify the objects of task instructions in the task instruction sequence. With each entity in the entity dataset Performing matching analysis to obtain visual matching coefficients includes the following steps: Step A1: Calculate each entity With object The matching degree coefficients constitute the matching set. The relevant expressions are as follows: ； ； in, Indicates the first The matching degree coefficient of each entity; For the first individual entities With object The semantic similarity, with values ​​ranging from 1 to 2. The value is 1 for a perfect match and 0 for a completely irrelevant match. It is an exponential function; Indicates the first individual entities Location; Indicates the robot's position; Represents Euclidean distance; This is the distance attenuation coefficient; For matching sets; , and The first The, the The and the first The matching degree coefficient corresponding to each entity; Step A2: Based on the matching set Choose the highest matching coefficient As a visual matching coefficient Its expression is as follows: ； in, Indicates the first Visual matching coefficients for each task instruction; Represents a matching set; This indicates taking the maximum value.

7. The robot task planning method based on a multimodal large model according to claim 5, characterized in that: The feasibility assessment unit is used to evaluate the feasibility of task execution and obtain a feasibility coefficient, the expression of which is as follows: ； in, Indicates the first The feasibility coefficient of each task instruction; Indicates the first The object of a task instruction; This represents the set of all locations that the robot can reach. For indicator functions, when When, the value is 1, when When the value is 0; Indicates the adjustment factor; Indicates the first Score the predicate complexity of each task instruction.

8. The robot task planning method based on a multimodal large model according to claim 5, characterized in that: The priority evaluation unit determines the priority coefficient of the task based on an emergency keyword database. Specifically, when the person inputs the first priority, the priority coefficient is determined as follows: The task instruction contains words from the emergency keyword database, the first one Priority coefficient corresponding to each task instruction ,otherwise .

9. The robot task planning method based on a multimodal large model according to claim 5, characterized in that: The evaluation and analysis module is used to calculate the overall task execution coefficient, which is expressed as follows: ； in, Indicates the first The overall execution coefficient of each task instruction; , , and They represent the first The semantic evaluation coefficient, visual matching coefficient, feasibility coefficient, and priority coefficient of each task instruction; , , and These are the weighting coefficients, .

10. The robot task planning method based on a multimodal large model according to claim 9, characterized in that: The task planning module is based on the overall task execution coefficient corresponding to each task instruction. Sequence of task instructions input by personnel Reorder the tasks to obtain the final task planning sequence. .

Images