Intelligent task planning method and device based on environment perception, equipment and medium

Abstract

The invention discloses an intelligent task planning method and device based on environment perception, equipment and a medium, and the method comprises the steps: fusing abstract environment information into a large language model, and carrying out the fine adjustment of the large language model, so as to enhance the environment perception capability of the large language model; inputting a task target and an environment state table into the fine-tuned large language model, and generating a multi-step action sequence with the highest probability through a step-by-step beam search strategy; according to the generated action sequence, the environment data state table is updated by simulating interaction between the robot agent and other objects, and therefore interaction tracks corresponding to the steps are generated. According to the method, the model is adjusted into an environment perception large language model fine tuning structure, so that higher accuracy and practicability are brought to intelligent task planning; the large language model can explore a wider search space by utilizing a step-by-step beam search strategy, so that more comprehensive search is beneficial to generating a more reasonable and effective plan.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an embodied intelligent task planning method, device, equipment and medium based on environment perception. Background Art

[0002] As a key research direction in the field of artificial intelligence, embodied intelligent task planning is committed to promoting the understanding and execution of natural language instructions by intelligent agents in physical environments. In recent years, the breakthroughs of large language models in natural language processing have brought unprecedented opportunities for the development of embodied intelligent task planning. With their excellent language understanding and reasoning capabilities, these models are able to parse complex instructions and generate a series of action steps based on them. For example, when receiving the instruction "Put the laptop under the desk lamp", the large language model can plan detailed actions such as "walk to the desk lamp", "pick up the laptop", "move under the desk lamp", "place the laptop", etc.

[0003] However, despite the great potential of large language models in theory, there are still many challenges and limitations in their practical application in embodied intelligent task planning.

[0004] First, current large language models mainly focus on processing text information, but lack the ability to understand physical environment information. Although these models can understand language instructions, they often cannot accurately grasp the objects, attributes, and relationships between them in the environment, making the generated action plans difficult to implement in the real world. For example, the model may not be able to generate an effective navigation path based on the specific layout of the room, or it may not be able to identify and correctly operate different objects. This limitation stems from the lack of environmental perception capabilities of large language models, making it difficult to effectively integrate environmental information with language instructions.

[0005] In the field of embodied intelligence task planning, existing large language models face some significant challenges in search strategies. Specifically, traditional maximum a posteriori probability methods and beam search methods often easily lead to local optimal solutions in the process of generating action sequences, and cannot guarantee that the generated plan is globally optimal.

[0006] First, when generating an action sequence, the maximum a posteriori probability method adopts a greedy strategy. It only considers the token with the highest probability in the current situation at each step. The problem with this method is that it ignores the overall probability distribution of the action sequence and only focuses on the maximum probability of a single token. This locally optimal choice may result in a logically incoherent action sequence generated finally, and may even violate the grammar rules. For example, even if an action is the most likely in the current situation, it may be incompatible with the previous actions or unable to provide a reasonable transition for subsequent actions, thus causing problems when the entire action sequence is actually executed.

[0007] Secondly, the beam search method is a breadth - first search strategy. It retains multiple candidate action sequences at each step and selects the optimal ones to continue expanding from them. Compared with the maximum a posteriori probability method, this method can explore a broader search space and thus has a greater possibility of finding the global optimal solution. However, the beam search method also has its limitations. First, its computational complexity is relatively high because it needs to maintain and evaluate multiple candidate sequences at each step. This is particularly obvious in large - scale or complex task planning, which may lead to a large consumption of computing resources. Secondly, the beam search method is very sensitive to the selection of hyperparameters, especially the size of the beam search width. If the width is set too small, it may lead to a restricted search space and miss the global optimal solution; while if it is set too large, it will increase the computational burden and may not significantly improve the quality of the solution.

[0008] In the research and practice of embodied intelligence task planning, the accuracy of the evaluation method is a crucial link. Unfortunately, the currently adopted evaluation methods have obvious deficiencies in accuracy. Specifically, both the simulation - based evaluation method and the key action sequence - based evaluation method fail to provide an accurate evaluation of the actual execution effect of the action plan.

[0009] First, the simulation - based evaluation method is usually carried out in a virtual environment. Although these environments can simulate some aspects of the real world, they often cannot fully reproduce the complexity and dynamics of the real environment. For example, the simulation environment may not be able to accurately simulate the movement trajectory of objects in reality, the change of light intensity, or unexpected disturbances in the environment, such as the movement of people, the change of temperature, etc. These factors may have a significant impact on the actions of the intelligent agent in the real world, but are often ignored in the simulation evaluation, resulting in a large deviation between the evaluation result and the actual execution effect.

[0010] Secondly, as a commonly used evaluation tool, the evaluation method based on key action sequences mainly focuses on whether the agent can complete key actions when performing tasks. However, a significant drawback of this evaluation method is that it only focuses on the actions themselves and ignores the objects and contexts of action execution. In actual task execution, the same action may produce different results for different objects, and the evaluation method based on key action sequences cannot capture this difference. Therefore, even if the agent executes tasks according to the predetermined action sequence, the task may still fail due to misunderstandings of object attributes or the influence of environmental changes.

[0011] The root cause of the inaccuracy of these evaluation methods lies in their failure to effectively track and simulate changes in environmental states. In the real world, the environment is dynamically changing. The actions of the agent not only affect the environmental state but also receive feedback from the environmental state. Therefore, in order to truly present the execution process of the action plan, the evaluation method needs to be able to capture and simulate changes in environmental states in real time, as well as the impact of these changes on the agent's actions. Summary of the Invention

[0012] To at least to some extent solve one of the technical problems existing in the prior art, the object of the present invention is to provide an embodied intelligent task planning method, device, equipment and medium based on environmental perception.

[0013] The first technical solution adopted by the present invention is as follows:

[0014] An embodied intelligent task planning method based on environmental perception, comprising the following steps:

[0015] By integrating abstract environmental information into the large language model, fine-tuning the large language model to enhance the environmental perception ability of the large language model;

[0016] Input the task objective and the environmental state table into the fine-tuned large language model, and generate the action sequence with the highest probability for multiple steps through a step-by-step beam search strategy; the environmental state table is obtained through environmental perception;

[0017] According to the generated action sequence, by simulating the interaction between the robotic agent and other objects, update the environmental data state table, thereby generating an interaction trajectory corresponding to the steps.

[0018] Further, the environmental information includes object attributes, object positions, and object relationships;

[0019] The expression for fine-tuning the large language model is:

[0020] W′ = W + ΔW

[0021] ΔW = LoRA(E, W)

[0022] Among them, W is the original weight of the large language model, ΔW is the weight adjustment obtained by the LoRA fine-tuning method, and W′ is the weight of the large language model after adjustment;

[0023] The fine-tuned large language model can generate effective navigation paths according to the spatial layout or perform corresponding operations according to the object name and location.

[0024] Furthermore, generating the action sequence with the highest probability for multiple steps through a step-by-step beam search strategy includes:

[0025] Determine the beam search width;

[0026] At each time step, use the beam search method to generate the n tokens with the highest probability as candidate steps;

[0027] At each time step, select the step with the highest probability from the candidate steps and add it to the plan;

[0028] Among them, the value of the beam search width is adjusted according to the task complexity and computational resource limitations.

[0029] Furthermore, the expression of the step-by-step beam search strategy is:

[0030]

[0031] In the formula, K is the beam width, that is, the search width; t = 0,..., T, where T is the maximum number of steps in the task, and [t] represents the set of natural numbers from 0 to t; S represents all possible step sequences, and S [t] ={S 1,[t] ,..., S K,[t]} represents the top K steps with the highest probability selected from all possible steps from 0 to t;

[0032] θ(s k,t ) = log P(s k,t |s k,t-1 ,..., s k,1 )

[0033] In the formula, θ(s k,t ) represents the logarithm of the conditional probability of the output step s k,t for all previous output steps s k,t-1 ,..., s k,1 .

[0034] Furthermore, updating the environmental data status table by simulating the interaction between the robot agent and other objects according to the generated action sequence, so as to generate an interaction trajectory corresponding to the steps, includes:

[0035] Based on the generated action sequence, the named entity recognition technology is used to obtain the keywords in each step;

[0036] Match the nouns in the keywords with the objects in the environmental status table, and match the verbs in the keywords with the predefined action library to obtain the action-object pairs for each step;

[0037] According to the obtained action-object pairs, simulate the agent interaction trajectory and update the environmental status table until all steps are simulated, and obtain the interaction trajectory including the environmental status table after each update.

[0038] Furthermore, the embodied intelligent task planning method further includes the step of calculating the trajectory matching score:

[0039] For each task, compare the environmental status table in the trajectory simulation with the expected interaction trajectory, and calculate the trajectory matching score for this task to evaluate the ability of the large language model in embodied task planning.

[0040] Furthermore, the calculation formula of the trajectory matching score is:

[0041]

[0042] In the formula, S success represents the number of successful steps, T total represents the total number of steps; w is the weight factor for the final state matching, and F represents the result of the final state matching.

[0043] The second technical solution adopted by the present invention is:

[0044] An embodied intelligent task planning device based on environmental perception, comprising:

[0045] A model fine-tuning module for fine-tuning the large language model by integrating abstract environmental information into the large language model to enhance the environmental perception ability of the large language model;

[0046] A step generation module for inputting the task objective and the environmental status table into the fine-tuned large language model, and generating the action sequence with the highest probability for multiple steps through a step-by-step beam search strategy; the environmental status table is obtained through environmental perception;

[0047] A trajectory generation module for updating the environmental data status table by simulating the interaction between the robot agent and other objects according to the generated action sequence, so as to generate an interaction trajectory corresponding to the steps.

[0048] The third technical solution adopted by the present invention is:

[0049] An electronic device, which includes a processor and a memory. At least one instruction, at least one program, a code set or an instruction set is stored in the memory, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the above-mentioned method for environment-aware embodied intelligent task planning.

[0050] The fourth technical solution adopted by the present invention is:

[0051] A computer-readable storage medium, in which at least one instruction, at least one program, a code set or an instruction set is stored, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the above-mentioned method for environment-aware embodied intelligent task planning.

[0052] The fifth technical solution adopted by the present invention is:

[0053] A computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to enable the computer device to execute the above-mentioned method.

[0054] The beneficial effects of the present invention are as follows: By adjusting the model to the fine-tuning structure of an environment-aware large language model, the present invention brings higher accuracy and practicability to embodied intelligent task planning; the large language model can explore a broader search space by using the step-by-step beam search strategy, and further, the more comprehensive search helps to generate more reasonable and effective plans. In addition, the step-by-step beam search strategy method helps to reduce the problem of error accumulation. Description of the Drawings

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following introduces the accompanying drawings related to the technical solutions in the embodiments of the present invention or the prior art. It should be understood that the accompanying drawings below only conveniently and clearly show some embodiments of the technical solutions in the present invention. For those skilled in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0056] Figure 1 is the flowchart of the steps of the method for environment-aware embodied intelligent task planning in the embodiment of the present invention;

[0057] Figure 2 is the schematic framework diagram of the step-by-step beam search strategy in the embodiment of the present invention;

[0058] Figure 3It is a framework schematic diagram of the embodied intelligent task planning method that integrates environment perception and language understanding capabilities in the embodiments of the present invention. Detailed implementation manners

[0059] The embodiments of the present invention will be described in detail below. Examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals represent the same or similar elements or elements with the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention and should not be construed as a limitation of the present invention. For the step numbers in the following embodiments, they are only set for the convenience of description and illustration, and no limitation is imposed on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0060] In the description of the present invention, it should be understood that for the orientation description, such as the upper, lower, front, rear, left, right, etc., the orientation or positional relationship indicated is based on the orientation or positional relationship shown in the accompanying drawings. It is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention.

[0061] In the description of the present invention, the meaning of several is one or more, the meaning of multiple is two or more, greater than, less than, exceeding, etc. are understood as not including the present number, and above, below, within, etc. are understood as including the present number. If there is a description of first and second, it is only for the purpose of distinguishing technical features and should not be understood as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features or implicitly indicating the sequence of the indicated technical features.

[0062] In the description of the present invention, unless otherwise clearly defined, words such as setting, installing, connecting, etc. should be understood in a broad sense, and those skilled in the art can reasonably determine the specific meanings of the above words in the present invention in combination with the specific content of the technical solution.

[0063] In order to overcome the problems of the existing large language models lacking environment perception ability, imperfect search strategies, and insufficient evaluation accuracy in embodied intelligent task planning, the present invention proposes a comprehensive solution, aiming to enhance the environment perception ability of the large language model, optimize the search strategy, and improve the evaluation accuracy, so as to effectively improve the performance of the large language model in embodied intelligent task planning.

[0064] Embodiment 1

[0065] As Figure 1 shown, this embodiment provides an embodied intelligent task planning method based on environment perception, including the following steps:

[0066] S1. By integrating abstract environmental information into the large language model, fine-tune the large language model to enhance its environmental perception ability.

[0067] To enhance the environmental perception ability of the large language model, in this embodiment, the large language model is fine-tuned, and the expression is as follows:

[0068] W′ = W + ΔW

[0069] ΔW = LoRA(E, W)

[0070] Where W is the original weight of the large language model, ΔW is the weight adjustment obtained through the LoRA fine-tuning method, and W′ is the weight of the large language model after adjustment.

[0071] By integrating abstract environmental information (such as object attributes, positions, relationships, etc.) into the large language model, it enables the model to understand the context of instructions and generate action plans that are more in line with the actual situation. For example, the large language model can generate an effective navigation path based on the room layout, or perform corresponding operations based on the object name and position.

[0072] In one embodiment, the large language model uses the GLM-130B model as the base model and uses the LoRA method to fine-tune the large language model. Specifically, the fine-tuning dataset is: the MOSS dataset and the ALFRED dataset are mixed in a ratio of 5:1 to fine-tune the language understanding and reasoning ability of the large language model while maintaining its ability to understand environmental information. The fine-tuned large language model has achieved significant performance improvement in the embodied intelligence task planning task. For example, in the ALFRED benchmark test, both the plan generation quality and the task success rate have been significantly improved.

[0073] It should be noted that in addition to using the GLM-130B model as the base model, the large language model can also use other types of large language models, such as BERT, GPT, etc., as the base model. In addition, in addition to the above datasets, other corpora containing environmental information, such as object description datasets, scene description datasets, etc., can also be used as the fine-tuning datasets.

[0074] S2. Input the task objective and the environmental state table into the fine-tuned large language model, and generate the action sequence with the highest probability for multiple steps through a step-by-step beam search strategy; the environmental state table is obtained through environmental perception.

[0075] See Figure 2, To optimize the generation search strategy of the large language model, the present embodiment creates and adopts an innovative step-by-step beam search strategy, using the steps in the planning result as the unit for search during the decoding process. During the decoding process of the large language model, the step-by-step beam search strategy generates the action sequence with the highest probability for multiple steps, rather than only selecting the action sequence with the highest probability for the smallest unit of information, 'token'. The specific expression is as follows:

[0076]

[0077] Where K is the beam width, that is, the search width. t = 0,..., T, where T is the maximum number of steps in the task. [t] represents the set of natural numbers from 0 to t. S represents all possible step sequences. S [t] ={S 1,[t] ,..., S K,[t]}, representing the top K steps with the highest probability selected from all possible steps from 0 to t.

[0078] θ(s k,t ) = log P(s k,t |s k,t-1 ,..., s k,1 ),

[0079] Where θ(s k,t ) represents the logarithm of the conditional probability of the output step s k,t for all previous output steps s k,t-1 ,..., s k,1 . Usually, the sum of logarithmic probabilities is used instead of the direct product to maintain numerical stability.

[0080] The step-by-step beam search strategy calculates the joint probability of each step in the generated plan, which helps to generate a more reasonable and coherent plan. Applying the step-by-step beam search strategy to the decoding process of the large language model can improve the quality of the generated plan, thereby increasing the success rate of task execution. Exemplarily, Figure 2 The bold steps in

[0081] represent the top K steps with the highest probability selected from all possible steps.

[0082] 1) Beam search width: Set K = 5, that is, generate the action sequence with the highest probability for 5 steps each time.

[0083] 2) Token-level beam search: At each time step, use the beam search method to generate the top 5 tokens with the highest probability as candidate steps.

[0084] 3) Progressive beam search: At each time step, select the step with the highest probability from the candidate steps and add it to the plan.

[0085] Among them, the value of the beam search width can be adjusted according to the task complexity and computing resource limitations. Compared with the traditional beam search method, the progressive beam search strategy generates a higher-quality plan and a higher task success rate.

[0086] S3. According to the generated action sequence, update the environmental data status table by simulating the interaction between the robot agent and other objects, thereby generating an interaction trajectory corresponding to the step.

[0087] In some embodiments, step S3 specifically includes the following steps:

[0088] S31. Perform named entity recognition according to the generated action sequence to obtain the keywords in each step;

[0089] S32. Match the nouns in the keywords with the objects in the environmental status table, and match the verbs in the keywords with the predefined action library to obtain the action-object pairs for each step;

[0090] S33. According to the obtained action-object pairs, simulate the agent interaction trajectory and update the environmental status table until all steps are simulated, and obtain an interaction trajectory including the environmental status table after each update.

[0091] S4. For each task, compare the environmental status table in the trajectory simulation with the expected interaction trajectory, and calculate the trajectory matching score of this task, which is used to evaluate the ability of the large language model in embodied task planning.

[0092] Finally, in order to improve the evaluation accuracy of embodied intelligent task planning, this embodiment of the invention adopts the trajectory matching score as the evaluation index. The trajectory matching index simulates the plan execution process, matches the generated action sequence with the environmental status, and calculates the degree of similarity with the target environmental status.

[0093] Through the above technical solutions, the present invention introduces text-based simulation through the status table, which is more efficient in simulating and evaluating the success rate of tasks in the output plan. Compared with other simulation methods, the proposed evaluation provides text-to-trajectory simulation through text analysis without a complex physical simulation process. It can effectively solve the problems existing in the existing large language model in embodied intelligent task planning, improve its performance and accuracy, enable it to better understand natural language instructions, and execute corresponding tasks in a complex physical environment.

[0094] The following combines the attached Figure 3 and specific implementation manners to detail the method of this embodiment.

[0095] (1) Environmental perception generation

[0096] For each task, extract the structured environmental data status table and add it to the input of the large language model. For example, if the goal of the task is "put the laptop under the lamp", the environmental data status table will include "laptop" and its location, enabling the large language model to consider environmental details during generation. The output of the large language model will be divided into multiple steps for trajectory simulation and task success simulation steps.

[0097] (2) Trajectory simulation

[0098] Trajectory simulation involves updating the environmental data status table by simulating the interaction of the robotic agent with other objects to generate interaction trajectories corresponding to the steps. Specifically, it includes the following steps:

[0099] 1) After removing stop words from the steps, apply named entity recognition technology to obtain the keywords in each step.

[0100] 2) Match the nouns in the keywords with the objects in the environmental status table, and match the verbs in the keywords with the predefined action library to obtain the action-object pairs for each step.

[0101] 3) Utilize these action-object pairs to simulate the interaction trajectory of the agent and update the environmental status table.

[0102] 4) This process continues until all steps are simulated and an interaction trajectory containing the environmental status table after each update is obtained.

[0103] (3) Trajectory matching score calculation

[0104] For each task, this embodiment applies the above process to its goal description to obtain the target object of the task and its desired location. Then, update the environmental status table according to the obtained action-object pairs to achieve the interaction trajectory representing the successful completion of the task.

[0105] Compare the environmental status table in the trajectory simulation with the desired interaction trajectory to calculate the trajectory matching score for this task, which represents the ability of the large language model in embodied task planning.

[0106] The formula for calculating the trajectory matching score is as follows:

[0107]

[0108] Where S success represents the number of successful steps, and T totalrepresents the total number of steps. w is the weight factor for the final state match, and F represents the result of the final state match. This matching method comprehensively evaluates the performance of the agent in executing tasks and emphasizes the importance of achieving the expected final state when the task is completed.

[0109] In summary, compared with the prior art, the method of the present invention has at least the following advantages and beneficial effects:

[0110] 1) Adopt the fine-tuning structure of the environment-aware large language model

[0111] Through this structure, the large language model can understand and analyze environmental information more deeply. This means that the large language model can not only process natural language instructions, but also fully consider various factors in the physical environment, such as spatial layout, object position, environmental state, etc. The improvement of this ability enables the large language model to more effectively combine environmental information with language instructions, thereby generating more accurate and practical action plans.

[0112] In addition, the large language model can also perform corresponding operations according to the name and position of the object. For example, when receiving the instruction "Put the book on the table", the large language model can not only identify the two objects "book" and "table", but also know their specific positions in the environment, and plan a complete action sequence from picking up the book to placing it on the table. This fine-grained operation planning significantly improves the success rate of the plan during actual execution, enabling the intelligent agent to complete tasks more reliably in complex and changing environments. Generally speaking, the introduction of the fine-tuning structure of the environment-aware large language model brings higher accuracy and practicality to embodied intelligent task planning.

[0113] 2) Adopt the method of step-by-step beam search strategy

[0114] The large language model can explore a broader search space by using the method of step-by-step beam search strategy. In the traditional beam search strategy method, the search process is usually linear, which limits the possibilities considered by the model when generating action sequences. The step-by-step beam search strategy method allows the model to explore multiple potential action paths at each step by performing beam search at each step of action selection, thereby increasing the diversity and depth of the search. This more comprehensive search helps to generate more reasonable and effective plans.

[0115] Secondly, the step-by-step beam search strategy method helps to reduce the problem of error accumulation. When generating a long sequence of action plans, early wrong decisions may lead to a chain reaction of subsequent actions, ultimately resulting in the failure of the entire plan. The step-by-step beam search strategy can timely discover and correct errors by making optimized selections at each step, thereby reducing the accumulation of errors in the plan and improving the accuracy and reliability of the entire action sequence.

[0116] 3) Adopt the trajectory matching score evaluation method

[0117] First of all, the trajectory matching score index can more accurately measure the actual execution result of the plan. Traditional evaluation methods often fail to fully consider various situations that may occur during the actual execution process. However, the trajectory matching score can more precisely evaluate the execution effect of the plan by comparing the matching degree between the generated action trajectory and the actual execution trajectory.

[0118] Secondly, the trajectory matching score evaluation method takes into account the dependency relationships between action steps. In embodied intelligence task planning, the successful execution of one step may depend on the correct completion of the previous step. By analyzing these dependency relationships, the trajectory matching score can more comprehensively evaluate the logic and coherence of the large language model when generating action sequences.

[0119] In addition, the trajectory matching score index can more realistically reflect the actual execution process of the plan. Compared with the simulation-based evaluation method, the trajectory matching score is not restricted by the simulation environment and can more accurately capture the impact of environmental state changes on plan execution.

[0120] Embodiment 2

[0121] This embodiment provides an embodied intelligence task planning device based on environmental perception, including:

[0122] A model fine-tuning module for fine-tuning the large language model by integrating abstract environmental information into the large language model to enhance the environmental perception ability of the large language model;

[0123] A step generation module for inputting the task objective and the environmental state table into the fine-tuned large language model and generating the action sequence with the highest probability for multiple steps through a step-by-step beam search strategy; the environmental state table is obtained through environmental perception;

[0124] A trajectory generation module for updating the environmental data state table by simulating the interaction between the robot agent and other objects according to the generated action sequence, thereby generating an interaction trajectory corresponding to the step.

[0125] Since this device is an embodied intelligence task planning device based on environmental perception in an embodiment of the present invention, and the principle of this device to solve problems is similar to that of this method, the implementation of this device can refer to the implementation process of the above method embodiment, and the repeated parts will not be elaborated.

[0126] Embodiment 3

[0127] An embodiment of the present invention further provides an electronic device, which includes a processor and a memory. At least one instruction, at least one program, a code set, or an instruction set is stored in the memory, and is loaded and executed by the processor to implement as Figure 1 a method for embodied intelligent task planning based on environmental perception as shown.

[0128] It can be understood that the memory may include a Random Access Memory (RAM), or may also include a Read-Only Memory. Optionally, the memory includes a non-transitory computer-readable storage medium. The memory is used to store instructions, programs, codes, code sets, or instruction sets. The memory may include a program storage area and a data storage area. Among them, the program storage area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the above method embodiments, etc.; the data storage area may store data created according to the use of the server, etc.

[0129] The processor may include one or more processing cores. The processor connects various parts within the entire server using various interfaces and lines, and by running or executing instructions, programs, code sets, or instruction sets stored in the memory, and by calling data stored in the memory, it executes various functions of the server and processes data. Optionally, the processor may be implemented in at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor may integrate a Central Processing Unit (CPU) and a modem, etc. in one or several combinations. Among them, the CPU mainly processes the operating system and application programs, etc.; the modem is used to process wireless communication. It can be understood that the above modem may not be integrated into the processor and may be implemented separately by a single chip.

[0130] Since this electronic device is the electronic device corresponding to a method for embodied intelligent task planning based on environmental perception in an embodiment of the present invention, and the principle of how this electronic device solves problems is similar to that of this method, the implementation of this electronic device can refer to the implementation process of the above method embodiment, and the repeated parts will not be elaborated.

[0131] Embodiment 4

[0132] An embodiment of the present invention further provides a computer-readable storage medium, in which at least one instruction, at least one program, a code set or an instruction set is stored, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement a Figure 1 method for embodied intelligent task planning based on environmental perception as shown.

[0133] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing relevant hardware through a program, and this program can be stored in a computer-readable storage medium. The storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electrically-erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc memories, magnetic disk memories, tape memories, or any other computer-readable medium capable of carrying or storing data.

[0134] Since this storage medium is the storage medium corresponding to the method for embodied intelligent task planning based on environmental perception in the embodiment of the present invention, and the principle of solving problems by this storage medium is similar to that of this method, the implementation of this storage medium can refer to the implementation process of the above method embodiment, and the repeated parts will not be described again.

[0135] Embodiment 5

[0136] In some possible embodiments, various aspects of the method of the embodiments of the present invention can also be implemented in the form of a program product, which includes program code. When the program product runs on a computer device, the program code is used to cause the computer device to execute the steps of a method for embodied intelligent task planning based on environmental perception according to various exemplary embodiments described above in this specification. Among them, the executable computer program code or "code" for executing each embodiment can be written in a high-level programming language such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, structured query language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0137] It should be understood that each part of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits with logic gate circuits for implementing logical functions on data signals, application-specific integrated circuits with appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0138] In the description of this specification, the description referring to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0139] The above embodiments are only for illustrating the technical concept and characteristics of the present invention, and their purpose is to enable those of ordinary skill in the art to understand the content of the present invention and implement it accordingly, and should not be used to limit the protection scope of the present invention. Any equivalent changes or modifications made according to the essence of the content of the present invention should be covered within the protection scope of the present invention.

Claims

1. An embodied intelligence task planning method based on environmental perception, characterized in that, Including the following steps: Fine-tune the large language model by integrating abstract environmental information into the large language model to enhance the environmental perception ability of the large language model; Input the task objective and the environmental state table into the fine-tuned large language model, and generate the action sequence with the highest probability for multiple steps through a step-by-step beam search strategy; According to the generated action sequence, update the environmental data state table by simulating the interaction between the robot agent and other objects, so as to generate the interaction trajectory corresponding to the steps.

2. The embodied intelligent task planning method based on environmental perception according to claim 1, wherein The environmental information includes object attributes, object positions, and object relationships; The expression for fine-tuning the large language model is: W′ = W + ΔW ΔW = LoRA(E, W) Where, W is the original weight of the large language model, ΔW is the weight adjustment obtained through the LoRA fine-tuning method, and W′ is the weight of the fine-tuned large language model; The fine-tuned large language model can generate an effective navigation path according to the spatial layout, or perform corresponding operations according to the object name and position.

3. The embodied intelligent task planning method based on environmental perception according to claim 1, characterized in that, The generating of the action sequence with the highest probability for multiple steps through a step-by-step beam search strategy includes: Determine the beam search width; At each time step, use the beam search method to generate the n tokens with the highest probability as candidate steps; At each time step, select the step with the highest probability from the candidate steps and add it to the plan; Where, the value of the beam search width is adjusted according to the task complexity and computing resource limitations.

4. The embodied intelligent task planning method based on environmental perception according to claim 1, characterized in that The expression of the step-by-step beam search strategy is: where K is the beam width, i.e., the search width; t = 0,..., T, where T is the maximum number of steps in the task, and [t] represents the set of natural numbers from 0 to t; S represents all possible sequences of steps, S [t] ={S 1,[t] ,..., S K,[t]} represents the top K steps with the highest probabilities selected from all possible steps from 0 to t; θ(s k,t ) = log P(s k,k |s k,t-1 ,..., s k,1 ) where θ(s k,t ) represents the conditional probability of the output step s k,t for all previous output steps s k,t-1 ,..., s k,1 is the logarithm of the conditional probability.

5. A method for embodied intelligent task planning based on environmental perception according to claim 1, characterized in that The updating of the environmental data state table according to the generated action sequence by simulating the interaction between the robot agent and other objects, so as to generate the interaction trajectory corresponding to the steps, includes: Based on the generated action sequence, use the named entity recognition technology to obtain the keywords in each step; Match the nouns in the keywords with the objects in the environmental state table, and match the verbs in the keywords with the predefined action library to obtain the action-object pairs for each step; According to the obtained action-object pairs, simulate the agent interaction trajectory and update the environmental state table until all steps are simulated, and obtain the interaction trajectory including the environmental state table after each update.

6. The embodied intelligent task planning method based on environmental perception according to claim 1, wherein The embodied intelligence task planning method further includes the step of calculating the trajectory matching score: For each task, compare the environmental state table in the trajectory simulation with the expected interaction trajectory, and calculate the trajectory matching score of this task to evaluate the ability of the large language model in embodied task planning.

7. A method for embodied intelligent task planning based on environmental perception according to claim 6, characterized in that, The calculation formula of the trajectory matching score is: where S success represents the number of successful steps, and T total represents the total number of steps; w is the weight factor for the final state match F represents the result of the final state matching.

8. An embodied intelligence task planning device based on environmental perception, characterized in that, Including: A model fine-tuning module for fine-tuning the large language model by integrating abstract environmental information into the large language model to enhance the environmental perception ability of the large language model; A step generation module for inputting the task objective and the environmental state table into the fine-tuned large language model, and generating the action sequence with the highest probability for multiple steps through a step-by-step beam search strategy; the environmental state table is obtained through environmental perception; A trajectory generation module for updating the environmental data state table by simulating the interaction between the robot agent and other objects according to the generated action sequence, so as to generate the interaction trajectory corresponding to the steps.

9. An electronic device, characterized in that, The electronic device includes a processor and a memory. At least one instruction, at least one program, a code set, or an instruction set is stored in the memory, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, At least one instruction, at least one program, a code set, or an instruction set is stored in the storage medium, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the method according to any one of claims 1 to 7.

Images