A method and system for home environment task decomposition based on large-scale language models

Abstract

The application belongs to the technical field of artificial intelligence, and discloses a household environment task decomposition method and system based on a large-scale language model; wherein the method comprises the following steps: a large-scale language model acquires the name of a household environment task to be decomposed, outputs relevant object names, and obtains the mapping of the object names in the household environment based on a first translation language model; the large-scale language model acquires the name of the household environment task to be decomposed, the mapping of the object names in the household environment, and the name and decomposition result of the most similar task to the household environment task to be decomposed, outputs a preliminary result, and maps the actions of the preliminary result to the action space allowed by the household environment through a second translation language model to obtain a final decomposition result. The technical scheme of the application solves the technical problem that a large-scale language model lacks context information when generating operation steps, and can better cope with the diversity and complexity of tasks.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, and specifically relates to a method and system for decomposing home environment tasks based on a large-scale language model. Background Technology

[0002] In recent years, the rapid development of artificial intelligence technology has given rise to large-scale language models. After being deeply trained on massive amounts of diverse data, large-scale language models have the remarkable ability to formulate action strategies for various complex tasks.

[0003] Currently, the research focuses on using large-scale language models to translate users' high-level natural language commands into feasible operational steps in a home environment; for example, breaking down "bring water" into a series of specific actions such as "pick up the cup", "pour water into the cup", "walk to the table", and "place the cup on the table".

[0004] In existing technologies, large-scale language models can directly generate logically sound action sequences for simple tasks. However, the action sequences generated by large-scale language models are not always executable in the agent's current environment, and the generated steps are not always applicable to actual operations. This problem mainly stems from the fact that large-scale language models are trained only on large text corpora, lacking interaction with specific environments. The generated action sequences lack contextual information about the agent's current environment and actual capabilities. Therefore, a new task decomposition scheme is urgently needed. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for decomposing home environment tasks based on a large-scale language model, in order to solve one or more of the aforementioned technical problems. Specifically, this invention discloses a task decomposition scheme that integrates environmental information, solving the technical problem of large-scale language models lacking contextual information when generating operation steps, and thus better addressing the diversity and complexity of tasks.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] This invention provides a method for decomposing home environment tasks based on a large-scale language model, comprising the following steps:

[0008] Obtain the name of the home environment task to be decomposed; a large-scale language model obtains the name of the home environment task to be decomposed and outputs the names of objects related to the home environment task to be decomposed; a first translation language model obtains the names of objects related to the home environment task to be decomposed and outputs the mapping of the object names in the home environment;

[0009] A large-scale language model obtains the name of the home environment task to be decomposed, the mapping of the object name in the home environment, the name of the task most similar to the home environment task to be decomposed, and the decomposition result, and outputs the preliminary result of the home environment task to be decomposed; the second translation language model maps the actions of the preliminary result to the action space allowed by the home environment to obtain the final decomposition result of the home environment task to be decomposed.

[0010] A further improvement of the method of the present invention is that the home environment graph corresponding to the home environment task to be decomposed stores the object information in the environment in the form of nodes and edges.

[0011] Each node represents an object in the environment, including the object name, numeric identifier, coordinates, object boundaries, and state; each edge represents the spatial relationship between two objects.

[0012] A further improvement to the method of the present invention is that the step of the first translation language model obtaining the object names related to the task of decomposing the home environment and outputting the mapping of the object names in the home environment specifically includes:

[0013] The first translation language model is based on a home environment map and obtains object names. With the name of an object in the environment e Embedding is performed to obtain vectors. and f(o) e The semantic distance between the two is calculated using cosine similarity. Where f represents the embedding function. The dot product of the object name vectors. and ||f(o e )|| represent vectors respectively and f(o) e The length of the vector;

[0014] pass Calculate the mapping of object names in the home environment.

[0015] A further improvement of the method of the present invention is that the name and decomposition result of the task most similar to the task to be decomposed are obtained based on the current home environment example dataset;

[0016] The example dataset stores information on tasks that have been executed and whose completion accuracy is higher than a preset threshold; the task information includes the task name and the decomposed task sequence as a result of the decomposition.

[0017] A further improvement to the method of the present invention is that the step of mapping the actions of the preliminary results to the action space allowed by the home environment through the second translation language model, and obtaining the final decomposition result of the home environment task to be decomposed, includes:

[0018] Actions on the preliminary results using the second translation language model With the allowable action space a e Embedding is performed to obtain vectors. and f(a) e The semantic distance between the two is calculated using cosine similarity. Where f represents the embedding function. Represents the dot product of action vectors. and ||f(a e )|| represent vectors respectively and f(a) e The length of the vector;

[0019] pass The final decomposition result of the home environment task to be decomposed is obtained through calculation.

[0020] This invention provides a home environment task decomposition system based on a large-scale language model, comprising:

[0021] The environmental observation unit includes a large-scale language model and a first translation language model; wherein, the large-scale language model is used to obtain the name of the home environment task to be decomposed and output the object names related to the home environment task to be decomposed; the first translation language model is used to obtain the object names related to the home environment task to be decomposed and output the mapping of the object names in the home environment;

[0022] The reasoning and analysis unit includes a large-scale language model and a second translational language model. The large-scale language model is used to obtain the name of the home environment task to be decomposed, the mapping of the object names in the home environment, the name of the task most similar to the home environment task to be decomposed, the decomposition result, and output the preliminary result of the home environment task to be decomposed. The second translational language model is used to map the actions of the preliminary result to the action space allowed by the home environment to obtain the final decomposition result of the home environment task to be decomposed.

[0023] A further improvement of the system of the present invention is that the home environment graph corresponding to the home environment task to be decomposed stores the object information in the environment in the form of nodes and edges;

[0024] Each node represents an object in the environment, including the object name, numeric identifier, coordinates, object boundaries, and state; each edge represents the spatial relationship between two objects.

[0025] A further improvement to the system of the present invention is that, in the first translation language model, the step of obtaining the object names related to the task of decomposing the home environment and outputting the mapping of the object names in the home environment specifically includes:

[0026] The first translation language model is based on a home environment map and obtains object names. With the name of an object in the environment e Embedding is performed to obtain vectors. and f(o) e The semantic distance between the two is calculated using cosine similarity. Where f represents the embedding function. The dot product of the object name vectors. and ||f(o e )|| represent vectors respectively and f(o) e The length of the vector;

[0027] pass Calculate the mapping of object names in the home environment.

[0028] A further improvement of the system of the present invention is that the name and decomposition result of the task most similar to the task to be decomposed are obtained based on the current home environment example dataset;

[0029] The example dataset stores information on tasks that have been executed and whose completion accuracy is higher than a preset threshold; the task information includes the task name and the decomposed task sequence as a result of the decomposition.

[0030] A further improvement to the system of the present invention is that, in the second translation language model, the step of mapping the actions of the preliminary results to the action space allowed by the home environment to obtain the final decomposition result of the task to be decomposed in the home environment includes:

[0031] Actions on the preliminary results through the second translation language model With the allowable action space a e Embedding is performed to obtain vectors. and f(a) e The semantic distance between the two is calculated using cosine similarity. Where f represents the embedding function. Represents the dot product of action vectors. and ||f(a e )|| represent vectors respectively and f(a) e The length of the vector;

[0032] pass The final decomposition result of the home environment task to be decomposed is obtained through calculation.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] This invention provides a method for decomposing home environment tasks based on a large-scale language model. The method consists of two steps: environmental observation and model inference. In the environmental observation step, a large-scale language model and a first translation language model are used to extract and analyze information about the home environment. By observing the current scene, rich environmental information is obtained, leading to an accurate and comprehensive understanding of the environment and improving adaptability and execution efficiency for complex tasks. In the model inference step, a large-scale language model and a second translation language model are used, combining the environmental information obtained from the environmental observation step with a pre-defined action space, to obtain a task decomposition result with higher accuracy and better suitability for practical operation. In summary, this invention utilizes the rich prior knowledge of a pre-trained large-scale language model to generate detailed guidance about the task, providing highly intelligent decision support for the system and enabling it to better cope with the diversity and complexity of tasks. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art are briefly introduced below; obviously, the drawings described below are some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0036] Figure 1 This is a flowchart illustrating the home environment task decomposition method based on a large-scale language model in an embodiment of the present invention.

[0037] Figure 2 This is a schematic diagram illustrating the principle of the task decomposition method based on a large-scale language model in an embodiment of the present invention.

[0038] Figure 3 This is a flowchart illustrating the home environment task decomposition method based on a large-scale language model in a specific embodiment of the present invention.

[0039] Figure 4 This is a schematic diagram of a home environment task decomposition system based on a large-scale language model in an embodiment of the present invention. Detailed Implementation

[0040] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0041] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0042] The present invention will now be described in further detail with reference to the accompanying drawings:

[0043] Please see Figure 1 The present invention provides a method for decomposing home environment tasks based on a large-scale language model, comprising the following steps:

[0044] Step 1: Obtain the name of the task to be decomposed in the current home environment; The large-scale language model obtains the name of the task to be decomposed and outputs the names of objects related to the task to be decomposed; The first translation language model obtains the names of objects and outputs the mapping of the object names in the current home environment.

[0045] Step 2: The large-scale language model obtains the mapping of the name of the task to be decomposed, the object name in the current home environment, and the name and decomposition result of the task most similar to the task to be decomposed, and outputs the preliminary result of the task to be decomposed; the second translation language model maps the actions of the preliminary result to the action space allowed by the current home environment to obtain the final decomposition result; interpretably, the name and decomposition result of the task most similar to the task to be decomposed are obtained based on the example dataset of the current home environment.

[0046] While large-scale language models excel at planning high-level tasks, their lack of accurate understanding of the agent's actual actions means that the steps they generate are not always applicable to real-world operations. The method provided in this invention, by incorporating an environmental observation step, observes the current home scene, acquiring rich environmental information and thus obtaining an accurate and comprehensive understanding of the environment. Furthermore, the model inference step incorporates the action space allowed by the environment. Therefore, the task decomposition results generated by the technical solution of this invention combine the specific home environment with the agent's actual capabilities, resulting in more accurate steps that are more suitable for practical operations.

[0047] In this embodiment of the invention, the environmental diagram of the current home environment is known; wherein,

[0048] The environment graph stores object information within the current environment in the form of nodes and edges; each node represents an object in the environment, including the object name, numeric identifier ID, coordinates, object boundaries, and their state; each edge represents the spatial relationship between two objects, such as INSIDE, ON, etc.

[0049] In a further preferred embodiment of the present invention, the object name output by the large-scale language model is mapped to the current environment graph through an untrained language model; the untrained language model mentioned above specifically uses a smaller-scale language model, which is referred to as the first translation language model in the present invention.

[0050] In the first translation language model, based on the environment graph, the input object names are... With the name of an object in the environment e Embedding is performed to obtain vectors. and f(o) e The semantic distance between the two is calculated using cosine similarity. Where f represents the embedding function. The dot product of the object name vectors. and ||f(o e )|| represent their vector lengths, and this distance reflects the degree of similarity between the objects;

[0051] The mapping of the object in the current environment graph can then be achieved through... We obtain o at this point. e This refers to the mapping of object names output by a large-scale language model to the current environment graph.

[0052] In a preferred embodiment of the present invention, the system observes the home environment scene where the task is located through a first translation language model to obtain rich environmental information, thereby gaining an accurate and comprehensive understanding of the environment and improving the system's adaptability and execution efficiency for complex tasks.

[0053] As further explained in the embodiments of the present invention, the example dataset described above stores information on tasks that have been executed and have a high success rate in completing the tasks, including task names and decomposed task sequences.

[0054] In a further preferred embodiment of the present invention, the preliminary task decomposition results obtained by the large-scale language model are used to obtain a mapping in the action space allowed by the environment through an untrained language model. Specifically, the present invention uses a second translation language model.

[0055] Among them, actions on the preliminary results With the allowable action space a e Embedding yields vectors and f(a) e The semantic distance between the two is calculated using cosine similarity. Where f represents the embedding function. Represents the dot product of action vectors. and ||f(a e )|| represent their vector lengths, and this distance reflects the degree of similarity between actions;

[0056] The mapping of this action in the current action space can be achieved through... We obtain a at this point. e This refers to the mapping of the actions decomposed into tasks onto the current environment graph.

[0057] In a preferred embodiment of the present invention, the rich prior knowledge of a pre-trained large-scale language model is used to generate detailed guidance on the task, providing highly intelligent decision support for the system. This enables the system to better cope with the diversity and complexity of tasks. By combining environmental information with the current action space through a second translation language model, a task decomposition result with higher accuracy and more suitable for actual operation is obtained.

[0058] In summary, this invention utilizes the capabilities of open-source large-scale language models to specifically provide a method for decomposing home environment tasks based on large-scale language models. This method consists of two parts: an environment observation part and a model inference part. In the environment observation part, a large-scale language model and an untrained language model, referred to as the first translation language model, are deployed. This is mainly used for information extraction and analysis of the home environment. By observing the current scene, rich environmental information is obtained, thereby achieving an accurate and comprehensive understanding of the environment and improving the system's adaptability and execution efficiency for complex tasks. In the model inference part, a large-scale language model and a second translation language model are deployed to combine the environmental information obtained from the environment observation part with a preset action space to obtain a task decomposition result with higher accuracy and better suitability for practical operation. This invention utilizes the rich prior knowledge of a pre-trained large-scale language model to generate detailed guidance on the task, providing highly intelligent decision support for the system and enabling it to better cope with the diversity and complexity of tasks. The principle explanation is as follows: Figure 2 As shown.

[0059] Please see Figure 3 Specifically, this invention provides a method for decomposing home environment tasks based on a large-scale language model, comprising the following steps:

[0060] Step 1: Obtain the tasks to be decomposed from the user's client;

[0061] Step 2: Combine the prompts for querying object names with the input of a large-scale language model to obtain the object names related to the task to be decomposed;

[0062] Step 3: Obtain the mapping of the object name output by the large model in the current environment. The object name output by the large model is mapped to the current environment graph through an untrained language model.

[0063] Step 4: Select the task most similar to the task to be decomposed in the current example dataset;

[0064] Step 5: Combine the prompts for the task action plan with the input of the large-scale language model to obtain the decomposition results of the current task;

[0065] Step 6: Map the decomposed actions to the action space allowed by the environment.

[0066] The large-scale language model chosen was chatglm2, and the translation language model chosen was sentence-transformers_all-mpnet-base-v2. The dataset was selected from manually edited task action planning data provided by the VirtualHome environment; 160 tasks were used as the example set, and the remaining 100 were used as the test set.

[0067] Evaluation metrics include the number of steps, measuring the total number of steps required to generate the action plan; the execution rate, measuring whether the generated actions follow a logical order and meet environmental constraints, calculating the number of steps executed without failure of the action plan; the longest common subsequence (LCS), measuring the ratio of the length of the longest common subsequence between the generated action plan and the actual actions to the total length; and the correctness assessment, which is the percentage of state object attributes that match between the environment graph after executing the generated action plan and the corresponding manually written plan. For example, the comparison results between the method of this embodiment and existing methods are shown in Table 1.

[0068] Table 1. Comparison Results

[0069] Existing technical methods 5.574 16.396％ 8.795％ 33.312％ Method of the present invention 15.72 41.45％ 9.87％ 41.35％

[0070] The following are embodiments of the apparatus of the present invention, which can be used to execute embodiments of the method of the present invention. For details not disclosed in the apparatus embodiments, please refer to the embodiments of the method of the present invention.

[0071] Please see Figure 4 In another embodiment of the present invention, a home environment task decomposition system based on a large-scale language model is disclosed, comprising:

[0072] The environmental observation unit includes a large-scale language model and a first translation language model; wherein, the large-scale language model is used to obtain the name of the home environment task to be decomposed and output the object names related to the home environment task to be decomposed; the first translation language model is used to obtain the object names related to the home environment task to be decomposed and output the mapping of the object names in the home environment;

[0073] The reasoning and analysis unit includes a large-scale language model and a second translational language model. The large-scale language model is used to obtain the name of the home environment task to be decomposed, the mapping of the object names in the home environment, the name of the task most similar to the home environment task to be decomposed, the decomposition result, and output the preliminary result of the home environment task to be decomposed. The second translational language model is used to map the actions of the preliminary result to the action space allowed by the home environment to obtain the final decomposition result of the home environment task to be decomposed.

[0074] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0075] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0076] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0077] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for decomposing home environment tasks based on a large-scale language model, characterized in that, Includes the following steps: Obtain the name of the home environment task to be decomposed; a large-scale language model obtains the name of the home environment task to be decomposed and outputs the names of objects related to the home environment task to be decomposed; a first translation language model obtains the names of objects related to the home environment task to be decomposed and outputs the mapping of the object names in the home environment; A large-scale language model obtains the name of the home environment task to be decomposed, the mapping of the object name in the home environment, the name of the task most similar to the home environment task to be decomposed, and the decomposition result, and outputs the preliminary result of the home environment task to be decomposed; the second translation language model maps the actions of the preliminary result to the action space allowed by the home environment to obtain the final decomposition result of the home environment task to be decomposed. in, The home environment graph corresponding to the home environment task to be decomposed stores object information in the environment in the form of nodes and edges; each node represents an object in the environment, including object name, numeric identifier, coordinates, object boundary and state; each edge represents the spatial relationship between two objects. The first translation language model obtains the object names related to the task of decomposing the home environment, and outputs the mapping of the object names in the home environment. Specifically, this includes: The first translation language model is based on a home environment map and obtains object names. Names of objects in the environment Embedding is performed to obtain vectors. and The semantic distance between the two is calculated using cosine similarity. ; in, Indicates an embedded function. The dot product of the object name vectors. and Representing vectors respectively and The length of the vector; pass Calculate the mapping of object names in the home environment.

2. The home environment task decomposition method according to claim 1, characterized in that, The name and decomposition result of the task most similar to the task to be decomposed are obtained based on the current home environment example dataset; The example dataset stores information on tasks that have been executed and whose completion accuracy is higher than a preset threshold; the task information includes the task name and the decomposed task sequence as a result of the decomposition.

3. The home environment task decomposition method according to claim 1, characterized in that, The step of mapping the actions of the preliminary results to the action space allowed by the home environment through the second translation language model, and obtaining the final decomposition result of the home environment task to be decomposed, includes: Actions on the preliminary results through the second translation language model Space of movement allowed by the environment Embedding is performed to obtain vectors. and The semantic distance between the two is calculated using cosine similarity. ;in, Indicates an embedded function. Represents the dot product of action vectors. and Representing vectors respectively and The length of the vector; pass The final decomposition result of the home environment task to be decomposed is obtained through calculation.

4. A home environment task decomposition system based on a large-scale language model, characterized in that, include: The environmental observation unit includes a large-scale language model and a first translation language model; wherein, the large-scale language model is used to obtain the name of the home environment task to be decomposed and output the object names related to the home environment task to be decomposed; the first translation language model is used to obtain the object names related to the home environment task to be decomposed and output the mapping of the object names in the home environment; The reasoning and analysis unit includes a large-scale language model and a second translational language model. The large-scale language model is used to obtain the name of the home environment task to be decomposed, the mapping of the object names in the home environment, the name of the task most similar to the home environment task to be decomposed, the decomposition result, and output the preliminary result of the home environment task to be decomposed. The second translational language model is used to map the actions of the preliminary result to the action space allowed by the home environment to obtain the final decomposition result of the home environment task to be decomposed. in, The home environment graph corresponding to the home environment task to be decomposed stores object information in the environment in the form of nodes and edges; each node represents an object in the environment, including object name, numeric identifier, coordinates, object boundary and state; each edge represents the spatial relationship between two objects. In the first translation language model, the steps of obtaining the object names related to the task of decomposing the home environment and outputting the mapping of the object names in the home environment specifically include: The first translation language model is based on a home environment map and obtains object names. Names of objects in the environment Embedding is performed to obtain vectors. and The semantic distance between the two is calculated using cosine similarity. ; in, Indicates an embedded function. The dot product of the object name vectors. and Representing vectors respectively and The length of the vector; pass Calculate the mapping of object names in the home environment.

5. The home environment task decomposition system according to claim 4, characterized in that, The name and decomposition result of the task most similar to the task to be decomposed are obtained based on the current home environment example dataset; The example dataset stores information on tasks that have been executed and whose completion accuracy is higher than a preset threshold; the task information includes the task name and the decomposed task sequence as a result of the decomposition.

6. The home environment task decomposition system according to claim 4, characterized in that, In the second translation language model, the steps of mapping the actions of the preliminary results to the action space allowed by the home environment to obtain the final decomposition results of the home environment task to be decomposed include: Actions on the preliminary results through the second translation language model Space of movement allowed by the environment Embedding is performed to obtain vectors. and The semantic distance between the two is calculated using cosine similarity. ;in, Indicates an embedded function. Represents the dot product of action vectors. and Representing vectors respectively and The length of the vector; pass The final decomposition result of the home environment task to be decomposed is obtained through calculation.

Images