Task testing method and device, language model training method, medium and equipment

Abstract

The invention provides a personal intelligent task test method and device based on a personal test platform, a language model training method, a computer readable storage medium and electronic equipment. The task testing method comprises the following steps: loading a corresponding task environment with an own body according to an obtained task, and generating a task environment text with the own body; matching a corresponding task description template according to the obtained task, and generating a task description text for the task through the task description template; and according to the subject task environment text and the task description text, controlling the subject intelligence to execute the action until the subject intelligence completes the task. According to the task testing platform, m kinds of task environments can be deployed in a unified mode, personalized configuration can be carried out on the intelligent agents, and therefore the task testing method of the intelligent agents can meet task testing of the intelligent agents under various kinds of environments and various kinds of tasks and is good in universality. The method is simple and light in configuration, low in requirement for equipment and capable of supporting large-batch task testing on a testing platform with a body.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method for controlling the embodied intelligence to perform actions according to the embodied task environment text and the task description text until the embodied intelligence completes the task. More specifically, it relates to an embodied intelligence task testing method and device based on an embodied testing platform, and a language model training method, medium, and equipment. Background Art

[0002] With the rapid development of general artificial intelligence technology, it has been widely used in various fields, and the various capabilities of embodied intelligence are in urgent need of testing. It is worth exploring whether embodied intelligence can complete some tasks in an embodied environment, such as indoor navigation, command following, and collaboration between multiple agents. However, there is no system in the prior art that can test the ability of embodied intelligence to complete multiple tasks in multiple embodied environments. Moreover, the single embodied environment testing system in the prior art often requires a real embodied environment or a simulated real embodied environment to test embodied intelligence tasks. Simulating a real embodied environment usually requires complex configuration, has relatively high requirements for equipment, and does not support large-scale testing. Summary of the invention

[0003] In view of this, the technical problem to be solved by the present application is to provide an embodied intelligence task testing method and device based on an embodied testing platform, a language model training method, medium, and equipment, which solves the problems in the existing artificial intelligence field that there is no system that can test the ability of embodied intelligence to complete multiple tasks in multiple embodied environments, the simulation of real embodied environments requires complex configuration, has high requirements for equipment, and does not support large-scale testing.

[0004] In order to solve the above technical problems, the specific implementation of the present application provides an embodied intelligence task testing method based on an embodied testing platform, wherein the embodied testing platform is configured to have m embodied task environments and n task description templates, m is an integer greater than or equal to 1, and n is an integer greater than or equal to 1. The embodied intelligence task testing method includes: loading the corresponding embodied task environment according to the acquired task, and generating an embodied task environment text, wherein the embodied task environment includes a scene layout and an embodied intelligence configuration, and the embodied intelligence configuration includes at least the number of embodied intelligences and the initial position of each of the embodied intelligences; matching the corresponding task description template according to the acquired task, generating a task description text for the task through the task description template; and controlling the embodied intelligence to execute actions according to the embodied task environment text and the task description text until the embodied intelligence completes the task.

[0005] According to the embodied intelligence task testing method based on the embodied test platform of the embodiment of the present application, the corresponding embodied task environment is loaded according to the acquired task, and the embodied task environment text is generated; the corresponding task description template is matched according to the acquired task, and the task description text is generated from the task through the task description template; the embodied intelligence execution action can be controlled according to the embodied task environment text and the task description text until the embodied intelligence completes the task. The embodied test platform of the present application can uniformly deploy m kinds of embodied task environments, and can personalize the embodied intelligence. Therefore, the embodied intelligence task testing method of the present application can meet the embodied intelligence task testing under various types of environments and various types of tasks, and has good versatility. In addition, the present application controls the embodied intelligence execution action through the embodied task environment text and the task description text, and provides a text world without the need to fully simulate the real scene. Therefore, the configuration is simple and lightweight, and the requirements for the equipment are relatively low, which can support large-scale task testing on the embodied test platform.

[0006] In some examples, the step of controlling the embodied intelligence to perform actions according to the embodied task environment text and the task description text until the embodied intelligence completes the task includes: generating initial execution information of the embodied intelligence according to the task description text and the embodied task environment text; the embodied intelligence performing actions according to the execution information, updating the position and scene layout of the embodied intelligence, and obtaining updated position text and updated scene text, wherein when the embodied intelligence performs an action for the first time in the loaded embodied task environment, the execution information based on the initial execution information is used; when it is evaluated that the updated position text and the updated scene text satisfy the conditions for completing the task, the embodied intelligence completes the task; and when it is evaluated that the updated position text and the updated scene text do not satisfy the conditions for completing the task, generating execution information of the embodied intelligence according to the task description text, the updated position text and the updated scene text, continuing the step of the embodied intelligence performing actions according to the execution information, updating the position and scene layout of the embodied intelligence, and obtaining updated position text and updated scene text.

[0007] In some examples, the steps of generating initial execution information of the embodied intelligence based on the task description text and the embodied task environment text, and the steps of generating execution information of the embodied intelligence based on the task description text, the updated location text and the updated scene text are both performed by a language model or an expert model.

[0008] In some examples, the language model has model parameters, which include a mapping relationship between model input and model output, the model input includes task description text, embodied intelligence location text and scene layout text, and the model output includes execution information of the embodied intelligence.

[0009] In some examples, the execution steps of the expert model include: determining an object information list of objects in the scene layout that need to be operated to complete the task based on the task description text, the location text of the embodied intelligence, and the scene layout text; sorting the objects in the object information list according to the shortest path algorithm based on the location text of the embodied intelligence and the coordinates of the objects in the object information list; and generating execution information of the embodied intelligence in the order in which the objects are sorted, wherein the execution information includes a movement path and an operation action.

[0010] In some examples, the embodied testing platform is configured to have a user operation interface, and the embodied intelligence task testing method also includes: after loading the corresponding embodied task environment according to the acquired task, it can respond to the user's operation on the embodied intelligence in the scene layout, control the embodied intelligence to perform actions until the embodied intelligence completes the task, and when the user operates the user operation interface, stop executing the steps of matching the corresponding task description template according to the acquired task, and generating a task description text for the task through the task description template; and control the embodied intelligence to perform actions according to the embodied task environment text and the task description text until the embodied intelligence completes the task.

[0011] In some examples, after controlling the embodied intelligence to perform actions according to the embodied task environment text and the task description text until the embodied intelligence completes the task, the embodied intelligence task testing method further includes: evaluating the degree of completion of the task by the embodied intelligence; and scoring according to the degree of completion to obtain a task score.

[0012] In some examples, the embodied testing platform is configured to be able to modify, add and / or delete the embodied task environment in real time; the embodied testing platform is configured to be able to modify, add and / or delete the task description template in real time.

[0013] Another aspect of an embodiment of the present application provides a method for training a language model, comprising: obtaining training data, wherein the training data is obtained from the execution step of an expert model in the embodied intelligence task testing method based on an embodied testing platform according to claim 5, and the training data includes input samples and labeled samples, the input samples are task description text, embodied intelligence location text and scene layout text in the execution step of the expert model, and the labeled samples are execution information in the execution step of the expert model; training initial model parameters of the language model according to the training data to obtain training model parameters, the training model parameters including a mapping relationship between task description text, location and scene layout of embodied intelligence and execution information; and applying the training model parameters as model parameters of the language model.

[0014] According to the training method of the language model of the embodiment of the present application, the expert model can be used to obtain automatically converted or annotated training data for language model learning, and the language model can be trained using the training data provided by the expert model, thereby obtaining a language model with a stronger ability to solve embodied intelligence tasks. The training method of the language model of the present application does not require manual annotation, saving annotation costs.

[0015] Another aspect of an embodiment of the present application provides an embodied intelligence task testing device based on an embodied testing platform, wherein the embodied testing platform is configured to have m embodied task environments and n task description templates, where m is an integer greater than or equal to 1, and n is an integer greater than or equal to 1. The embodied intelligence task testing device includes: a loading module, wherein the loading module is used to execute the loading of the corresponding embodied task environment according to the acquired task, and generate an embodied task environment text, wherein the embodied task environment includes a scene layout and an embodied intelligence configuration, and the embodied intelligence configuration includes at least the number of embodied intelligences and the initial position of each of the embodied intelligences; a generation module, wherein the generation module is used to execute the matching of the corresponding task description template according to the acquired task, and generate a task description text for the task through the task description template; and a control module, wherein the control module is used to control the embodied intelligence to perform an action according to the embodied task environment text and the task description text until the embodied intelligence completes the task.

[0016] Another aspect of an embodiment of the present application provides an electronic device, including one or more processors and a storage device, wherein the storage device is used to store executable instructions, and when the executable instructions are executed by the processor, the method of the embodiment of the present application is implemented.

[0017] Another aspect of an embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the method of the embodiment of the present application when executed by a processor.

[0018] Another aspect of an embodiment of the present application provides a computer program, wherein the computer program includes computer executable instructions, and the instructions are used to implement the method of the embodiment of the present application when executed.

[0019] It should be understood that the above general description and the following specific embodiments are only exemplary and illustrative and do not limit the scope of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The attached drawings below are part of the specification of the present application, which illustrate exemplary embodiments of the present application. The attached drawings together with the description of the specification are used to explain the principles of the present application.

[0021] Figure 1 The flowchart of the embodied intelligent task testing method based on the embodied testing platform according to an embodiment of the present application is schematically shown.

[0022] Figure 2 A flowchart is schematically shown for controlling the embodied intelligence to perform actions according to the embodied task environment and the task description text until the embodied intelligence completes the task according to an embodiment of the present application.

[0023] Figure 3 The flowchart schematically shows the execution steps of the expert model according to the embodiment of the present application.

[0024] Figure 4 The flowchart of the embodied intelligent task testing method based on the embodied testing platform according to an embodiment of the present application is schematically shown.

[0025] Figure 5 The flowchart of the language model training method according to the embodiment of the present application is schematically shown.

[0026] Figure 6 The structural block diagram of an embodied intelligent task testing device based on an embodied testing platform according to an embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of the embodiments of the present application more clearly understood, the spirit of the contents disclosed in the present application will be clearly explained with the accompanying drawings and detailed descriptions below. After understanding the embodiments of the contents of the present application, any technician in the relevant technical field can change and modify them according to the techniques taught by the contents of the present application without departing from the spirit and scope of the contents of the present application.

[0028] The exemplary embodiments and descriptions of the present application are used to explain the present application, but are not intended to limit the present application. In addition, elements / components with the same or similar reference numerals used in the drawings and embodiments are used to represent the same or similar parts.

[0029] The terms “first”, “second”, etc. used in this document do not particularly refer to an order or sequence, nor are they used to limit this application. They are only used to distinguish elements or operations described with the same technical terms.

[0030] The directional terms used in this document, such as up, down, left, right, front or back, etc., are only used to refer to the directions of the drawings. Therefore, the directional terms used are used to illustrate and not to limit the present invention.

[0031] The words “include,” “including,” “have,” “contain,” etc. used in this document are open-ended terms, meaning including but not limited to.

[0032] As used herein, "and / or" includes any or all combinations of the items described.

[0033] As used herein, “plurality” includes “two” and “more than two”; as used herein, “plurality groups” includes “two groups” and “more than two groups”.

[0034] The terms "substantially" and "approximately" used herein are used to modify any quantity or error that may vary slightly, but these slight changes or errors do not change their essence. Generally speaking, the range of slight changes or errors modified by such terms may be 20% in some embodiments, 10% in some embodiments, 5% in some embodiments, or other values. Those skilled in the art should understand that the aforementioned values ​​can be adjusted according to actual needs and are not limited thereto.

[0035] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0036] In the case of using expressions such as "at least one of A, B, and C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.). In the case of using expressions such as "at least one of A, B, or C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, or C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0037] Those skilled in the art will also appreciate that any transitional conjunction and / or phrase that represents two or more optional items, whether in the specification, claims or drawings, should be understood to include the possibility of one of these items, either of these items, or both items. For example, the phrase "A or B" should be understood to include the possibility of "A" or "B", or "A and B".

[0038] With the rapid development of general artificial intelligence technology, it has been widely used in various fields, and the various capabilities of embodied intelligence are in urgent need of testing. It is worth exploring whether embodied intelligence can complete some tasks in an embodied environment, such as indoor navigation, command following, and collaboration between multiple agents. However, there is no system in the prior art that can test the ability of embodied intelligence to complete multiple tasks in multiple embodied environments. Moreover, the single embodied environment testing system in the prior art often requires a real embodied environment or a simulated real embodied environment to test embodied intelligence tasks. Simulating a real embodied environment usually requires complex configuration, has relatively high requirements for equipment, and does not support large-scale testing.

[0039] Reference below Figure 1-Figure 6 The present invention describes an embodied intelligent task testing method and device based on an embodied testing platform, a language model training method, an electronic device, a computer-readable storage medium, and a computer program product according to an embodiment of the present application.

[0040] Figure 1 The flowchart of the embodied intelligence task testing method based on the embodied test platform according to the embodiment of the present application is schematically shown. Among them, the embodied test platform is configured to have m embodied task environments and n task description templates, m is an integer greater than or equal to 1, and n is an integer greater than or equal to 1. Here, the embodied task environment can be a virtual embodied environment. Embodied intelligence may include but is not limited to: intelligent embodiment, embodied agent, embodied intelligent body, embodied virtual body, agent robot, etc.

[0041] like Figure 1 As shown, the embodied intelligent task testing method based on the embodied testing platform of this embodiment includes operations S110 to S130.

[0042] In operation S110, the corresponding embodied task environment is loaded according to the acquired task, and an embodied task environment text is generated, wherein the embodied task environment includes a scene layout and an embodied intelligence configuration, and the embodied intelligence configuration includes at least the number of embodied intelligences and the initial position of each embodied intelligence.

[0043] In some examples, the task may be a natural language text, and by matching the keywords of the task with the keywords of the embodied task environment, it is possible to load the corresponding embodied task environment according to the acquired task.

[0044] In some examples, the embodied task environment may be a three-dimensional rendered environment.

[0045] In some examples, since the virtual three-dimensional scene needs to load a three-dimensional engine and render the scene, in some embodiments, the embodied task environment can be a two-dimensional environment, retaining only the necessary object attributes, and does not require real-time rendering, and can quickly get feedback. In this task environment, the actions of embodied intelligence are symbolic actions, that is, changing the symbolic attributes of the environment, and there will be no physical changes, so it is relatively simple and light, and can be used for large-scale testing. The task environment can be set as a multi-room home scene, where room types include living rooms, kitchens, bedrooms, and toilets. Walls, as physical divisions of the room, will affect the line of sight and running trajectory of the embodied intelligence. Objects include ordinary objects and container objects. Container objects can hold ordinary objects, and embodied intelligence cannot see objects in closed containers.

[0046] In some examples, the embodied intelligence configuration includes at least the number of embodied intelligences and the initial position of each embodied intelligence. In addition, it may also include whether the embodied intelligence can interact with humans and whether the embodied intelligences can interact with each other, etc.

[0047] In some examples, the embodied intelligence configuration can also configure the type of embodied intelligence, the initial position in the scene, the visual focal length, the maximum visual distance, the maximum operating distance, the step length, etc. The embodied intelligence configuration can also be configured to save historical dialogue information, historical action sequences, etc. during the completion of the entire task. For tasks with multiple embodied intelligences, multiple embodied intelligences can be loaded when the task is initialized.

[0048] In operation S120, a corresponding task description template is matched according to the acquired task, and a task description text is generated from the task through the task description template.

[0049] In some examples, the n types of task description templates include but are not limited to instruction-type task description templates, teaching task description templates, scene restoration task description templates, interactive question-and-answer task description templates, active help task description templates, and entry-level intelligent task description templates.

[0050] Among them, directive tasks can be understood as tasks that the embodied intelligence needs to complete according to instructions. For example, if an instruction is given to the embodied intelligence, the embodied intelligence needs to perform actions according to the instruction to complete the task. The instructions in directive tasks can be carried out in home scenes or in other scenes. For example, a directive task can be "put a can of cold Coke on the table."

[0051] Among them, the teaching task can be understood as a task of multiple embodied intelligences, one part of which is the commander, and the other part of which is the follower. The commander has access to all information about the task and can communicate with the follower using natural language. The follower navigates and interacts with the environment to complete tasks of varying complexity, such as from "making coffee" to "preparing breakfast", etc. The follower can ask questions and get more information from the commander.

[0052] Among them, the scene restoration task can be understood as a task that allows the embodied intelligence to set the scene to the user's desired state by moving objects, opening and closing objects, etc. First, the target scene is described to the embodied intelligence in natural language, and then several objects are randomly selected to change their positions or open and close states, thereby changing the scene. The embodied intelligence restores the scene to the original target scene by exploring and changing the states of objects in the scene.

[0053] Among them, the interactive question-answering task can be understood as providing the embodied intelligence with a specific scenario and a question, such as "Are there any apples in the refrigerator?" The embodied intelligence must navigate in the scenario, explore the scenario, interact with the objects in the scenario, such as opening the refrigerator, formulate a series of action plans based on the question, and finally answer the question.

[0054] Among them, the active help task can be understood as having two embodied intelligences, one of which needs to assist the other to complete a complex task together. In order to succeed, embodied intelligence 1 needs to observe and understand the goal of embodied intelligence 2, and complete this goal together with embodied intelligence 2. In this process, the two embodied intelligences can actively initiate communication.

[0055] Among them, elementary intelligent tasks can be understood as using synthetic natural language instructions to require embodied intelligence to navigate the environment and move objects to specified locations, such as "put the red ball next to the box on your left."

[0056] In some examples, the keywords of the task can be matched with the keywords of the task description template, so as to match the corresponding task description template according to the acquired task. The task description template can be used to automatically generate a task description text with a specific format.

[0057] In some examples, the task description text may include descriptive text such as a task definition, the types of actions allowed by the task, and task examples.

[0058] In some specific examples, the task description text may include the task setting instructions output by the embodied test platform, the task instructions that the embodied intelligence should currently complete, the environment observed by the embodied intelligence, the environment observed by the embodied intelligence, what the embodied intelligence has currently obtained, the actions and explanations that the embodied intelligence can perform, and some rules that the embodied intelligence should follow, etc. The above is only for illustration and should not be understood as limiting the present application.

[0059] In some examples, the action types allowed by the task may include but are not limited to forward: indicating that the embodied intelligence moves forward one step, and the step length is the initialization step length, for example, the default may be 0.25m; turn left: indicating that the embodied intelligence turns 90° to the left; turn right: indicating that the embodied intelligence turns 90° to the right; pick up: indicating that the embodied intelligence picks up a given object within the visible range and the operable range, and after picking up the object, the object will disappear from the original placement scene and appear in the hands of the embodied intelligence; place: indicating that the embodied intelligence puts the given object in the hand into another given container object, and a new object will be added to the scene after the operation; put down: indicating that the embodied intelligence puts the given object in the hand in front of its own eyes, and a new object will be added to the scene after the operation; open: indicating that the embodied intelligence will open a given openable object within the visible range and the operable range, and open it. After opening, the embodied intelligence will be able to see the objects in the container; closing: means that the embodied intelligence closes a given openable object within the visible range and the operable range, and the objects in the container will not be visible after the operation; turning on the switch: means that the embodied intelligence turns on the switch of a given object within the visible range and the operable range, and the appliance will run after the operation; turning off the switch: means that the embodied intelligence turns off the switch of a given object within the visible range and the operable range, and the given object will stop running after the operation; slicing: means that the embodied intelligence slices a given object within the visible range and the operable range, and some sliced ​​objects will be added to the environment after slicing; walking to: means that the embodied intelligence walks to the specified object position; chatting: means that the embodied intelligence can initiate a conversation; empty: means an empty action, and the embodied intelligence does not perform any operation.

[0060] In operation S130, the embodied intelligence is controlled to perform actions according to the embodied task environment text and the task description text until the embodied intelligence completes the task.

[0061] According to the embodied intelligence task testing method based on the embodied test platform of the embodiment of the present application, the corresponding embodied task environment is loaded according to the acquired task, and the embodied task environment text is generated; the corresponding task description template is matched according to the acquired task, and the task description text is generated from the task through the task description template; the embodied intelligence execution action can be controlled according to the embodied task environment text and the task description text until the embodied intelligence completes the task. The embodied test platform of the present application can uniformly deploy m kinds of embodied task environments, and can personalize the embodied intelligence. Therefore, the embodied intelligence task testing method of the present application can meet the embodied intelligence task testing under various types of environments and various types of tasks, and has good versatility. In addition, the present application controls the embodied intelligence execution action through the embodied task environment text and the task description text, and provides a text world without the need to fully simulate the real scene. Therefore, the configuration is simple and lightweight, and the requirements for the equipment are relatively low, which can support large-scale task testing on the embodied test platform.

[0062] In some examples, the embodied intelligence can actively provide feedback on the execution of each action after it is completed. If the action is successfully executed, the feedback will include information about the successful execution of the action and the scene layout after the action is executed, including all objects within the visual range of the embodied intelligence, objects in the hands of the embodied intelligence, etc. If the action fails, the feedback will include information about the failure of the action and a natural language description of the reason for the failure.

[0063] Figure 2 A flowchart is schematically shown for controlling embodied intelligence to perform actions according to embodied task environment text and task description text until the embodied intelligence completes the task according to an embodiment of the present application.

[0064] like Figure 2 As shown, operation S130 controls the embodied intelligence to perform actions according to the embodied task environment text and the task description text until the embodied intelligence completes the task, including operations S131 to S134.

[0065] In operation S131, initial execution information of embodied intelligence is generated according to the task description text and the embodied task environment text.

[0066] In operation S132, the embodied intelligence executes an action according to the execution information, updates the position and scene layout of the embodied intelligence, and obtains an updated position text and an updated scene text, wherein when the embodied intelligence executes an action for the first time in the loaded embodied task environment, the execution information used is the initial execution information.

[0067] In some examples, the execution information may include a movement path and an operation action, and the initial execution information may include an initial movement path and an initial operation action. When the embodied intelligence performs an action for the first time in the loaded embodied task environment, the execution information based on it is the initial execution information; after the first execution of the action, the execution information based on it is the execution information obtained in operation S134.

[0068] In operation S133, when it is evaluated that the updated position text and the updated scene text meet the conditions for completing the task, the embodied intelligence completes the task.

[0069] In operation S134, when it is evaluated that the updated position text and the updated scene text do not meet the conditions for completing the task, the execution information of the embodied intelligence is generated according to the task description text, the updated position text and the updated scene text, and the embodied intelligence continues to execute actions according to the execution information, and the position and scene layout of the embodied intelligence are updated to obtain the steps of updating the position text and updating the scene text.

[0070] Through operations S131 to S134, it is easy to control the embodied intelligence to perform actions according to the embodied task environment text and the task description text until the embodied intelligence completes the task. At the same time, when the evaluation update position text and the update scene text do not meet the conditions for completing the task, the execution information of the embodied intelligence will be generated according to the task description text, the update position text and the update scene text, that is, each decision in the task execution process will receive feedback from the environment, and a new decision will be generated based on the feedback, thereby better testing the action history memory and internal world modeling capabilities of the language model or expert model, and improving the planning accuracy of the language model or expert model.

[0071] According to some embodiments of the present application, the step of generating the initial execution information of the embodied intelligence according to the task description text and the embodied task environment text in operation S131, and the step of generating the execution information of the embodied intelligence according to the task description text, the updated position text and the updated scene text in operation S134 are both performed by a language model or an expert model. Thus, the language model or the expert model can be used to make decisions on the process of executing the task of the embodied intelligence, that is, to generate execution information, thereby controlling the embodied intelligence to perform actions according to the execution information.

[0072] According to some embodiments of the present application, the language model has model parameters, the model parameters include a mapping relationship between model input and model output, the model input includes task description text, embodied intelligence location text and scene layout text, and the model output includes execution information of embodied intelligence.

[0073] In some examples, operation S131 may be performed by a language model to generate initial execution information of embodied intelligence based on the task description text and the embodied task environment text, wherein the position text and scene layout text of the embodied intelligence included in the task description text and the embodied task environment text are used as inputs of the language model, and the initial execution information of the embodied intelligence may be obtained using model parameters.

[0074] In some examples, the generation of the execution information of the embodied intelligence according to the task description text, the update position text and the update scene text in operation S134 can be performed by a language model. The task description text, the update position text and the update scene text of the embodied intelligence are used as inputs of the language model, and the execution information of the embodied intelligence can be obtained by using the model parameters.

[0075] Therefore, the language model can be used to make decisions on the process of embodied intelligence executing tasks, that is, to generate execution information, thereby controlling the embodied intelligence to perform actions according to the execution information.

[0076] According to some embodiments of the present application, Figure 3 As shown, the execution steps of the expert model include operations S210 to S230.

[0077] In operation S210, a list of object information of objects in the scene layout that need to be operated to complete the task is determined based on the task description text, the location text of the embodied intelligence, and the scene layout text.

[0078] In operation S220, objects in the object information list are sorted according to the location text of the embodied intelligence and the coordinates of the objects in the object information list using a shortest path algorithm.

[0079] In operation S230, execution information of embodied intelligence is generated according to the order in which the objects are sorted, wherein the execution information includes a moving path and an operation action.

[0080] Through operations S210 to S230, it is easy to implement operation S131 through an expert model to generate initial execution information of embodied intelligence according to task description text and embodied task environment text; through operations S210 to S230, it is easy to implement operation S134 through an expert model to generate execution information of embodied intelligence according to task description text, updated location text and updated scene text.

[0081] According to some embodiments of the present application, the embodied testing platform is configured to have a user operation interface, and the embodied intelligence task testing method also includes: after loading the corresponding embodied task environment according to the acquired task, it can respond to the user's operation on the embodied intelligence in the scene layout, control the embodied intelligence to perform actions until the embodied intelligence completes the task, and when the user operates the user operation interface, stop executing the steps of matching the corresponding task description template according to the acquired task, and generating a task description text for the task through the task description template; and control the embodied intelligence to perform actions according to the embodied task environment and the task description text until the embodied intelligence completes the task.

[0082] It is understandable that the user can drag the embodied intelligence to make corresponding actions in the user operation interface. The user operation interface can reflect the position and scene status of the embodied intelligence in real time according to the action plan made by the user. Therefore, through the user operation interface, the user can independently control the embodied intelligence to perform actions, thereby improving the flexibility of testing the embodied intelligence.

[0083] Figure 4 The flowchart of the embodied intelligent task testing method based on the embodied testing platform according to an embodiment of the present application is schematically shown.

[0084] like Figure 4 As shown, after operation S130 controls the embodied intelligence to perform actions according to the embodied task environment and the task description text until the embodied intelligence completes the task, the embodied intelligence task testing method based on the embodied testing platform further includes operations S140 and S150.

[0085] In operation S140 , the degree of completion of the task by the embodied intelligence is evaluated.

[0086] In some examples, the completion of tasks by embodied intelligence can be reflected by evaluation indicators. Different evaluation indicators can be set according to different tasks according to actual needs. The completion of the same task can be obtained by weighted summing up all evaluation indicators under the task. Of course, there are other ways to achieve completion, and here we only use weighted summing as an example.

[0087] In some examples, for instruction-type tasks, there can be two evaluation indicators: the first is the task success rate, that is, if all object states (such as position, opening and closing, etc.) are consistent with expectations, it is successful, otherwise it is a failure; the second is the conditional success rate, that is, the proportion of successful partial object states to all expected object states.

[0088] In some examples, for scene restoration tasks, there can be three evaluation indicators: the first is the task success rate, that is, if all object states (such as position, opening and closing, etc.) are consistent with expectations, it is successful, otherwise it is a failure; the second is the error placement rate, that is, the ratio of the number of object state errors after all operations to the number of initial object state errors. This indicator is used to measure the incorrect operations in the task completion process; the third is the correct placement rate, that is, the ratio of the number of correct object states after all operations to the number of initial object state errors. This indicator is used to measure the completion status of the task completion process.

[0089] In some examples, for entry-level intelligent tasks, the evaluation indicator is the task success rate. When the embodied intelligence reaches the specified location and the status of all objects meets expectations, the task is successful, otherwise it fails.

[0090] In some examples, for interactive question-answering tasks, the evaluation indicator is accuracy, that is, when the given answer is consistent with the preset answer, it is judged to be successful.

[0091] In operation S150, a score is given according to the degree of completion to obtain a task score. It is understandable that there may be a mapping relationship between the degree of completion and the score, and the corresponding score may be matched to the degree of completion of the task for the embodied intelligence according to the mapping relationship between the degree of completion and the score. Operations S140 and S150 may facilitate scoring of the task completion of the embodied intelligence, and R&D personnel may evaluate the performance of the embodied intelligence, language model or expert model through the score, thereby improving the efficiency of task testing of the embodied intelligence.

[0092] According to some embodiments of the present application, the embodied test platform is configured to be able to modify, add and / or delete the embodied task environment in real time; the embodied test platform is configured to be able to modify, add and / or delete the task description template in real time. Thus, the embodied task environment can be modified, added and / or deleted according to actual needs; the task description template can be modified, added and / or deleted according to actual needs. Based on this, the embodied intelligence task testing method of the present application can meet the task testing of embodied intelligence in various environments and various tasks, and has good versatility and wide application.

[0093] In some examples, the embodied testing platform presets a general interface that can be expanded to more scenarios and tasks.

[0094] Figure 5 The flowchart of the language model training method according to the embodiment of the present application is schematically shown.

[0095] like Figure 5 As shown, the language model training method of this embodiment includes operations S310 to S330.

[0096] In operation S310, training data is obtained, wherein the training data is obtained according to the execution steps of the expert model in the embodied intelligence task testing method based on the embodied testing platform, and the training data includes input samples and labeled samples, the input samples are the task description text, the location text of the embodied intelligence and the scene layout text in the execution steps of the expert model, and the labeled samples are the execution information in the execution steps of the expert model.

[0097] In operation S320, initial model parameters of the language model are trained according to the training data to obtain training model parameters, where the training model parameters include a mapping relationship between the task description text, the location of the embodied intelligence, and the scene layout and execution information.

[0098] In operation S330, the training model parameters are applied as model parameters of the language model.

[0099] According to the training method of the language model of the embodiment of the present application, the expert model can be used to obtain automatically converted or annotated training data for language model learning, and the language model can be trained using the training data provided by the expert model, thereby obtaining a language model with a stronger ability to solve embodied intelligence tasks. The training method of the language model of the present application does not require manual annotation, saving annotation costs.

[0100] Figure 6 The structure block diagram of the embodied intelligent task testing device based on the embodied testing platform according to the embodiment of the present application is schematically shown. The embodied testing platform is configured to have m embodied task environments and n task description templates, where m is an integer greater than or equal to 1 and n is an integer greater than or equal to 1.

[0101] like Figure 6 As shown, the embodied intelligent task testing device 10 includes a loading module 1, a generating module 2 and a controlling module 3.

[0102] Loading module 1, loading module 1 is used to perform operation S110: load the corresponding embodied task environment according to the acquired task, and generate an embodied task environment text, wherein the embodied task environment includes a scene layout and an embodied intelligence configuration, and the embodied intelligence configuration includes at least the number of embodied intelligences and the initial position of each embodied intelligence.

[0103] Generating module 2, generating module 2 is used to perform operation S120: matching the corresponding task description template according to the acquired task, and generating a task description text from the task through the task description template.

[0104] Control module 3, control module 3 is used to perform operation S130: control the embodied intelligence to perform actions according to the embodied task environment text and the task description text until the embodied intelligence completes the task.

[0105] According to the embodied intelligence task testing device 10 based on the embodied test platform of the embodiment of the present application, by loading the corresponding embodied task environment according to the acquired task, and generating the embodied task environment text; matching the corresponding task description template according to the acquired task, generating the task description text from the task through the task description template; the embodied intelligence execution action can be controlled according to the embodied task environment text and the task description text, until the embodied intelligence completes the task. The embodied test platform of the present application can uniformly deploy m kinds of embodied task environments, and can personalize the embodied intelligence. Therefore, the embodied intelligence task testing method of the present application can meet the embodied intelligence task testing under various types of environments and various types of tasks, and has good versatility. In addition, the present application controls the embodied intelligence execution action through the embodied task environment text and the task description text, and provides a text world without the need to fully simulate the real scene. Therefore, the configuration is simple and light, and the requirements for the equipment are relatively low, which can support large-scale task testing on the embodied test platform.

[0106] In addition, according to an embodiment of the present application, any multiple modules among the loading module 1, the generating module 2 and the controlling module 3 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module.

[0107] According to an embodiment of the present application, at least one of the loading module 1, the generating module 2 and the controlling module 3 may be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or may be implemented by hardware or firmware such as any other reasonable method of integrating or packaging the circuit, or may be implemented in any one of the three implementation methods of software, hardware and firmware or in a suitable combination of any of them.

[0108] Alternatively, at least one of the loading module 1 , the generating module 2 and the controlling module 3 may be at least partially implemented as a computer program module, and when the computer program module is executed, the corresponding function may be executed.

[0109] The present application also provides an electronic device, including: one or more processors; one or more memories for storing executable instructions, which implement the above method when executed by the processor, and the electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0110] The device includes a computing unit that can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) or a computer program loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for the operation of the device can also be stored. The computing unit, ROM, and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0111] Multiple components in the device are connected to the I / O interface, including: input units, such as keyboards, mice, etc.; output units, such as various types of displays, speakers, etc.; storage units, such as disks, optical disks, etc.; and communication units, such as network cards, modems, wireless communication transceivers, etc. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunication networks.

[0112] The computing unit may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of computing units include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit performs the various methods and processes described above, such as image recognition methods. For example, in some embodiments, the image recognition method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit.

[0113] In some embodiments, part or all of the computer program can be loaded and / or installed on the device via the ROM and / or the communication unit. When the computer program is loaded into the RAM and executed by the computing unit, one or more steps of the image recognition method described above can be performed. Alternatively, in other embodiments, the computing unit can be configured to perform the image recognition method in any other appropriate manner (e.g., by means of firmware).

[0114] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0115] According to an embodiment of the present application, the method flow according to the embodiment of the present application can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. According to an embodiment of the present application, the electronic device, device, apparatus, module, unit, etc. described above can be implemented by a computer program module.

[0116] The present application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present application is implemented.

[0117] According to an embodiment of the present application, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0118] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0119] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present application may be combined in a variety of ways and / or combinations, even if such combinations or combinations are not explicitly described in the present application. In particular, without departing from the spirit and teachings of the present application, the features described in the various embodiments and / or claims of the present application may be combined in a variety of ways and / or combinations. All of these combinations and / or combinations fall within the scope of the present application.

[0120] The embodiments of the present application are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present application. Although the embodiments are described above, this does not mean that the measures in the various embodiments cannot be used in combination. The scope of the present application is limited by the attached claims and their equivalents. Without departing from the scope of the present application, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present application.

Claims

1. A method for testing embodied intelligent tasks based on an embodied testing platform, characterized in that: The embodied test platform is configured to have m embodied task environments and n task description templates, where m is an integer greater than or equal to 1, and n is an integer greater than or equal to 1. The embodied intelligent task testing method includes: Loading a corresponding embodied task environment according to the acquired task, and generating an embodied task environment text, wherein the embodied task environment includes a scene layout and an embodied intelligence configuration, and the embodied intelligence configuration includes at least the number of embodied intelligences and an initial position of each of the embodied intelligences; Matching the acquired task with a corresponding task description template, generating a task description text from the task using the task description template; and The embodied intelligence is controlled to perform actions according to the embodied task environment text and the task description text until the embodied intelligence completes the task.

2. The embodied intelligent task testing method based on the embodied testing platform according to claim 1, characterized in that: The step of controlling the embodied intelligence to perform actions according to the embodied task environment text and the task description text until the embodied intelligence completes the task includes: Generating initial execution information of the embodied intelligence according to the task description text and the embodied task environment text; The embodied intelligence performs an action according to the execution information, updates the position and scene layout of the embodied intelligence, and obtains an updated position text and an updated scene text, wherein when the embodied intelligence performs an action for the first time in the loaded embodied task environment, the execution information based on it is the initial execution information; When evaluating that the updated position text and the updated scene text meet the conditions for completing the task, the embodied intelligence completes the task; and When it is evaluated that the updated position text and the updated scene text do not satisfy the conditions for completing the task, the execution information of the embodied intelligence is generated according to the task description text, the updated position text and the updated scene text, the embodied intelligence continues to execute actions according to the execution information, the position and scene layout of the embodied intelligence are updated, and the steps of updating the position text and updating the scene text are obtained.

3. The embodied intelligent task testing method based on the embodied testing platform according to claim 2, characterized in that: The step of generating the initial execution information of the embodied intelligence based on the task description text and the embodied task environment text, and the step of generating the execution information of the embodied intelligence based on the task description text, the updated position text and the updated scene text are both performed by a language model or an expert model.

4. The embodied intelligent task testing method based on the embodied testing platform according to claim 3 is characterized in that: The language model has model parameters, which include a mapping relationship between model input and model output, the model input includes task description text, embodied intelligence location text and scene layout text, and the model output includes execution information of embodied intelligence.

5. The embodied intelligent task testing method based on the embodied testing platform according to claim 3, characterized in that: The execution steps of the expert model include: Determine, according to the task description text, the location text of the embodied intelligence, and the scene layout text, a list of object information of objects in the scene layout that need to be operated to complete the task; sorting the objects in the object information list according to the location text of the embodied intelligence and the coordinates of the objects in the object information list according to the shortest path algorithm; and According to the order in which the objects are sorted, execution information of embodied intelligence is generated, wherein the execution information includes a movement path and an operation action.

6. The embodied intelligent task testing method based on the embodied testing platform according to claim 1, characterized in that: The embodied testing platform is configured to have a user operation interface, and the embodied intelligence task testing method also includes: after loading the corresponding embodied task environment according to the acquired task, it can respond to the user's operation on the embodied intelligence in the scene layout, control the embodied intelligence to perform actions until the embodied intelligence completes the task, and when the user operates the user operation interface, stop executing the steps of matching the corresponding task description template according to the acquired task and generating a task description text for the task through the task description template; and control the embodied intelligence to perform actions according to the embodied task environment text and the task description text until the embodied intelligence completes the task.

7. The embodied intelligent task testing method based on the embodied testing platform according to claim 1, characterized in that: After the step of controlling the embodied intelligence to perform actions according to the embodied task environment text and the task description text until the embodied intelligence completes the task, the embodied intelligence task testing method further includes: evaluating the degree to which the embodied intelligence performs the task; and The task score is obtained by scoring the degree of completion.

8. The embodied intelligent task testing method based on the embodied testing platform according to claim 1, characterized in that: The embodied test platform is configured to be able to modify, add and / or delete the embodied task environment in real time; the embodied test platform is configured to be able to modify, add and / or delete the task description template in real time.

9. A method for training a language model, characterized in that: include: Acquire training data, wherein the training data is acquired according to the execution step of the expert model in the embodied intelligence task testing method based on the embodied testing platform according to claim 5, and the training data includes input samples and labeled samples, the input samples are the task description text, the location text of the embodied intelligence and the scene layout text in the execution step of the expert model, and the labeled samples are the execution information in the execution step of the expert model; Training initial model parameters of the language model according to the training data to obtain training model parameters, wherein the training model parameters include a mapping relationship between task description text, a location of embodied intelligence, and a scene layout and execution information; and The training model parameters are applied as model parameters of the language model.

10. An embodied intelligent task testing device based on an embodied testing platform, characterized in that: The embodied test platform is configured to have m embodied task environments and n task description templates, where m is an integer greater than or equal to 1, and n is an integer greater than or equal to 1. The embodied intelligent task test device includes: A loading module, the loading module is used to load the corresponding embodied task environment according to the acquired task and generate an embodied task environment text, wherein the embodied task environment includes a scene layout and an embodied intelligence configuration, and the embodied intelligence configuration includes at least the number of embodied intelligences and the initial position of each of the embodied intelligences; A generating module, the generating module is used to match the acquired task with a corresponding task description template, and generate a task description text from the task through the task description template; and A control module, wherein the control module is used to control the embodied intelligence to perform actions according to the embodied task environment text and the task description text until the embodied intelligence completes the task.

11. An electronic device, characterized in that: include: one or more processors; One or more memories, used to store executable instructions, wherein when the executable instructions are executed by the processor, the method according to any one of claims 1 to 9 is implemented.

12. A computer-readable storage medium, characterized in that: The storage medium stores executable instructions, which, when executed by a processor, implement the method according to any one of claims 1 to 9.

Images