Task scheduling strategy generation method and device, storage medium and program product

Abstract

The invention relates to a task scheduling strategy generation method and device, a storage medium and a program product. The method can comprise the steps of obtaining task description information of a to-be-executed task of an agent with a body and scene information of a target scene; wherein the task description information comprises expected execution durations of at least two sub-tasks, and the target scene is an operation scene in which the agent with the body executes the task; on the basis of the task description information and the scene information, a task scheduling strategy of an agent executing tasks in the target scene is generated, the task scheduling strategy comprises the execution sequence of the at least two sub-tasks, and the execution time interval of at least part of the sub-tasks is overlapped with the execution time interval of other sub-tasks. According to the technical scheme, the task scheduling flexibility and the task execution efficiency can be improved, and the total task execution duration is shortened.

Description

Technical Field

[0001] This disclosure relates to the field of task scheduling technology, and in particular to a method, device, storage medium, and program product for generating task scheduling strategies. Background Technology

[0002] With the continuous development of artificial intelligence and natural language processing technologies, intelligent systems (such as embodied agents) have made significant progress in scene understanding and task execution capabilities, and their applications have widely penetrated into many important fields such as service robots, smart homes, and industrial automation. In these application scenarios, users are increasingly inclined to use language commands to instruct intelligent systems to complete scene interaction tasks, such as "start the production line and complete product quality inspection."

[0003] In related technologies, intelligent systems typically generate step-by-step execution plans based on user instructions, decomposing the original task corresponding to the user instructions into several logically coherent and operable subtasks. However, these subtasks must be executed sequentially in a strict serial order. This results in excessively long total execution time for all subtasks, leading to low task execution efficiency and failing to meet users' demands for a highly efficient interactive experience. Summary of the Invention

[0004] This disclosure provides a method, device, storage medium, and program product for generating task scheduling strategies to address the shortcomings of related technologies.

[0005] According to a first aspect of the present disclosure, a method for generating a task scheduling strategy is proposed, comprising: Obtain task description information of the task to be performed by the embodied intelligent agent, and scene information of the target scene; wherein, the task description information includes the expected execution time of at least two sub-tasks, and the target scene is the operation scene in which the embodied intelligent agent performs the task; Based on the task description information and the scene information, a task scheduling strategy is generated for the embodied intelligent agent to perform tasks in the target scene. The task scheduling strategy includes the execution order of the at least two sub-tasks, wherein the execution time intervals of at least some sub-tasks overlap with the execution time intervals of other sub-tasks.

[0006] Optionally, the execution time interval of at least some subtasks overlaps with the execution time interval of other subtasks, including: the execution time interval of at least some subtasks is within the execution time interval of another subtask.

[0007] Optionally, the step of generating a task scheduling strategy for an embodied agent to perform tasks in the target scene based on the task description information and the scene information includes: inputting the scene information and the task description information into a task localization model to obtain the parallelism attribute of each of the at least two subtasks, wherein the parallelism attribute indicates whether the subtask can be executed in parallel with other subtasks; and generating the task scheduling strategy based on the parallelism attribute of each subtask, the expected execution time in the task description information, and the inter-task dependencies.

[0008] Optionally, the step of inputting the scene information and the task description information into the task localization model to obtain the parallelism attribute of each of the at least two subtasks includes: the task localization model determines the parallelism attribute of each subtask based on the task description information and the scene information, and encodes the parallelism attribute of each subtask, the expected execution time in the task description information, and the inter-task dependency relationship into a planning token for output; the planning token is used to indicate: calling the optimization solver to generate the task scheduling strategy based on the planning token; the step of generating the task scheduling strategy based on the parallelism attribute of each subtask, the expected execution time in the task description information, and the inter-task dependency relationship includes: generating the task scheduling strategy through the optimization solver based on the parallelism attribute, expected execution time, and inter-task dependency relationship contained in the planning token.

[0009] Optionally, inputting the scene information and the task description information into the task localization model includes: converting the scene information into a corresponding scene token and the task description information into a corresponding task token; and inputting the scene token and the task token into the task localization model.

[0010] Optionally, the step of inputting the scene information and the task description information into the task localization model to obtain the parallelism attribute of each of the at least two sub-tasks includes: the task localization model obtaining the semantic association between the scene information and the task description information through a cross-modal attention mechanism; and determining the parallelism attribute of each sub-task based on the semantic association.

[0011] Optionally, after generating a task scheduling strategy for the embodied intelligent agent to perform tasks in the target scene, the method further includes: inputting the task scheduling strategy and the task description information into the task localization model to obtain a task execution plan corresponding to the task to be executed, wherein the task execution plan includes: the expected execution object corresponding to each sub-task in the target scene; and determining the spatial location of the expected execution object corresponding to each sub-task in the target scene based on the task execution plan and the scene information of the target scene.

[0012] Optionally, the scene information of the target scene includes spatial perception data of the target scene and object information of foreground objects; determining the spatial location of the expected execution object corresponding to each subtask in the target scene based on the task execution plan and the scene information of the target scene includes: determining the similarity between the expected execution object and the foreground object based on the object information of the expected execution object and the object information of the foreground object, and determining the foreground object whose similarity meets the preset conditions as the target execution object corresponding to the expected execution object; determining the spatial location of each target execution object in the target scene based on each target execution object and the spatial perception data.

[0013] Optionally, the task execution plan further includes a location token corresponding to each expected execution object; the step of calculating the similarity between each expected execution object and each foreground object based on the object information of the expected execution object and the object information of the foreground object includes: mapping the location token corresponding to each expected execution object to obtain an expected execution object vector set, which contains the feature vector corresponding to each expected execution object; calculating the similarity between each expected execution object and each foreground object based on the expected execution object vector set and the foreground object vector set; the foreground object vector set is obtained by vector transformation based on the object information of the foreground object.

[0014] Optionally, before determining the foreground object with the highest similarity as the target execution object corresponding to the expected execution object, the method further includes: obtaining the temperature parameter corresponding to the scene type based on the scene type of the target scene; and correcting the calculated similarity based on the obtained temperature parameter.

[0015] According to a second aspect of the present disclosure, a method for training a task localization model is proposed, comprising: The sample task description information, target task scheduling strategy, and sample scenario information of the embodied intelligent agent sample execution task are obtained. The sample task description information includes the expected execution time of at least two sample sub-tasks and the dependencies between tasks. The target task scheduling strategy includes the execution order of the at least two sample sub-tasks. The execution time intervals of at least some sample sub-tasks overlap with the execution time intervals of other sample sub-tasks. The sample scene information and the sample task description information are input into the pre-trained task localization model to obtain the sample parallelism attribute of each sample subtask in the at least two sample subtasks. The sample parallelism attribute indicates whether the sample subtask can be executed in parallel with other sample subtasks. Based on the sample parallelism attribute of each sample subtask, the expected execution time in the sample task description information, and the inter-task dependencies, a sample task scheduling strategy is generated for the embodied intelligent agent to execute tasks in the sample scenario. Based on the difference between the sample task scheduling strategy and the target task scheduling strategy, a strategy loss value is calculated, and the pre-trained task localization model is iteratively trained based on the strategy loss value.

[0016] Optionally, calculating the strategy loss value based on the difference between the sample task scheduling strategy and the target task scheduling strategy includes: obtaining the target task execution plan corresponding to the sample execution task, wherein the target task execution plan contains the target execution object corresponding to each sample subtask in the sample scenario; inputting the sample task scheduling strategy and the sample task description information into the pre-trained task localization model to obtain the sample task execution plan corresponding to the sample execution task, wherein the sample task execution plan contains the predicted execution object corresponding to each sample subtask in the sample scenario; and calculating the loss value as the strategy loss value based on the difference between the target task execution plan and the sample task execution plan.

[0017] Optionally, the method further includes: inputting the sample task execution plan and the sample scene information into a pre-trained spatial localization model to obtain the predicted spatial position of the predicted execution object corresponding to each sample sub-task in the sample scene; calculating a spatial localization loss value based on the predicted spatial position of the predicted execution object corresponding to each sample sub-task and the target spatial position of the target execution object corresponding to the sample sub-task in the sample scene; and iteratively training the pre-trained task localization model based on the policy loss value, including: iteratively training the pre-trained task localization model and the pre-trained spatial localization model together based on the policy loss value and the spatial localization loss value.

[0018] According to a third aspect of the present disclosure, an apparatus for generating a task scheduling strategy is provided, comprising: The acquisition unit is used to acquire task description information of the task to be executed by the embodied intelligent agent and scene information of the target scene; wherein, the task description information includes the expected execution time of at least two sub-tasks, and the target scene is the operation scene in which the embodied intelligent agent performs the task; The strategy generation unit is used to generate a task scheduling strategy for the embodied intelligent agent to perform tasks in the target scene based on the task description information and the scene information. The task scheduling strategy includes the execution order of the at least two sub-tasks, wherein the execution time intervals of at least some sub-tasks overlap with the execution time intervals of other sub-tasks.

[0019] Optionally, the strategy generation unit is specifically used to: input the scene information and the task description information into the task localization model to obtain the parallelism attribute of each of the at least two subtasks, wherein the parallelism attribute indicates whether the subtask can be executed in parallel with other subtasks; and generate the task scheduling strategy based on the parallelism attribute of each subtask, the expected execution time in the task description information and the inter-task dependency relationship.

[0020] Optionally, the device further includes: a spatial positioning unit, configured to: after generating a task scheduling strategy for the embodied intelligent agent to perform a task in the target scene, input the task scheduling strategy and the task description information into the task positioning model to obtain a task execution plan corresponding to the task to be executed, wherein the task execution plan includes: the expected execution object corresponding to each sub-task in the target scene; and determine the spatial location of the expected execution object corresponding to each sub-task in the target scene based on the task execution plan and the scene information of the target scene.

[0021] According to a fourth aspect of the embodiments of this disclosure, an electronic device is provided, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to implement the method described in the embodiments of the first or second aspect above.

[0022] According to a fifth aspect of the present disclosure, a computer-readable storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the steps of the methods described in the embodiments of the first or second aspect above.

[0023] According to a sixth aspect of the present disclosure, a computer program product is provided, comprising a computer program / instructions that, when executed by a processor, implement the steps of the methods described in the embodiments of the first or second aspect above.

[0024] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects: As described in the above embodiments, this disclosure obtains task description information and target scene information of the task to be executed by the embodied intelligent agent when the agent performs a task. The target scene is the operational scenario in which the embodied intelligent agent performs the task, and the task description information includes the expected execution time of at least two sub-tasks. These at least two sub-tasks are logically feasible and operable, obtained by decomposing the task to be executed. Furthermore, based on the task description information and scene information, a task scheduling strategy for the embodied intelligent agent to execute the task in the target scene is generated. This task scheduling strategy includes the execution order of the aforementioned at least two sub-tasks, and the execution time intervals of at least some sub-tasks overlap with the execution time intervals of other sub-tasks. This means that these at least some sub-tasks and other sub-tasks can be executed in parallel within the overlapping time intervals, thereby reducing waiting time and resource waste during sub-task execution. Through this flexible task scheduling strategy, the total execution time of the task to be executed can be effectively reduced, task execution efficiency can be improved, and the user's interactive experience can be enhanced.

[0025] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a schematic diagram illustrating a task scheduling strategy generation system according to an embodiment of the present disclosure.

[0028] Figure 2 This is a schematic flowchart illustrating a method for generating a task scheduling strategy according to an embodiment of the present disclosure.

[0029] Figure 3 This is a flowchart illustrating a training method for a task localization model according to an embodiment of the present disclosure.

[0030] Figure 4 This is a schematic diagram illustrating the entire process of generating a task scheduling strategy according to an embodiment of the present disclosure.

[0031] Figure 5 This is a schematic block diagram of a task scheduling strategy generation apparatus according to an embodiment of the present disclosure.

[0032] Figure 6This is a schematic block diagram of a training apparatus for a task localization model according to an embodiment of the present disclosure.

[0033] Figure 7 This is a schematic block diagram of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0034] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this disclosure.

[0035] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the present disclosure. The singular forms “a” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0036] It should be understood that although the terms first, second, third, etc., may be used to describe various information in embodiments of this disclosure, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of embodiments of this disclosure, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0037] For the sake of brevity and ease of understanding, this document uses the terms "greater than" or "less than", "higher than" or "lower than" to describe size relationships. However, it will be understood by those skilled in the art that the term "greater than" also includes the meaning of "greater than or equal to", and "less than" also includes the meaning of "less than or equal to"; the term "higher than" also includes the meaning of "higher than or equal to", and "lower than" also includes the meaning of "lower than or equal to".

[0038] In related technologies, embodied agents, as intelligent systems with environmental awareness and physical interaction capabilities, are widely used in various complex scenarios. Embodied agents typically rely on the semantic understanding and reasoning capabilities of Large Language Models (LLMs) to understand and execute tasks corresponding to user instructions. However, current task planning methods for embodied agents (such as TaPA and LEO algorithms) often generate logically correct, sequential task execution plans that adhere to scenario constraints. These execution plans must be executed strictly in a serial order, which can easily lead to excessively long total execution time for all subtasks, resulting in low task execution efficiency.

[0039] In view of this, this disclosure proposes a method for generating task scheduling strategies, which can improve the flexibility of task scheduling, thereby reducing the total execution time of tasks and improving task execution efficiency.

[0040] The following describes one or more embodiments of this disclosure in detail.

[0041] The embodiments of this disclosure can be applied to electronic devices, including but not limited to smartphones, desktop computers, tablets, laptops, e-book readers, smartwatches, smart bracelets, and embodied intelligent agents. One or more embodiments of this disclosure are not limited thereto. During operation, the electronic device can run a task scheduling strategy generation system to generate a task scheduling strategy and send the task scheduling strategy to the execution module of the embodied intelligent agent, so that the execution module can drive the embodied intelligent agent to execute tasks according to the task scheduling strategy. The application program for the task scheduling strategy generation system can be pre-installed on the electronic device, allowing the task scheduling strategy generation system to be started and run on the electronic device; of course, when using technologies such as HTML5, it is not necessary to install the corresponding application on the electronic device to obtain and run the task scheduling strategy generation system.

[0042] In another embodiment, the task scheduling policy generation system may include, for example, Figure 1 The diagram shows a server 11, a network 12, and an electronic device 13. During operation, the server 11 can run the server-side program of the task scheduling strategy generation system to perform related data processing and strategy generation functions. Meanwhile, the electronic device 13 can run the client-side program of the task scheduling strategy generation system to perform related data acquisition and human-computer interaction functions. Thus, the server 11 and the electronic device 13 work together to realize the task scheduling strategy generation scheme.

[0043] Server 11 can be a physical server containing an independent host, or it can be a virtual server hosted by a host cluster. Electronic device 13 can be any type of device, such as a PC, tablet, laptop, PDA (Personal Digital Assistants), mobile phone, wearable device (such as smart glasses, smartwatches, etc.), etc., and this specification does not limit this to any of the embodiments described herein. The network 12 for interaction between electronic device 13 and server 11 can include various types of wired or wireless networks. In one embodiment, network 12 can include the Public Switched Telephone Network (PSTN) and the Internet.

[0044] Figure 2 This is a schematic flowchart illustrating a method for generating a task scheduling strategy according to embodiments of this disclosure. It should be noted that the solution disclosed herein is applicable to the fields of robot task scheduling, automated industrial process optimization, or other fields or scenarios with flexible task scheduling requirements. Figure 2 As shown, the method may include the following steps: S201: Obtain task description information of the task to be executed by the embodied intelligent agent, and scene information of the target scene; wherein, the task description information includes the expected execution time of at least two sub-tasks, and the target scene is the operation scene in which the embodied intelligent agent executes the task.

[0045] An embodied intelligent agent refers to an intelligent system that possesses a physical or virtual body and is capable of sensing and acting in relation to its environment. It can receive external input and take actions based on its understanding of the environment to complete specific tasks. In this embodiment, it is necessary to obtain the task description information of the task to be performed by the embodied intelligent agent and the scene information of the target scene.

[0046] The task description information is used to characterize the task to be performed by the embodied intelligent agent, and it includes the expected execution time of at least two sub-tasks. The task to be performed can be reasonably decomposed into at least two logically coherent and executable sub-tasks. For example, the task to be performed, "heating food in a microwave oven," can be broken down into five sub-tasks: "putting the food in the microwave oven," "starting the microwave oven," "waiting for the microwave oven to heat," "removing the food," and "turning off the microwave oven." The expected execution time of each sub-task reflects the time cost of that sub-task. Furthermore, the task description information may also include the task name, task content, or other information related to each sub-task. It should be noted that the form of the task description information can be natural language text, voice commands, or other forms of expression; this disclosure does not impose any limitations on this.

[0047] The target scenario refers to the operational context in which the embodied intelligent agent performs a task. Within this scenario, the embodied intelligent agent can complete the task according to user instructions. This disclosure does not specifically limit the type of the target scenario; it can be a home service scenario, an industrial operation scenario, a virtual simulation scenario, etc. Scenario information of the target scenario refers to environmental information that comprehensively describes the environment upon which the embodied intelligent agent relies to perform its task. This information may include, but is not limited to: the spatial layout of the environment, lighting conditions, temperature, dynamic obstacles, static facilities, and various objects (such as furniture, tools, and people) and their attributes (such as location, state, interactivity, etc.) existing in the target scenario. The forms of expression for scenario information may include, but are not limited to, images, point cloud data, semantic maps, text descriptions, and sensor data.

[0048] S202: Based on the task description information and the scene information, generate a task scheduling strategy for the embodied intelligent agent to perform tasks in the target scene. The task scheduling strategy includes the execution order of the at least two sub-tasks, wherein the execution time intervals of at least some sub-tasks overlap with the execution time intervals of other sub-tasks.

[0049] By integrating task description information and target scenario information, a reasonable task planning and scheduling strategy can be generated by understanding the task objective and incorporating environmental constraints from the scenario information. For example, based on the expected execution time of each subtask in the task description information and the resource information in the scenario information, the execution time intervals and scenario resources of each subtask can be reasonably allocated to avoid resource contention. In one possible implementation, the generated task scheduling strategy may also include time optimization explanation information to explain the rationality of the scheduling strategy and the basis for the parallel execution of subtasks, thereby assisting the embodied agent in understanding and facilitating traceability.

[0050] In this embodiment, the task scheduling strategy includes the execution order of at least two sub-tasks, and the execution time intervals of at least some sub-tasks overlap with the execution time intervals of other sub-tasks. That is, within the overlapping execution time intervals, the aforementioned at least some sub-tasks and other sub-tasks can be executed in parallel, thereby shortening the overall execution time of the task to be executed. For example, the target scenario is a kitchen, and the task description information for the task to be executed is "tidying up the kitchen, including wiping the table - 5 minutes, boiling water - 10 minutes, washing dishes - 8 minutes". Then, by combining the task description information and the scenario information of the target scenario, it can be analyzed that the "washing dishes" sub-task can be executed while waiting for the water to boil, thus shortening the overall execution time of the task to be executed. That is, the execution time interval of the "washing dishes" sub-task can be designed to overlap with the execution time interval of the "boiling water" sub-task. The generated task scheduling strategy can be: first execute the "boiling water" sub-task, wait 5 minutes, then execute the "washing dishes" sub-task; when the water boils, pause the "washing dishes" sub-task, pour the water into a thermos, and then continue executing the "washing dishes" sub-task; finally, execute the "wiping the table" sub-task. Thus, the execution time intervals of the subtasks "washing dishes" and "boiling water" overlap by 5 minutes, and the estimated total execution time for the tasks to be executed is 18 minutes. Compared to the task scheduling strategies in related technologies that strictly execute subtasks sequentially (estimated to take 23 minutes), this significantly shortens the total execution time and improves task execution efficiency. For example, the generated task scheduling strategy can be sent to the execution module of the embodied intelligent agent, so that the execution module can execute the tasks to be executed in the target scenario according to the task scheduling strategy.

[0051] In the above embodiments, when planning the task scheduling strategy, the execution time intervals of some sub-tasks are designed to overlap, so that multiple sub-tasks can be executed in parallel within the overlapping execution time intervals. This greatly reduces the waiting time and resource waste during task execution, effectively reduces the total execution time of the tasks to be executed, helps to improve task execution efficiency, and enhances the user's interactive experience.

[0052] In one embodiment, the generated task scheduling strategy ensures that the execution time intervals of at least some subtasks fall within the execution time interval of another subtask. In other words, the execution time intervals of at least some subtasks are completely covered by the execution time interval of another subtask. This means that the aforementioned at least some subtasks can be completed while the other subtask is being executed.

[0053] Referring to the example in the previous embodiment, the target scenario is a kitchen, and the task description information for the task to be performed is "tidying up the kitchen, including wiping the table - 5 minutes, boiling water - 10 minutes, washing dishes - 8 minutes". Through analysis, the execution time interval of the "washing dishes" subtask can be designed to be within the execution time interval of the "boiling water" subtask, while the execution time interval of the "wiping the table" subtask overlaps with that of the "boiling water" subtask. That is, the "wiping the table" and "washing dishes" subtasks can be performed while waiting for the water to boil. The generated task scheduling strategy can be: first execute the "boiling water" subtask, and while waiting for the water to boil, execute the "washing dishes" subtask first, then execute the "wiping the table" subtask; when the water boils, pause the "wiping the table" subtask, pour the water into a thermos, and then continue executing the "wiping the table" subtask until it ends. Thus, the total execution time of the task to be performed is expected to be 13 minutes.

[0054] This embodiment designs at least some subtasks to have execution time intervals within the execution time interval of another subtask, thereby making full use of the waiting time of the other subtask to complete at least some subtasks, improving the utilization rate of the waiting time of the other subtask, and further shortening the total execution time of the tasks to be executed, thus obtaining a task scheduling strategy with the minimum total task execution time.

[0055] In one embodiment, during the generation of a task scheduling strategy, the task description information of the task to be executed and the scene information of the target scene can be input into the task localization model. The task localization model can determine the parallelism attribute of each subtask based on the input scene information and task description information, and output the result. The parallelism attribute refers to whether a subtask can be executed in parallel with other subtasks, i.e., whether a subtask has the possibility of being executed simultaneously with other subtasks. For example, the task localization model can determine whether each subtask has the possibility of being executed simultaneously with other subtasks based on the availability of scene resources, device status, number of devices, etc., contained in the scene information, and the task type, inter-task dependencies, execution constraints, etc., contained in the task description information, through semantic analysis to determine whether each subtask has resource conflicts, data dependencies, or logical order contradictions with other subtasks. It should be noted that this disclosure does not particularly limit the specific implementation form of the task localization model; it can be any model with semantic understanding and learning capabilities, such as models based on deep neural networks, traditional machine learning models, large language models, etc.

[0056] Furthermore, based on the parallelism attributes of each subtask output by the task localization model and the expected execution time and inter-task dependencies in the task description information, a task scheduling strategy is generated. Inter-task dependencies refer to the fact that the start or completion of one subtask requires the start or completion of another (or more) subtasks as a prerequisite. For example, the "watering flowers" subtask can only be executed after the "filling the watering can" subtask is completed. Therefore, even if one subtask can be executed in parallel with another, the execution time intervals of these two subtasks may not overlap due to inter-task dependencies. Therefore, when generating a task scheduling strategy, it is necessary to comprehensively consider the parallelism attributes of each subtask, inter-task dependencies, and expected execution time to generate a logically coherent and effective task scheduling strategy, thereby avoiding errors when scheduling tasks based on the task scheduling strategy. Of course, inter-task dependencies can also be empty, meaning that each subtask is independent. In this case, the task scheduling strategy can be generated based on the parallelism attributes and expected execution time of each subtask.

[0057] This embodiment leverages the powerful language understanding and analysis capabilities of the task localization model to accurately identify the parallelism attributes of each subtask. Based on the parallelism attributes of the subtasks, the expected execution time, and the dependencies between tasks, a more efficient and accurate task scheduling strategy can be planned, which can improve the overall resource utilization of the target scenario and the execution efficiency of the tasks to be executed.

[0058] In one embodiment, during the process of inputting scene information of the target scene and task description information of the task to be executed into the task localization model, the scene information can first be converted into a corresponding scene token, and the task description information can be converted into a corresponding task token. This disclosure does not specifically limit the specific conversion method. For example, scene information can be encoded into a corresponding scene token using a scene encoder, and task description information can be encoded into a corresponding task token using a text segmenter. In the field of natural language processing, a token can be understood as a basic unit obtained after segmenting the original data. Through a unified token representation method, data from different modalities can be aligned and fused in the same semantic space. Then, the scene token and task token are input into the task localization model, and the parallelism attribute of each subtask is output.

[0059] This embodiment places the token conversion process outside the task localization model, that is, it completes the mapping from multimodal data to a unified token form before the model input. This approach eliminates the need for the task localization model to directly process the raw multimodal data, thereby reducing the computational complexity of the task localization model; furthermore, the task localization model can focus on learning the semantic relationships between tokens and the task logic, thus reducing the complexity of model training.

[0060] In one embodiment, after inputting the scene information of the target scene and the task description information of the task to be executed into the task localization model, the task localization model can obtain the semantic association between the scene information and the task description information through a cross-modal attention mechanism, and then determine the parallelism attribute of each sub-task based on the semantic association.

[0061] The core idea of ​​cross-modal attention mechanisms is to dynamically guide or weight the representation of data from one modality, thereby achieving more effective multimodal understanding. In other words, through cross-modal attention mechanisms, task localization models can better achieve fine-grained semantic alignment of input data from different modalities, capturing the semantic relationships between cross-modal data. This enhances the understanding of task description and scene information, accurately identifying the scene object corresponding to each subtask within the scene information. Furthermore, based on cross-modal semantic relationships, the parallelism attribute of each subtask can be accurately determined, improving the accuracy of parallelism attributes.

[0062] In one embodiment, an optimization solver can be invoked to generate a task scheduling strategy. Based on operations research principles, the optimization solver, under a given set of constraints, searches for values ​​of one or more variables that optimize a certain objective function. In this embodiment, the objective of the optimization solver can be that the total execution time of the task to be executed is less than the sum of the expected execution times of each subtask. Using this as the objective, the optimization solver can generate a task scheduling strategy where the execution time intervals of at least some subtasks overlap with the execution time intervals of other subtasks, based on the parallelism attributes of each subtask, its expected execution time, and the dependencies between tasks.

[0063] For example, planning tokens can be used to establish an interaction interface between the task localization model and the external optimization solver, thereby enabling explicit optimization of the time scheduling of multiple subtasks. The following is a detailed explanation: The scenario information of the target scene and the task description information of the tasks to be executed are input into the task localization model. Based on the input, the task localization model can identify the parallelism attributes of each subtask within the task to be executed. The task localization model can extract the expected execution time and inter-task dependencies of each subtask from the task description information, and then encode the parallelism attributes, expected execution time, and inter-task dependencies of each subtask into planning tokens. Finally, the task localization model outputs planning tokens. Planning tokens can be understood as a set of predefined special symbols output by the task localization model, used to transmit the parallelism attributes, expected execution time, and inter-task dependencies of each subtask to the external optimization solver in a structured manner. It is equivalent to a communication protocol, instructing the optimization solver to generate a task scheduling strategy, thus acting as an "interface" between the task localization model and the optimization solver.

[0064] The optimization solver uses planning tokens to obtain the parallelism attributes, expected execution time, and inter-task dependencies of each subtask. It solves the problem with the objective that "the total execution time of the tasks to be executed is less than the sum of the expected execution times of each subtask" and outputs a task scheduling strategy.

[0065] This embodiment encodes the parallelism attributes, expected execution time, and inter-task dependencies of each subtask into planning tokens, and passes the planning tokens to the optimization solver outside the model. This allows the two completely different systems, the task localization model and the optimization solver (one handles natural language, and the other handles mathematical optimization), to collaborate seamlessly through planning tokens, generate a better task scheduling strategy, significantly improve the collaboration efficiency between the task localization model and the optimization solver, and enhance the quality of the task scheduling strategy.

[0066] In one embodiment, after generating the task scheduling strategy, in order to ensure the accuracy of the embodied agent in executing tasks in the target scene, it is also necessary to accurately locate the execution objects of each subtask included in the task scheduling strategy in the target scene. However, related technologies can usually only achieve spatial positioning of a single action, with poor positioning accuracy, and cannot achieve the positioning of continuous, complex, and multi-subtask overall task objects.

[0067] Therefore, in this embodiment, after generating the task scheduling strategy for the embodied intelligent agent to perform tasks in the target scene, the task scheduling strategy and task description information are input into the task localization model to obtain the task execution plan corresponding to the task to be executed. This task execution plan includes the expected execution object corresponding to each subtask in the target scene. The expected execution object refers to the object used to execute the subtask in the target scene. Each subtask can correspond to one or more expected execution objects. For example, in a kitchen scene, the expected execution object corresponding to the "heating food" subtask is a microwave oven; the expected execution objects corresponding to the "cleaning the sink" subtask are a rectangular sink and a dishcloth.

[0068] For example, the task localization model can parse the task scheduling strategy and task description information to obtain information such as the task objective, task content, execution order, execution time interval, and overlap relationship of each sub-task. Then, based on this information, a task execution plan is generated. The task execution plan clarifies the execution order of each sub-task and the expected execution object corresponding to each sub-task. For example, combining the task to be executed in the aforementioned embodiment, "tidying up the kitchen, including wiping the table - 5 minutes, boiling water - 10 minutes, washing dishes - 8 minutes," the task execution plan can be: 1) First, start the water heater to boil water; 2) While the water heater is working, wash the dishes in the sink; 3) After washing the dishes, start wiping the table; 4) After the water heater finishes boiling, pause wiping the table and pour the water from the water heater into a thermos; 5) Continue wiping the table, and the process ends.

[0069] Then, based on the task execution plan and the scene information of the target scene, the spatial location of the expected execution object corresponding to each subtask in the target scene is determined. It is understood that the scene information of the target scene encompasses the spatial location information or relative positional relationships of all identifiable objects in the target scene. Therefore, by combining the scene information and the expected execution object corresponding to each subtask in the task execution plan, the spatial location of each expected execution object in the target scene can be accurately matched and located. This disclosure does not specifically limit the specific implementation method for determining the spatial location. For example, positioning can be achieved using high-level semantic information such as natural language description and spatial relationship reasoning, or by using semantic SLAM (Simultaneous Localization and Mapping) technology.

[0070] This embodiment leverages the powerful semantic understanding capabilities of the task localization model to obtain the semantic association between the task scheduling strategy and task description information, thereby generating a task execution plan containing the expected execution object corresponding to each subtask. Then, the task execution plan is matched with the scene information of the target scene to determine the spatial location of the expected execution object corresponding to each subtask within the target scene, significantly improving the accuracy of localization. In this way, the embodied agent can execute the corresponding subtask using the execution object located at the precisely located spatial position, ensuring the smooth execution of each subtask and avoiding task failure or errors due to localization errors, thus significantly improving the overall completion rate and execution stability of the tasks to be executed.

[0071] In one embodiment, the scene information of the target scene may include spatial perception data of the target scene and object information of foreground objects. Spatial perception data refers to data used to understand or reconstruct spatial geometry and positional relationships, thereby supporting spatial positioning, and may include, but is not limited to, 3D point cloud data, 2D images, depth maps, and UWB (Ultra-Wideband) ranging / positioning information. Foreground objects refer to the main objects or targets located in the foreground of the image corresponding to the target scene. The object information of the foreground objects may include, but is not limited to: object name, object type, object spatial location, object color, and object function. The spatial perception data of the target scene and the object information of the foreground objects can be obtained from a scene database, which stores a large amount of pre-acquired spatial perception data and foreground object information for different scenes. Alternatively, the spatial perception data of the target scene and the object information of the foreground objects can also be obtained temporarily. For example, an embodied intelligent agent captures scene images of its surroundings through its deployed sensors, and then obtains the object information of the foreground objects by processing the scene images (such as foreground segmentation, semantic segmentation, instance segmentation, etc.). The above are merely examples, and this disclosure does not specifically limit the method, source, or type of acquisition of spatial perception data and object information of foreground objects.

[0072] In determining the spatial location of the expected execution object in the target scene based on the task execution plan and scene information of the target scene, the object information of each expected execution object can be extracted from the task execution plan first. Then, based on the object information of each expected execution object and the object information of the foreground objects, the similarity between each expected execution object and each foreground object can be calculated. For example, semantic features (such as category labels, attribute sets, embedding vectors, etc.) can be extracted from the object information of each expected execution object and foreground object, and the similarity between each expected execution object and each foreground object can be calculated based on the extracted semantic features using methods such as cosine similarity, Euclidean distance, and Jaccard similarity. Alternatively, the similarity between each expected execution object and each foreground object can be calculated by further combining the subtask context corresponding to the expected execution object and the contextual association between each foreground object and other foreground objects. The above are merely examples, and this disclosure does not limit the specific implementation method of similarity calculation.

[0073] After obtaining the similarity results between each expected execution object and each foreground object, for each expected execution object, the similarity that meets the preset conditions is determined from all its corresponding similarity results. The preset conditions can be maximum similarity, similarity meeting a preset threshold, or other conditions; this disclosure does not limit this. Then, the foreground object corresponding to the similarity that meets the preset conditions is determined as the best match for that expected execution object in the target scene, i.e., the target execution object. In other words, when the embodied intelligent agent actually executes each sub-task in the target scene, it completes the task by identifying and acting on target execution objects that are semantically or functionally similar to its expected execution object. Subsequently, since the spatial perception data of the target scene contains the spatial location information of the foreground objects contained in the target scene, the spatial location of each target execution object in the target scene can be determined based on the spatial perception data of the target execution objects corresponding to each sub-task and the target scene itself.

[0074] Taking spatial perception data as 3D point cloud data as an example, 3D point cloud data records a large number of spatial points in the target scene in the form of a high-density point set. It usually includes 3D coordinate information and may also include information such as color, intensity or normal vector. Through point cloud segmentation, clustering or target recognition and other technologies, a subset of point cloud belonging to each target execution object is extracted from the 3D point cloud data; then, based on the spatial distribution characteristics of each point cloud subset (such as centroid, bounding box, key point or edge, etc.), the spatial position of the corresponding target execution object in the target scene is accurately inferred. For example, a 3D mask of each target execution object can be generated based on the object information of each target execution object and the 3D point cloud data. The core idea of ​​the 3D mask is to mark or select a specific region / object / pixel through a binary or probabilistic structure aligned with the 3D data space, so as to mark certain positions in the 3D space and indicate which parts belong to the target. For the calculation principle of the 3D mask, please refer to the following formula (1): (1) In formula (1), F represents the three-dimensional point cloud features corresponding to the three-dimensional point cloud data. A feature vector representing any target execution object (which can be obtained by converting the object information of the target execution object). The sigmoid function represents the probability that each point in the 3D point cloud data belongs to the target execution object. Formula (1) represents the probability that each point in the 3D point cloud data belongs to the target execution object by performing a dot product operation between the feature vector corresponding to any target execution object and the 3D point cloud features, and then normalizing it through the sigmoid function, thereby obtaining the 3D mask of the target execution object. The 3D mask can accurately identify the spatial position of the expected execution object corresponding to each subtask in the 3D space corresponding to the target scene, thereby achieving accurate alignment between the task execution plan and the target scene, and providing a semantically interpretable spatial positioning basis for the operation execution of the embodied intelligent agent.

[0075] This embodiment calculates the similarity between each expected execution object in the task execution plan and each foreground object in the target scene to find the optimal matching object in the target scene for each expected execution object. Furthermore, by combining spatial perception data of the target scene, each optimal matching object is precisely located. This approach achieves accurate association between expected execution objects and physical entities in the target scene, thereby enabling precise location of the execution object for each subtask based on this association. This significantly improves the reliability of the embodied agent in executing tasks and effectively enhances the embodied agent's perception and operational capabilities in complex environments.

[0076] In one embodiment, the task execution plan output by the task localization model may further include a location token corresponding to each expected execution object. A location token is a set of learnable vectors in the task localization model used to represent the location of a potential target. Therefore, the similarity between each expected execution object and each foreground object can be calculated based on the location tokens. For example, the task execution plan might be "1. First, I will start the microwave oven..." <gru>1. Cooking food; 2. Cleaning the rectangular sink while the microwave is running. <gru>;……",in, <gru>A location token is used to encode the spatial location information of a corresponding expected execution object. This location token can indicate the execution object that the operation of each subtask needs to act on in the target scene.

[0077] For example, the location token corresponding to each expected execution object can be mapped to obtain a set of expected execution object vectors, which contains the feature vectors corresponding to each expected execution object. This disclosure does not specifically limit the specific implementation of the location token mapping. For example, the location token can be mapped through a lightweight multilayer perceptron (MLP) to obtain a set of embedding vectors representing the expected execution objects of each subtask, where each embedding vector in the set represents the semantic-spatial joint representation of the corresponding expected execution object under its subtask. The mapping principle can be referred to formula (2): (2) In formula (2), s represents the number of objects to be executed. A location token representing any expected execution object. G The representation embedding vector set (i.e., the set of vectors of the expected execution object) is used. Through mapping, the positioning token can be mapped from its original dimension to the same dimensional space as the 3D scene features of the target scene, achieving feature dimension alignment.

[0078] Then, the similarity between the expected execution object vector set and the foreground object vector set is calculated to obtain the similarity between each expected execution object and each foreground object. The foreground object vector set contains the feature vectors corresponding to each foreground object, which can be obtained by vector encoding the object information of each foreground object.

[0079] This embodiment introduces a location token into the output sequence of the task localization model. This token indicates the expected execution object that each subtask should act upon, enabling precise identification of the expected execution object for each subtask from the task execution plan. This significantly improves the accuracy of identifying the expected execution object, especially when a subtask contains multiple objects. Furthermore, based on the accurate expected execution object, the optimal match can be found in the target scene, ensuring that the embodied agent executes the corresponding subtask through the accurate optimal match, thus improving the reliability and stability of task execution.

[0080] In one embodiment, before determining the foreground object with the highest similarity as the target execution object corresponding to the expected execution object, the scene type of the target scene can be determined, and the temperature parameter corresponding to the scene type can be obtained. Then, the calculated similarity is corrected based on the obtained temperature parameter. Taking cosine similarity as an example, the principle of similarity correction is explained below with reference to formula (3): (3) In formula (3), A feature vector that represents any expected execution object (which can be obtained by converting the object information of the expected execution object). The feature vector representing any foreground object (which can be obtained by converting the object information of the foreground object). Characterizing cosine similarity, Characterizing temperature parameters, Characterizes the corrected similarity.

[0081] Among them, temperature parameter This parameter controls the smoothness of the similarity score or the sharpness of the selection. When the temperature parameter is 1, the original similarity remains unchanged. When the temperature parameter is greater than 1, the original similarity is amplified, increasing the difference between the corrected similarities, resulting in a more concentrated selection (sharp distribution), suitable for scenarios where the best match is selected. When the temperature parameter is less than 1, the original similarity is compressed, the corrected similarities tend to be closer, and the selection will be more ambiguous and uniform (smooth distribution), suitable for scenarios where multiple possible matches are retained. A larger temperature parameter indicates a stronger preference for the "most similar item," meaning the most similar match stands out more prominently in the probability distribution, making it easier to find the most similar foreground object. Therefore, different temperature parameters can be set for different types of operational scenarios. Scenario types can be categorized based on the characteristics of the operational scenario. For example, semantic analysis can be performed on the content of the operational scenario to identify its semantic category, and a preset temperature parameter can be set for each category. This method can adjust the matching sharpness of scenarios with different semantic categories, such as using a larger temperature parameter in complex, detailed operational scenarios to enhance discriminative power. Alternatively, scenario types can be categorized based on the number of similar objects in the operational scenario. When there are many objects with similar appearance or semantics in the operational scenario, using a larger temperature parameter helps to amplify the similarity difference between the true match and the distractors, thereby effectively highlighting the optimal matching result. The above is just an example, and this disclosure does not impose any special limitations on the method of classifying scene types, the method of setting temperature parameters, or specific values.

[0082] This embodiment sets different temperature parameters for different scene types, thereby obtaining the corresponding temperature parameters based on the scene type of the target scene, and using the temperature parameters to correct the similarity, which can effectively improve the accuracy of similarity matching, thereby improving the accuracy of identifying the target execution object.

[0083] The above embodiments describe the scheme for generating task scheduling strategies according to this disclosure. In implementation, a task localization model can be constructed and trained to utilize the trained model to complete the task scheduling strategy generation scheme described in the foregoing embodiments. Figure 3 This is a flowchart illustrating a task localization model training method provided in an exemplary embodiment. Figure 3 As shown, the model training process may include the following steps: S301: Obtain sample task description information, target task scheduling strategy, and sample scenario information of the embodied intelligent agent sample execution task. The sample task description information includes the expected execution time of at least two sample sub-tasks and the dependencies between tasks. The target task scheduling strategy includes the execution order of the at least two sample sub-tasks. The execution time intervals of at least some sample sub-tasks overlap with the execution time intervals of other sample sub-tasks.

[0084] A sample execution task can consist of at least two sample subtasks. The sample task description information can include the expected execution time of these at least two sample subtasks and the dependencies between tasks (such as parallel conditions, execution order, etc.). The target task scheduling strategy is a scheduling scheme for the sample execution task. Its core content includes the execution order of each sample subtask in the sample execution task, and at least some sample subtasks have execution time intervals overlapping with the execution time intervals of other sample subtasks, meaning the target task scheduling strategy supports at least some subtasks to be executed in parallel in time. Therefore, by adopting the target task scheduling strategy, the total execution time of the entire sample execution task will be less than the sum of the expected execution times of all sample subtasks. Therefore, the target task scheduling strategy can be used as a ground truth label during model training. A sample scenario refers to the operational scenario in which the embodied intelligent agent performs the sample execution task. Sample scenario information refers to information used to describe the sample scenario, including but not limited to environmental layout, temperature, and various objects existing in the sample scenario. This disclosure does not specifically limit the acquisition method, specific content, or expression form of the sample task description information, target task scheduling strategy, and sample scenario information.

[0085] For example, the training dataset used in this embodiment covers 4376 sample scenarios and 60825 sample execution tasks, forming a large-scale embodied agent task scheduling sample dataset that integrates operations research knowledge and 3D scene understanding. Each sample includes sample scenario information, a sample execution task, language instructions for each subtask step within the sample execution task, the corresponding target task scheduling strategy, and a scheduling logic description. For instance, a trained deep learning model can be used to generate natural language instructions corresponding to each subtask step, and an optimization solver can be combined to generate a target task scheduling strategy, such as generating a target task scheduling strategy with the minimum total execution time. The average text length of the samples used in this embodiment reaches 311 words, indicating high task complexity, which effectively trains the task localization model's learning capabilities in 3D scene understanding and task parallelism attribute judgment.

[0086] S302: Input the sample scene information and the sample task description information into the pre-trained task localization model to obtain the sample parallelism attribute of each sample subtask in the at least two sample subtasks. The sample parallelism attribute indicates whether the sample subtask can be executed in parallel with other sample subtasks.

[0087] A pre-trained task localization model refers to a model that has completed initial training based on a large amount of unlabeled general data. To improve the performance of the pre-trained task localization model on the vertical task of judging task parallelism attributes, the samples in S301 can be used for secondary training. It should be noted that this disclosure does not limit the specific network framework of the pre-trained task localization model. For example, a multimodal large language model framework can be used, with Vicuna-1B as the core model as the language backbone, thereby realizing the perception and semantic modeling of 3D scene information. The overall training goal is to enable the pre-trained task localization model to accurately judge the parallelism attributes of each sample subtask based on the sample scene information and sample task description information.

[0088] During training, sample scene information and sample task description information are input into a pre-trained task localization model. Based on the input, the model determines whether each sample subtask has the potential to be executed simultaneously with other sample subtasks, thereby outputting the sample parallelism attribute of each sample subtask. In one possible implementation, the pre-trained task localization model can acquire the semantic relationship between sample scene information and sample task description information through a cross-modal attention mechanism, and generate the sample parallelism attribute of each sample subtask based on this semantic relationship.

[0089] S303: Based on the sample parallelism attribute of each sample subtask, the expected execution time in the sample task description information, and the inter-task dependencies, generate a sample task scheduling strategy for the embodied intelligent agent to execute tasks in the sample scenario.

[0090] The process of generating sample task scheduling strategies can be referenced. Figure 2 The details regarding the generation of the task scheduling strategy in the illustrated embodiment will not be elaborated here. In one possible implementation, the sample parallelism attributes, expected execution time, and inter-task dependencies of each sample subtask can be packaged into a planning token. This planning token is then used to invoke an optimization solver located outside the pre-trained task localization model. The optimization solver then generates a sample task scheduling strategy based on the planning token. The sample task scheduling strategy will include the execution order of each sample subtask.

[0091] S304: Calculate the policy loss value based on the difference between the sample task scheduling policy and the target task scheduling policy, and iteratively train the pre-trained task localization model based on the policy loss value.

[0092] The execution order of each sample subtask in the sample task scheduling strategy and the target task scheduling strategy can be compared, as can the overlap in the execution time intervals of the subtasks in these two strategies. The strategy loss value can be calculated, for example using the cross-entropy loss algorithm, or other algorithms. Then, the pre-trained task localization model is iteratively trained based on the strategy loss value.

[0093] In this embodiment, since the sample task scheduling strategy is generated based on the sample parallelism attribute of each subtask output by the pre-trained task localization model, the strategy loss value between the sample task scheduling strategy and the target task scheduling strategy can be calculated. This loss signal is then propagated back to the task localization model for iterative training, thereby effectively guiding the task localization model to continuously adjust its ability to judge the parallelism attribute of subtasks and improving the generalization ability and accuracy of the task localization model in predicting task parallelism attributes.

[0094] In one embodiment, when acquiring samples, the target execution plan corresponding to the sample execution task can also be acquired. This target execution plan corresponds to a target task scheduling strategy and includes the execution order of each sample subtask, the target execution object corresponding to each sample subtask in the sample scenario, and the parallel execution information of the sample subtasks. The target execution object refers to the actual object in the sample scenario where the sample subtask acts or is used to complete the sample subtask. With the target execution plan corresponding to the sample execution task acquired, it is used as the ground truth label during model training. The overall training objective is to enable the pre-trained task localization model to output an execution plan whose total execution time for the sample execution task is less than the sum of the expected execution times of all sample subtasks, based on the sample scenario information and sample task description information.

[0095] For example, during training, the aforementioned sample task scheduling strategy and sample task description information can be concatenated, and the concatenation result can be input into a pre-trained task localization model. This task localization model can predict the predicted execution object corresponding to each sample subtask in the sample scenario based on the input, thereby generating and outputting a sample task execution plan. The sample task execution plan contains the predicted execution object corresponding to each sample subtask in the sample scenario. Then, when calculating the policy loss value, the loss value can be calculated based on the difference between the target task execution plan and the sample task execution plan, serving as the policy loss value. The difference between the target task execution plan and the sample task execution plan can include: the difference between the predicted execution object and the target execution object corresponding to each sample subtask, the difference between the execution order of each sample subtask in the target task execution plan and the sample task execution plan, and so on.

[0096] In the above approach, since the sample task execution plan is output by the pre-trained task localization model and is generated based on the sample task scheduling strategy, the strategy loss value between the target task execution plan and the sample task execution plan is calculated, and this loss signal is backpropagated to the task localization model for iterative training. This effectively improves both the task localization model's ability to judge task parallelism attributes and its task scheduling planning ability, thus effectively expanding the model's generalization capability.

[0097] In one embodiment, each sample may include a sample execution task, language instructions for the steps of each subtask within the sample execution task, a corresponding target task scheduling strategy and scheduling logic description, and a 3D target localization result. The 3D target localization result refers to the actual localization result of the target execution object corresponding to each sample subtask in the sample scene. For example, a trained deep learning model can be used to generate the 3D target localization result based on the spatial perception data of the sample scene and the object information of the sample foreground object, or other methods can be used to obtain the 3D target localization result; this disclosure does not limit this approach.

[0098] In this embodiment, the 3D target localization results and the target task scheduling strategy are used as ground truth labels during model training. The overall training objective is to input sample scene information and sample task description information into the pre-trained task localization model to obtain a task scheduling scheme / execution plan whose total execution time is less than the sum of the expected execution times of all sample sub-tasks, and to accurately locate the real execution objects corresponding to each sample sub-task in the sample scene.

[0099] During training, the sample task execution plan and sample scene information output by the pre-trained task localization model can be input into the pre-trained spatial localization model to obtain the predicted spatial location of the predicted execution object in the sample scene for each sample sub-task. This process can be referenced... Figure 2 The content regarding "predicting the spatial location of the expected execution object corresponding to each subtask in the target scene" in the illustrated embodiment will not be elaborated here. The pre-trained spatial localization model is similar to the pre-trained task localization model; both are models that have completed initial training based on a large amount of unlabeled general data. To improve the performance of the pre-trained spatial localization model in vertical tasks such as 3D scene understanding and 3D spatial localization, it needs to be trained a second time.

[0100] In calculating the loss value, on the one hand, the policy loss value is calculated based on the difference between the target task execution plan and the sample task execution plan; on the other hand, the spatial localization loss value is calculated based on the predicted spatial position of the predicted execution object corresponding to each sample subtask in the sample scene and the target spatial position of the target execution object corresponding to that sample subtask in the sample scene. For example, the binary focus loss of the similarity matrix can be used to calculate the spatial localization loss value; of course, other loss functions can also be used. Furthermore, when using a sample 3D mask to represent the predicted spatial position of the predicted execution object corresponding to each sample subtask in the sample scene, SigmoidFocal Loss can be used as the loss function to calculate the spatial localization loss value between the sample 3D mask generated by the pre-trained spatial localization model and the real target region of the target execution object in the sample scene. Then, based on the policy loss value and the spatial localization loss value, the pre-trained task localization model and the pre-trained spatial localization model are iteratively trained together.

[0101] This embodiment can jointly train the pre-trained task localization model and the pre-trained spatial localization model to simultaneously optimize the pre-trained task localization model's ability to judge task parallelism attributes and its ability to plan task scheduling, as well as the pre-trained spatial localization model's spatial localization ability, thereby improving its spatial localization accuracy.

[0102] Figure 4 This is a schematic diagram illustrating the entire process of generating a task scheduling strategy, provided by an exemplary embodiment. For example... Figure 4 As shown, the model parameters of the trained task localization model and spatial localization model can be loaded from the model database. In this embodiment, the task localization model and spatial localization model support multimodal data input. The entire process of generating the task scheduling strategy is as follows: (4-1) Obtain the 3D point cloud data and foreground object sequence of the target scene. The task description information includes the objectives and constraints of the task to be executed (such as the expected execution time of each subtask), and the 3D point cloud data provides the spatial geometry information of the target scene. The 3D point cloud data of the target scene and the foreground object sequence are input into the 3D scene encoder to obtain the scene token T. S Additionally, the task description information is input into the text encoder to obtain the text token T. t .

[0103] (4-2) Set the scene token T S and text token T t Input the task localization model. The task localization model can identify the parallelism attributes of each subtask through semantic parsing and generate a sequence of parallelism attributes, as shown in formula (4): (4) In formula (4), This represents the i-th subtask. This represents the parallelism attribute of the i-th subtask. These represent parallelizable and non-parallelizable data, respectively. It is a set consisting of n subtasks and their corresponding parallelism attributes.

[0104] Then, the task localization model encodes the parallelism attributes of each subtask, the expected execution time of each subtask contained in the text token Tt, and the inter-task dependencies into a planning token. <sch>The planning token is then passed to the external optimization solver, Solver. The optimization solver parses the planning token to obtain the parallelism attributes, expected execution time, and inter-task dependencies of each subtask. It then solves the problem with the goal of minimizing the total execution time of the tasks to be executed, and outputs the optimal subtask scheduling sequence. The solution principle is shown in Equation (5). (5) In formula (5), The information contained in the planning token is represented by S*, which represents the subtask scheduling sequence. In this embodiment, the subtask scheduling sequence is subtask 2 → subtask 1 → subtask 3 → subtask 2 → subtask 4.

[0105] (4-3) The optimized solver converts the solved subtask scheduling sequence into a natural language description to obtain the task scheduling strategy. This disclosure does not limit the specific conversion method. For example, it can be converted into a natural language description based on a predefined template. In this embodiment, the task scheduling strategy is "to execute each subtask in the following order: start subtask 2, start subtask 1, start subtask 3, review subtask 2, start subtask 4". This means that while executing subtask 2, subtask 1 and subtask 3 are executed sequentially, realizing the parallel execution of multiple subtasks.

[0106] Then, the task scheduling strategy is encoded into a token sequence T using a text segmenter. l . Token sequence T l With text token T t After concatenation, the task localization model is re-inputted, causing it to output a step-by-step task execution plan. This plan includes the action instructions and expected execution objects for each subtask. In this embodiment, the task execution plan is: "1. First, I will start the microwave oven." <gru>1. Cooking food; 2. Cleaning the rectangular sink while the microwave is running. <gru>;……",in, <gru>For location tokens.

[0107] (4-4) This embodiment supports simultaneously locating the specific position of the execution object of each subtask in three-dimensional space while generating the task execution plan, thereby realizing the executability and accuracy of the embodied intelligent agent in the real physical environment. For example, the location token in the task execution plan can be used to indicate the spatial position of the object corresponding to the corresponding action. In order to ensure dimensional matching with the features of the three-dimensional scene, the location token is mapped using a multilayer perceptron to obtain a set of embedding vectors G representing the expected execution object corresponding to each subtask, which can be referred to the aforementioned formula (2), and will not be elaborated here.

[0108] The embedding vector set G, the 3D point cloud data of the target scene, and the foreground object set are used. Input spatial localization model (e.g.) Figure 4 (The 3D positioning head in the image). It should be noted that the foreground object sequence Q can be input into the multilayer perceptron to obtain the foreground object set. This enables dimensional alignment and allows for the learning of more complex feature representations in the foreground object sequence through a multilayer perceptron, which is beneficial for subsequent similarity calculations.

[0109] The spatial positioning model determines the corresponding temperature parameters based on the scene type of the target scene. Then, combining the temperature parameter, the expected execution object and the foreground object set in the embedding vector set G are calculated. The similarity between each foreground object is included, and the principle of similarity calculation can be referred to the aforementioned formula (3). Then, the matching probability between each expected execution object and each foreground object is obtained through the sigmoid function, and the foreground object with the highest probability is selected as the target execution object corresponding to the expected execution object in the target scene. Please refer to formula (6): (6) In formula (6), The similarity of any expected execution object is represented by formula (3). This represents the target execution object corresponding to the expected execution object in the target scenario.

[0110] Then, the spatial localization model generates and outputs a 3D mask for each target execution object based on the 3D point cloud features corresponding to the 3D point cloud data. The 3D mask can accurately identify the specific spatial location of the expected execution object involved in each subtask in the 3D space corresponding to the target scene, thereby achieving precise alignment between the task execution plan (language description) and the target scene.

[0111] In summary, the final output can include a task execution plan generated in text format (containing action descriptions of each subtask, expected execution objects, and execution order), time optimization explanation text (used to explain the rationality and parallelism basis of the generated schedule), and the three-dimensional position of the target object (3D bounding box or point cloud mask) predicted by the 3D positioning head.

[0112] The above methods can synergistically enhance the task planning efficiency of the task localization model and the three-dimensional precise localization capability of the spatial localization model, enabling accurate prediction of task attributes and generating parallel task scheduling strategies with optimized execution time. This reduces waiting time and resource waste during task execution, effectively improving the overall planning efficiency and decision-making quality of the embodied agent in complex environments. Simultaneously, it achieves deep integration of task planning and 3D object localization, significantly improving the target recognition and localization accuracy of the embodied agent in complex three-dimensional environments.

[0113] Table 1 is a performance comparison table of task scheduling in the present disclosure and related technologies provided by an exemplary embodiment.

[0114] Table 1 Table 2 is a comparative table of the 3D positioning accuracy of the present disclosure solution and related technologies provided in an exemplary embodiment.

[0115] Table 2 Clearly, as shown in Table 1-2, the proposed solution can effectively shorten the task execution time of the embodied intelligent agent, improve execution efficiency, and at the same time achieve precise positioning of objects in 3D space, thereby enhancing the operational stability and reliability of the embodied intelligent agent.

[0116] Corresponding to the aforementioned embodiments of the task scheduling strategy generation method, this disclosure also provides embodiments of the task scheduling strategy generation apparatus.

[0117] Please see Figure 5 , Figure 5 This is a schematic block diagram of a task scheduling policy generation apparatus provided in an exemplary embodiment. The apparatus may include: an acquisition unit 501 and a policy generation unit 502. The acquisition unit 501 is configured to acquire task description information of the task to be executed by the embodied intelligent agent and scene information of the target scene; wherein, the task description information includes the expected execution time of at least two sub-tasks, and the target scene is the operation scene in which the embodied intelligent agent performs the task.

[0118] The strategy generation unit 502 is configured to generate a task scheduling strategy for an embodied intelligent agent to perform tasks in the target scene based on the task description information and the scene information. The task scheduling strategy includes the execution order of the at least two sub-tasks, wherein the execution time intervals of at least some sub-tasks overlap with the execution time intervals of other sub-tasks.

[0119] In one embodiment, the execution time interval of at least some subtasks overlaps with the execution time interval of other subtasks, including: the execution time interval of at least some subtasks is within the execution time interval of another subtask.

[0120] In one embodiment, the strategy generation unit 502 is specifically used to: input the scene information and the task description information into the task localization model to obtain the parallelism attribute of each of the at least two subtasks, wherein the parallelism attribute indicates whether the subtask can be executed in parallel with other subtasks; and generate the task scheduling strategy based on the parallelism attribute of each subtask, the expected execution time in the task description information and the inter-task dependency relationship.

[0121] In one embodiment, the strategy generation unit 502 is specifically used for: the task localization model determining the parallelism attribute of each subtask based on the task description information and the scenario information, and encoding the parallelism attribute of each subtask, the expected execution time in the task description information, and the inter-task dependency relationship into a planning token for output; the planning token is used to indicate: calling the optimization solver to generate the task scheduling strategy according to the planning token; and generating the task scheduling strategy through the optimization solver based on the parallelism attribute, expected execution time, and inter-task dependency relationship contained in the planning token.

[0122] In one embodiment, the strategy generation unit 502 is specifically used to: convert the scene information into a corresponding scene token and convert the task description information into a corresponding task token; and input the scene token and the task token into the task positioning model.

[0123] In one embodiment, the policy generation unit 502 is specifically used for: the task localization model obtaining the semantic association between the scene information and the task description information through a cross-modal attention mechanism; and determining the parallelism attribute of each sub-task based on the semantic association.

[0124] In one embodiment, the device further includes: The spatial positioning unit 503 is configured to, after generating a task scheduling strategy for an embodied intelligent agent to perform a task in the target scene, input the task scheduling strategy and the task description information into the task positioning model to obtain a task execution plan corresponding to the task to be executed. The task execution plan includes: the expected execution object corresponding to each sub-task in the target scene; and, based on the task execution plan and the scene information of the target scene, determine the spatial location of the expected execution object corresponding to each sub-task in the target scene.

[0125] In one embodiment, the scene information of the target scene includes spatial perception data of the target scene and object information of foreground objects; The spatial positioning unit 503 is specifically used to: determine the similarity between the expected execution object and the foreground object based on the object information of the expected execution object and the object information of the foreground object, and determine the foreground object whose similarity meets the preset conditions as the target execution object corresponding to the expected execution object; and determine the spatial position of each target execution object in the target scene according to each target execution object and the spatial perception data.

[0126] In one embodiment, the task execution plan further includes a location token corresponding to each expected execution object; the step of calculating the similarity between each expected execution object and each foreground object based on the object information of the expected execution object and the object information of the foreground object includes: mapping the location token corresponding to each expected execution object to obtain an expected execution object vector set, which contains the feature vector corresponding to each expected execution object; calculating the similarity between each expected execution object and each foreground object based on the expected execution object vector set and the foreground object vector set; the foreground object vector set is obtained by vector transformation based on the object information of the foreground object.

[0127] In one embodiment, the device further includes: The similarity correction unit 504 is configured to, before determining the foreground object with the highest similarity as the target execution object corresponding to the expected execution object, obtain the temperature parameter corresponding to the scene type based on the scene type of the target scene; and correct the calculated similarity based on the obtained temperature parameter.

[0128] Please see Figure 6 , Figure 6 This is a schematic block diagram of a training apparatus for a task localization model provided in an exemplary embodiment. The apparatus may include: a sample acquisition unit 601, an input unit 602, a generation unit 603, and an iterative training unit 604. The sample acquisition unit 601 is configured to: acquire sample task description information, target task scheduling strategy and sample scene information of the embodied intelligent agent sample execution task, the sample task description information includes the expected execution time of at least two sample sub-tasks and the inter-task dependency relationship, the target task scheduling strategy includes the execution order of the at least two sample sub-tasks, wherein the execution time interval of at least some sample sub-tasks overlaps with the execution time interval of other sample sub-tasks. The input unit 602 is configured to input the sample scene information and the sample task description information into a pre-trained task localization model to obtain the sample parallelism attribute of each sample subtask in the at least two sample subtasks. The sample parallelism attribute indicates whether the sample subtask can be executed in parallel with other sample subtasks. The generation unit 603 is configured to generate a sample task scheduling strategy for the embodied intelligent agent to perform tasks in the sample scenario based on the sample parallelism attribute of each sample subtask, the expected execution time in the sample task description information, and the inter-task dependency relationship. The iterative training unit 604 is configured to calculate a policy loss value based on the difference between the sample task scheduling policy and the target task scheduling policy, and to iteratively train the pre-trained task localization model based on the policy loss value.

[0129] In one embodiment, the iterative training unit 604 is specifically used to: obtain the target task execution plan corresponding to the sample execution task, wherein the target task execution plan includes the target execution object corresponding to each sample subtask in the sample scenario; input the sample task scheduling strategy and the sample task description information into the pre-trained task localization model to obtain the sample task execution plan corresponding to the sample execution task, wherein the sample task execution plan includes the predicted execution object corresponding to each sample subtask in the sample scenario; and calculate the loss value as the policy loss value based on the difference between the target task execution plan and the sample task execution plan.

[0130] In one embodiment, the device further includes a spatial positioning loss calculation unit 605, configured to: input the sample task execution plan and the sample scene information into a pre-trained spatial positioning model to obtain the predicted spatial position of the predicted execution object corresponding to each sample sub-task in the sample scene; and calculate a spatial positioning loss value based on the predicted spatial position of the predicted execution object corresponding to each sample sub-task and the target spatial position of the target execution object corresponding to the sample sub-task in the sample scene.

[0131] The iterative training unit 604 is specifically used to: iteratively train the pre-trained task localization model and the pre-trained spatial localization model together based on the policy loss value and the spatial localization loss value.

[0132] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments of the relevant methods, and will not be elaborated upon here.

[0133] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0134] Figure 7 This is a schematic block diagram illustrating an electronic device according to embodiments of the present disclosure. For example, the electronic device 700 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0135] Reference Figure 7 The electronic device 700 may include one or more of the following components: a processing component 702, a memory 704, a power supply component 706, a multimedia component 708, an audio component 710, an input / output (I / O) interface 712, a sensor component 714, and a communication component 716.

[0136] Processing component 702 typically controls the overall operation of electronic device 700, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 702 may include one or more processors 720 to execute instructions to complete all or part of the steps of the methods described above. Furthermore, processing component 702 may include one or more modules to facilitate interaction between processing component 702 and other components. For example, processing component 702 may include a multimedia module to facilitate interaction between multimedia component 708 and processing component 702.

[0137] Memory 704 is configured to store various types of data to support the operation of electronic device 700. Examples of this data include instructions for any application or method operating on electronic device 700, contact data, phonebook data, messages, pictures, videos, etc. Memory 704 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0138] Power supply component 706 provides power to various components of electronic device 700. Power supply component 706 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 700.

[0139] Multimedia component 708 includes a screen that provides an output interface between the electronic device 700 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 708 includes a front-facing camera and / or a rear-facing camera. When the electronic device 700 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0140] Audio component 710 is configured to output and / or input audio signals. For example, audio component 710 includes a microphone (MIC) configured to receive external audio signals when electronic device 700 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 704 or transmitted via communication component 716. In some embodiments, audio component 710 also includes a speaker for outputting audio signals.

[0141] I / O interface 712 provides an interface between processing component 702 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0142] Sensor assembly 714 includes one or more sensors for providing state assessments of various aspects of electronic device 700. For example, sensor assembly 714 can detect the on / off state of electronic device 700, the relative positioning of components such as the display and keypad of electronic device 700, changes in position of electronic device 700 or a component of electronic device 700, the presence or absence of user contact with electronic device 700, orientation or acceleration / deceleration of electronic device 700, and temperature changes of electronic device 700. Sensor assembly 714 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 714 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 714 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0143] Communication component 716 is configured to facilitate wired or wireless communication between electronic device 700 and other devices. Electronic device 700 can access wireless networks based on communication standards, such as WiFi, 2G or 3G, 4G LTE, 5G NR, or combinations thereof. In one exemplary embodiment, communication component 716 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 716 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0144] In an exemplary embodiment, the electronic device 700 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described in any of the above embodiments.

[0145] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 704 including instructions, which can be executed by a processor 720 of an electronic device 700 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0146] In an exemplary embodiment, this disclosure also provides a computer program product including a computer program / instructions that, when executed by a processor, implement the steps of the method as described in any of the above embodiments.

[0147] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0148] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.< / gru> < / gru> < / gru> < / sch> < / gru> < / gru> < / gru>

Claims

1. A method for generating a task scheduling strategy, characterized in that, include: Obtain task description information of the task to be performed by the embodied intelligent agent, and scene information of the target scene; wherein, the task description information includes the expected execution time of at least two sub-tasks, and the target scene is the operation scene in which the embodied intelligent agent performs the task; Based on the task description information and the scene information, a task scheduling strategy is generated for the embodied intelligent agent to perform tasks in the target scene. The task scheduling strategy includes the execution order of the at least two sub-tasks, wherein the execution time intervals of at least some sub-tasks overlap with the execution time intervals of other sub-tasks.

2. The method according to claim 1, characterized in that, The execution time intervals of at least some subtasks overlap with the execution time intervals of other subtasks, including: The execution time interval of at least some of the subtasks falls within the execution time interval of another subtask.

3. The method according to claim 1, characterized in that, The step of generating a task scheduling strategy for an embodied intelligent agent to perform tasks in the target scene based on the task description information and the scene information includes: The scene information and the task description information are input into the task localization model to obtain the parallelism attribute of each of the at least two subtasks. The parallelism attribute indicates whether the subtask can be executed in parallel with other subtasks. The task scheduling strategy is generated based on the parallelism attribute of each subtask, the expected execution time in the task description information, and the dependencies between tasks.

4. The method according to claim 3, characterized in that, The step of inputting the scene information and the task description information into the task localization model to obtain the parallelism attribute of each of the at least two subtasks includes: the task localization model determines the parallelism attribute of each subtask based on the task description information and the scene information, and encodes the parallelism attribute of each subtask, the expected execution time in the task description information, and the inter-task dependency relationship into a planning token for output; the planning token is used to indicate: calling the optimization solver to generate the task scheduling strategy according to the planning token; The step of generating the task scheduling strategy based on the parallelism attribute of each subtask, the expected execution time in the task description information, and the inter-task dependencies includes: generating the task scheduling strategy through the optimization solver based on the parallelism attribute, expected execution time, and inter-task dependencies contained in the planning token.

5. The method according to claim 3, characterized in that, The step of inputting the scene information and the task description information into the task localization model includes: The scene information is converted into a corresponding scene token, and the task description information is converted into a corresponding task token; Input the scene token and the task token into the task location model.

6. The method according to claim 3, characterized in that, The step of inputting the scene information and the task description information into the task localization model to obtain the parallelism attribute of each of the at least two sub-tasks includes: The task localization model obtains the semantic association between the scene information and the task description information through a cross-modal attention mechanism; Based on the semantic association, the parallelism attribute of each subtask is determined.

7. The method according to claim 3, characterized in that, After generating a task scheduling strategy for an embodied intelligent agent to perform tasks in the target scenario, the method further includes: The task scheduling strategy and the task description information are input into the task localization model to obtain the task execution plan corresponding to the task to be executed. The task execution plan includes: the expected execution object of each subtask in the target scene. Based on the task execution plan and the scene information of the target scene, determine the spatial location of the expected execution object corresponding to each subtask in the target scene.

8. The method according to claim 7, characterized in that, The scene information of the target scene includes the spatial perception data of the target scene and the object information of the foreground objects; The step of determining the spatial location of the expected execution object corresponding to each subtask in the target scene based on the task execution plan and the scene information of the target scene includes: Based on the object information of the expected execution object and the object information of the foreground object, the similarity between the expected execution object and the foreground object is determined, and the foreground object whose similarity meets the preset conditions is determined as the target execution object corresponding to the expected execution object. Based on each target execution object and the spatial perception data, the spatial position of each target execution object in the target scene is determined.

9. The method according to claim 8, characterized in that, The task execution plan also includes a location token corresponding to each expected execution object; the calculation of the similarity between each expected execution object and each foreground object based on the object information of the expected execution object and the object information of the foreground object includes: Map the location token corresponding to each expected execution object to obtain a set of expected execution object vectors, which contains the feature vectors corresponding to each expected execution object. Similarity is calculated based on the expected execution object vector set and the foreground object vector set to obtain the similarity between each expected execution object and each foreground object; the foreground object vector set is obtained by vector transformation based on the object information of the foreground object.

10. The method according to claim 8, characterized in that, Before determining the foreground object with the highest similarity as the target execution object corresponding to the expected execution object, the method further includes: Based on the scene type of the target scene, obtain the temperature parameters corresponding to the scene type; The calculated similarity is corrected based on the obtained temperature parameters.

11. A training method for a task localization model, characterized in that, include: The sample task description information, target task scheduling strategy, and sample scenario information of the embodied intelligent agent sample execution task are obtained. The sample task description information includes the expected execution time of at least two sample sub-tasks and the inter-task dependencies. The target task scheduling strategy includes the execution order of the at least two sample sub-tasks. The execution time intervals of at least some sample sub-tasks overlap with the execution time intervals of other sample sub-tasks. The sample scene information and the sample task description information are input into the pre-trained task localization model to obtain the sample parallelism attribute of each sample subtask in the at least two sample subtasks. The sample parallelism attribute indicates whether the sample subtask can be executed in parallel with other sample subtasks. Based on the sample parallelism attribute of each sample subtask, the expected execution time in the sample task description information, and the inter-task dependencies, a sample task scheduling strategy is generated for the embodied intelligent agent to execute tasks in the sample scenario. Based on the difference between the sample task scheduling strategy and the target task scheduling strategy, a strategy loss value is calculated, and the pre-trained task localization model is iteratively trained based on the strategy loss value.

12. The method according to claim 11, characterized in that, The step of calculating the policy loss value based on the difference between the sample task scheduling policy and the target task scheduling policy includes: Obtain the target task execution plan corresponding to the sample execution task, wherein the target task execution plan contains the target execution object corresponding to each sample subtask in the sample scenario; The sample task scheduling strategy and the sample task description information are input into the pre-trained task localization model to obtain the sample task execution plan corresponding to the sample execution task. The sample task execution plan includes the predicted execution object of each sample subtask in the sample scenario. The loss value is calculated as the strategy loss value based on the difference between the target task execution plan and the sample task execution plan.

13. The method according to claim 12, characterized in that, The method further includes: The sample task execution plan and the sample scene information are input into a pre-trained spatial localization model to obtain the predicted spatial location of the predicted execution object in the sample scene for each sample sub-task. Calculate the spatial positioning loss value based on the predicted spatial location of the predicted execution object corresponding to each sample subtask and the target spatial location of the target execution object corresponding to the sample subtask in the sample scene. The step of iteratively training the pre-trained task localization model based on the policy loss value includes: iteratively training the pre-trained task localization model and the pre-trained spatial localization model together based on the policy loss value and the spatial localization loss value.

14. A device for generating a task scheduling strategy, characterized in that, include: The acquisition unit is used to acquire task description information of the task to be executed by the embodied intelligent agent and scene information of the target scene; wherein, the task description information includes the expected execution time of at least two sub-tasks, and the target scene is the operation scene in which the embodied intelligent agent performs the task; The strategy generation unit is used to generate a task scheduling strategy for the embodied intelligent agent to perform tasks in the target scene based on the task description information and the scene information. The task scheduling strategy includes the execution order of the at least two sub-tasks, wherein the execution time intervals of at least some sub-tasks overlap with the execution time intervals of other sub-tasks.

15. The apparatus according to claim 14, characterized in that, The strategy generation unit is specifically used for: The scene information and the task description information are input into the task localization model to obtain the parallelism attribute of each of the at least two subtasks. The parallelism attribute indicates whether the subtask can be executed in parallel with other subtasks. The task scheduling strategy is generated based on the parallelism attribute of each subtask, the expected execution time in the task description information, and the dependencies between tasks.

16. The apparatus according to claim 15, characterized in that, The device further includes: a spatial positioning unit, used for: After generating a task scheduling strategy for the embodied intelligent agent to perform tasks in the target scene, the task scheduling strategy and the task description information are input into the task localization model to obtain the task execution plan corresponding to the task to be executed. The task execution plan includes: the expected execution object of each subtask in the target scene. Based on the task execution plan and the scene information of the target scene, determine the spatial location of the expected execution object corresponding to each subtask in the target scene.

17. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor implements the method as described in any one of claims 1 to 13 by executing the executable instructions.

18. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1 to 13.

19. A computer program product, characterized in that, Includes a computer program / instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1 to 13.

Images