Method and apparatus for task planning

Abstract

The application provides a task planning method and device. The method comprises the following steps: acquiring scene information of a first real scene collected by a sensor; generating a first digital scene according to the scene information of the first real scene; performing task planning on a first task according to the scene information of the first digital scene to obtain one or more target subtasks of the first task, wherein the scene information of the first digital scene is determined based on an orthogonal projection of the first digital scene on a first plane; and outputting the one or more target subtasks of the first task. The scheme of the embodiment of the application is beneficial to guarantee the reliability of the task planning result, thereby guaranteeing the successful execution of the task.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a method and apparatus for task planning. Background Technology

[0002] Embodied tasks refer to tasks accomplished by intelligent agents (such as robots or virtual agents) through interaction with physical or virtual environments. In embodied tasks, the real-world environment is typically complex and diverse. During the execution of an embodied task, the agent needs to accurately perceive environmental information and plan tasks and actions based on that information. Environmental information is usually acquired from the relative or absolute viewpoints of fixed sensors. Consequently, embodied systems are highly sensitive to changes in sensor viewpoints, which can affect the execution of the embodied task.

[0003] Therefore, ensuring the successful execution of the task has become an urgent problem to be solved. Summary of the Invention

[0004] This application provides a method and apparatus for task planning, which helps to ensure the reliability of task planning results, thereby ensuring the successful execution of the task.

[0005] Firstly, a task planning method is provided, including: acquiring scene information of a first real scene collected by sensors;

[0006] A first digital scene is generated based on the scene information of the first real scene; a task is planned for the first task based on the scene information of the first digital scene to obtain one or more target sub-tasks of the first task, wherein the scene information of the first digital scene is determined based on the orthogonal projection of the first digital scene onto a first plane; one or more target sub-tasks of the first task are output.

[0007] According to the scheme of the embodiments of this application, a digital scene is obtained by digitizing a real scene, and then task planning is performed based on the scene information of the digital scene under a unified perspective (i.e., the scene information determined by the orthogonal projection of the digital scene on the same plane). This helps to avoid the influence of the sensor's perspective on task planning, thereby helping to ensure the reliability and correctness of the task planning results, and thus ensuring the successful execution of the task.

[0008] The first task is the embodied task to be completed.

[0009] Scene information in a digital scenario can be determined based on a unified perspective. In other words, for different digital scenarios, their scene information is obtained by mapping the digital scenario onto the same plane.

[0010] In conjunction with the first aspect, in some implementations of the first aspect, generating a first digital scene based on scene information of a first real scene includes: obtaining object instances and poses of one or more objects in the first real scene based on scene information of the first real scene; and generating a first digital scene based on object instances and poses of one or more objects.

[0011] According to the solution of the embodiments of this application, a digital scene can be generated based on the object instance and pose of the object, that is, the generation of a digital scene centered on the object can be realized. This helps to avoid the adverse effects of background and lighting factors in the real scene on the task planning results, thereby helping to ensure the reliability and correctness of the task planning results, and thus ensuring the successful execution of the task.

[0012] In conjunction with the first aspect, in some implementations of the first aspect, the first plane is a horizontal plane.

[0013] In conjunction with the first aspect, in some implementations of the first aspect, the scene information of the first digital scene includes an image of the orthogonal projection of the first digital scene onto the first plane, and / or descriptive information of the orthogonal projection of the first digital scene onto the first plane.

[0014] In the scheme of this application embodiment, the input information for task planning may include an image of the orthogonal projection of the first digital scene onto the first plane and descriptive information of the orthogonal projection of the first digital scene onto the first plane, which is beneficial to improve processing efficiency while ensuring the successful execution of the task.

[0015] In conjunction with the first aspect, in some implementations of the first aspect, task planning is performed on the first task based on the scenario information of the first digital scenario to obtain one or more target sub-tasks of the first task, including: performing task planning on the first task based on the scenario information of the first digital scenario to generate the first sub-task; and performing scenario editing on the first digital scenario to obtain a second digital scenario, wherein the second digital scenario is based on the expected execution result of the first sub-task, and one or more target sub-tasks of the first task are determined based on the scenario information of the second digital scenario.

[0016] The scene information of the second digital scene can be determined based on the orthogonal projection of the second digital scene onto the first plane.

[0017] In the solution of this application embodiment, during the execution of the task, the digital scene can be edited according to the sub-tasks, and the target sub-task can be determined according to the editing results, which helps to ensure the reliability and correctness of the task planning results and helps to ensure the successful execution of the task.

[0018] In conjunction with the first aspect, in some implementations of the first aspect, task planning is performed on the first task based on the scenario information of the first digital scenario to obtain one or more target sub-tasks of the first task, and further includes: optimizing the first sub-task based on the scenario information of the second digital scenario to obtain a second sub-task, wherein one or more target sub-tasks of the first task are further optimized based on the second sub-task.

[0019] In the solution of this application embodiment, during the execution of the task, the digital scene can be edited according to the sub-tasks, and the sub-tasks can be optimized according to the editing results, which helps to ensure the reliability and correctness of the task planning results and helps to ensure the successful execution of the task.

[0020] In conjunction with the first aspect, in some implementations of the first aspect, task planning is performed on the first task based on the scenario information of the first digital scenario to obtain one or more target sub-tasks of the first task. It also includes: task planning is performed on the first task based on the scenario information of the second digital scenario to obtain a third sub-task. The third sub-task is a sub-task executed after the first sub-task. The one or more target sub-tasks of the first task are further determined based on the third sub-task.

[0021] In the solution of this application embodiment, during the execution of the task, the digital scene can be edited according to the sub-tasks, and the sub-tasks can be deduced according to the editing results to realize multi-step task planning. This helps to avoid cumulative errors, thereby further ensuring the reliability and correctness of the task planning results and ensuring the successful execution of the task.

[0022] In conjunction with the first aspect, in some implementations of the first aspect, task planning is performed on the first task based on the scenario information of the first digital scenario to obtain one or more target sub-tasks of the first task, including: parallel processing of task planning of the first task based on the scenario information of the first digital scenario through multiple task instances to obtain one or more target sub-tasks of the first task.

[0023] In the scheme of this application embodiment, the optimization and / or deduction of subtasks can be performed in parallel, which is beneficial to improving the efficiency of task planning.

[0024] In conjunction with the first aspect, in some implementations of the first aspect, the scene information of the first real scene includes: image data of the first real scene, and / or, point cloud data of the first real scene.

[0025] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: acquiring scene information of a second real scene collected by a sensor; generating a third digital scene based on the scene information of the second real scene; performing task planning on the second task based on the scene information of the third digital scene to obtain one or more target sub-tasks of the second task, wherein the scene information of the third digital scene is determined based on the orthogonal projection of the third digital scene onto a first plane; and outputting one or more target sub-tasks of the second task.

[0026] Taking the example of the first and second tasks having the same content, i.e., executing the same embodied task multiple times, the task planning in both tasks is based on scene information determined by the orthogonal projection of the digital scene onto the same plane.

[0027] Taking the difference in task content between the first and second tasks as an example, that is, performing different embodied tasks. In the execution of both tasks, task planning is carried out based on scene information determined by the orthogonal projection of the digital scene on the same plane.

[0028] In a second aspect, a task planning apparatus is provided, comprising modules / units for performing the methods of the first aspect and any implementation thereof.

[0029] It should be understood that the extensions, limitations, explanations and descriptions of the relevant content in the first aspect above also apply to the same content in the second aspect.

[0030] Thirdly, an electronic device is provided. The electronic device includes a processor configured to be coupled to a memory, read and execute instructions and / or program code in the memory to perform the methods described in the first aspect or any possible implementation thereof.

[0031] Fourthly, a chip system is provided. The chip system includes logic circuitry for coupling with an input / output interface through which data is transmitted to perform the methods described in the first aspect or any possible implementation thereof.

[0032] Fifthly, a computer program product containing instructions is provided, which, when executed by a computing device, cause the computing device to perform the method as described in the first aspect or any implementation thereof.

[0033] In a sixth aspect, a computer-readable storage medium is provided, including computer program instructions that, when executed by a computing device, perform the method as described in the first aspect or any implementation thereof.

[0034] As examples, these computer-readable storage devices include, but are not limited to, one or more of the following: read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), flash memory, electrically EPROM (EEPROM), and hard drive.

[0035] Alternatively, as one implementation method, the aforementioned storage medium can specifically be a non-volatile storage medium. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of an embodied hierarchical system.

[0037] Figure 2 This is a schematic diagram of a system architecture applicable to embodiments of this application.

[0038] Figure 3 This is a schematic flowchart of a task planning method according to an embodiment of this application.

[0039] Figure 4 This is a schematic diagram of another task planning system according to an embodiment of this application.

[0040] Figure 5 This is a schematic diagram of a task planning process according to an embodiment of this application.

[0041] Figure 6 This is a schematic diagram of another task planning process according to an embodiment of this application.

[0042] Figure 7 This is a schematic diagram of another task planning process according to an embodiment of this application.

[0043] Figure 8 This is a schematic block diagram of a task planning device according to an embodiment of this application.

[0044] Figure 9 This is a schematic diagram of the architecture of a computing device according to an embodiment of this application. Detailed Implementation

[0045] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0046] The terminology used in the following embodiments is for the purpose of describing specific embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” and “the” are intended to include expressions such as “one or more,” unless the context clearly indicates otherwise. It should also be understood that in the following embodiments of this application, “at least one,” “at least one,” and “one or more” refer to one, two, or more than two. “First,” “second,” and various numerical designations are merely distinctions for descriptive convenience and are not intended to limit the scope of the embodiments of this application. “And / or” is used to describe the correspondence between corresponding objects, indicating that three relationships can exist. For example, “A and / or B” can represent: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character “ / ” generally indicates that the preceding and following related objects are in an “or” relationship. The order of the process numbers below does not imply the order of execution. The execution order of each process should be determined by its function and internal logic and should not constitute any limitation on the implementation process of the embodiments of this application. For example, in the embodiments of this application, the words "301", "401", "501" etc. are merely identifiers made for the convenience of description and do not limit the order of execution steps.

[0047] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. In this application, the words "exemplary" or "for example" are used to indicate that something is illustrative, exemplary, or descriptive. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the words "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner. The terms "comprising," "including," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized. In the embodiments of this application, descriptions such as "when," "in the case of," "if," and "if" all refer to the fact that the device will perform a corresponding processing under certain objective circumstances, and are not a limitation on time, nor do they require the device to perform a judgment action during implementation, nor do they imply any other limitations.

[0048] In this application, "for indicating" can include both direct and indirect indication. When describing an indication message as indicating A, it can include whether the indication message directly indicates A or indirectly indicates A, but does not necessarily mean that the indication message carries A.

[0049] To facilitate understanding of the solutions in the embodiments of this application, the terms that may be involved in the embodiments of this application will be explained below.

[0050] (1) Embodied task;

[0051] Embodied tasks refer to a series of tasks that an intelligent agent (such as a robot or virtual agent) accomplishes by interacting with a physical or virtual environment, such as by observing, moving, speaking, and interacting with the world.

[0052] Examples of embodied tasks include, but are not limited to:

[0053] Embodied navigation: An agent needs to find a path from one point to another in the environment. In other words, the agent moves to a specified location in the environment.

[0054] Embodied question answering: Intelligent agents need to explore and answer questions in a specific environment, which usually involves understanding the environment and language processing capabilities.

[0055] Robot manipulation: Intelligent agents need to physically interact with objects, such as grasping, moving, or assembling them. For example, opening a drawer, picking up items, or assembling parts.

[0056] Embodied tasks can be applied to a variety of fields. For example, embodied tasks can be applied to the following fields.

[0057] Robots: Home service, industrial automation.

[0058] Virtual Reality: Virtual Agents, Game AI.

[0059] Autonomous driving: Navigation and decision-making of vehicles in complex environments.

[0060] (2) Bird's-eye view (BEV);

[0061] BEV (Browser-oriented View) is a perspective commonly used in mapping, video surveillance, and autonomous driving, referring to a top-down, vertically directed view. Typically, a BEV provides a comprehensive and clear view of the scene. In autonomous driving and robotics, BEV is often used to represent environmental perception, enabling a better understanding of surrounding objects and road layouts.

[0062] (3) Computer-aided design (CAD) model;

[0063] A CAD model is a three-dimensional or two-dimensional object model created using computer software. CAD object models are commonly used in product design, architectural design, mechanical engineering, and other fields to help designers accurately model, modify, and analyze objects on a computer.

[0064] (4) Simulation to reality (Sim2Real);

[0065] Sim2Real is the process of transforming a simulated environment (such as virtual simulation or computer simulation) into the real world, and it is commonly used in fields such as robotics, autonomous driving, and reinforcement learning. The conditions for testing in a simulated environment may differ from those in the real world, and Sim2Real needs to overcome these differences so that the model can be effectively applied in the real world.

[0066] (5) Reality to simulation (Real2Sim);

[0067] Real2Sim is the process of transferring real-world data or models into a simulation environment. The goal of Real2Sim is to accurately reproduce real-world objects, scenes, or behaviors in a simulation environment. Real2Sim is commonly used to train robots, test new algorithms, or conduct simulations, enabling experiments to be performed in a safe and controlled environment.

[0068] (6) Two-dimensional (2D) detection and 2D segmentation;

[0069] 2D detection can be used to identify the location of objects in a two-dimensional image (usually represented by bounding boxes). 2D segmentation, on the other hand, labels each pixel in a two-dimensional image as belonging to a category of different objects or background.

[0070] (7) Six degrees of freedom (6D) pose estimation;

[0071] 6D pose estimation refers to the estimation of an object's position and orientation in three-dimensional space. Specifically, 6D pose refers to a pose with six degrees of freedom, including three degrees of translation and three degrees of rotation. Translation represents the object's position in three-dimensional space, typically represented by a three-dimensional vector (x, y, z) in the world coordinate system or camera coordinate system. Rotation represents the object's orientation in three-dimensional space; for example, the object's rotation can be represented by pitch, yaw, and roll angles.

[0072] In embodied tasks, the real-world environment is typically complex and diverse. During the execution of an embodied task, the agent needs to accurately perceive environmental information and, based on this information, plan tasks and actions. This involves breaking down complex tasks into multiple simpler sub-tasks and executing those sub-tasks.

[0073] Figure 1 A schematic diagram of an embodied hierarchical system is shown. This embodied hierarchical system is used to perform embodied tasks.

[0074] like Figure 1 As shown, the embodied hierarchical system includes an embodied brain, an embodied cerebellum, and a real-world scene information module.

[0075] Real-world information modules are used to collect and process data from the real world. For example, a real-world information module might acquire real-time data through various sensors and other IoT devices.

[0076] The embodied brain is used to perform task planning, such as Figure 1 As shown, the long sequence task L is broken down into multiple subtasks l.

[0077] Specifically, the embodied brain can perform task planning based on data obtained from real-world scene information modules.

[0078] The embodied cerebellum is used to perform action planning tasks. For example, the embodied cerebellum can be used to generate specific control commands to control the execution of specific operations.

[0079] For example, the embodied cerebellum receives subtask l from the embodied brain and transforms subtask l into action control instructions based on data obtained from the real scene information module, in order to control the execution of specific actions, and the action results are returned to the real scene information module.

[0080] pass Figure 1 The system shown demonstrates how the embodied brain and embodied cerebellum work together to complete long-sequence tasks.

[0081] In current embodied tasks, environmental information is typically acquired from the relative or absolute perspective of fixed sensors. For example, Figure 1 As shown, the data acquired by both the concrete brain and the embodied cerebellum from the real-world information module are from fixed-viewpoint observations. The embodied brain and embodied cerebellum are highly sensitive to changes in the sensor's viewpoint, affecting the overall generalization ability of the system. For example, for the same task, the processing results may differ when performing the task based on environmental information from different viewpoints.

[0082] In related solutions, the overall generalization ability of the system is usually improved by generalizing the viewpoint. Specifically, the dataset is expanded to generalize to viewpoint variations by randomizing the viewpoint domain. Therefore, viewpoint generalization requires a large amount of training data. Consequently, the cost of data acquisition and model training also increases.

[0083] Furthermore, embodied tasks require closed-loop interaction with the real environment, which is typically very costly. Common solutions employ model-based methods to construct a "world model" and learn policies to execute embodied tasks. However, the connection between the "world model" and "policy learning" built using model-based methods is weak. Errors may exist at each step during execution, potentially leading to significant cumulative errors that could hinder successful task execution.

[0084] In view of this, embodiments of this application provide a task planning method that can perform embodied tasks based on environmental information from a unified perspective, which helps to avoid the influence of sensor perspective, thereby ensuring the reliability of task planning results and improving the generalization of embodied systems.

[0085] The solutions proposed in this application can be applied to various scenarios involving embodied tasks such as navigation and object manipulation. For example, the solutions proposed in this application can be applied to task planning scenarios for robot manipulation.

[0086] To better illustrate the solution of this application, the system architecture of the embodiments of this application will be described below.

[0087] Figure 2 A schematic diagram of a system architecture applicable to embodiments of this application is shown. Figure 2 The system 200 shown can be deployed on a cloud server, a local device, or both.

[0088] like Figure 2 As shown, system 200 may include: object instantiation module 210, scene instantiation module 220 and task planning module 230.

[0089] The object instantiation module 210 and the scene instantiation module 220 can be used to realize the digitization of real scenes.

[0090] Specifically, the object instantiation module 210 can be used to obtain object instances and poses of objects in the real scene based on scene information of the real scene.

[0091] The scene instantiation module 220 can be used to generate a digital scene based on the object instance and pose of an object.

[0092] Furthermore, the scene instantiation module 220 can also be used to acquire scene information of the digital scene. The scene information of the digital scene is determined based on a unified perspective. In other words, the scene information of the digital scene is determined based on the orthogonal projection of the digital scene onto the same plane.

[0093] The task planning module 230 is used to plan tasks based on the scene information of the digital scene to generate sub-tasks. The task planning module 230 can also be regarded as an embodied brain.

[0094] For example, the subtasks output by the task planning module 230 can be output to a real-world scenario. Alternatively, the subtasks can be optimized before being output to a real-world scenario.

[0095] Optionally, system 200 may also include scene editing module 240.

[0096] The scene editing module 240 can be used to edit the digital scene according to the sub-tasks output by the task planning module 230 to obtain the digital scene after scene editing.

[0097] Furthermore, the scene editing module 240 can also be used to provide the scene information of the edited digital scene to the task planning module 230. The task planning module 230 can then generate the next subtask based on the scene information of the edited digital scene, and finally output one or more target subtasks.

[0098] Optionally, system 200 may also include task optimization module 250.

[0099] The task optimization module 250 can be used to determine whether the subtasks output by the task planning module 230 need optimization. Determining whether a subtask needs optimization can also be replaced with determining whether a subtask is feasible, or other descriptions. For example, if a subtask is determined not to need optimization, it is considered feasible; if a subtask is determined to need optimization, it is considered infeasible.

[0100] Furthermore, the task optimization module 250 can also be used to optimize the subtasks output by the task planning module 230.

[0101] Furthermore, in this case, the scene editing module 240 can also edit the digital scene according to the optimized sub-tasks output by the task optimization module 250. The scene information of the digital scene after scene editing can be provided to the task planning module 230, which will then generate the next sub-task.

[0102] For example, the scene editing module 240 can edit the digital scene based on the sub-tasks output by the task planning module 230 to obtain the edited digital scene. The task optimization module 250 can determine whether the sub-tasks output by the task planning module 230 need optimization based on the scene information of the edited digital scene. Furthermore, the task optimization module 250 can optimize the sub-tasks output by the task planning module 230 based on the scene information of the edited digital scene. Further, the optimized sub-tasks can be provided to the scene editing module 240, which then performs scene editing based on the optimized sub-tasks. The edited digital scene is then provided to the task planning module 230 so that the task planning module 230 can generate the next sub-task.

[0103] Optionally, system 200 may also include a task instantiation module. The task instantiation module can be used to provide parallelization services.

[0104] In system 200, some or all modules can be implemented by calling tools. For example, the object instantiation module 210, the scene instantiation module 220, the scene editing module 240, and the task instantiation module can be implemented by calling the object instantiation tool, the scene instantiation tool, the scene editing tool, and the task instantiation tool, respectively.

[0105] For a detailed description of the above process, please refer to method 300 below.

[0106] It should be understood that Figure 2 The system shown is merely a schematic diagram of a system architecture provided in this application embodiment. The quantity and positional relationship of the devices, components, modules, etc. shown in the figure do not constitute any limitation. For example, Figure 2 The task planning module 230 and the task optimization module 250 are two independent modules. In other possible implementations, the task optimization module 250 can also be located in the task planning module 230. In this case, the task planning module 230 can be used to optimize sub-tasks based on scene information of the digital scene after scene editing, and / or generate the next sub-task, and finally output one or more sub-tasks.

[0107] Figure 3 A schematic flowchart illustrating a task planning method according to an embodiment of this application is shown. Figure 3 The method 300 shown can be executed by a cloud server, a local device, or a combination of both.

[0108] For example, method 300 can be derived from Figure 2 The task planning system 200 in the application executes the task. For ease of description, the solution in the embodiment of this application is mainly composed of... Figure 2The method 300 is described using system 200 as an example, and does not limit the execution subject of the embodiments of this application.

[0109] like Figure 3 As shown, method 300 may include the following steps.

[0110] 310. Obtain scene information of real scene #1 (i.e., the first real scene) collected by the sensor.

[0111] 320. Generate digital scene #1 (i.e., the first digital scene) based on the scene information of real scene #1.

[0112] 330. Based on the scenario information of digital scenario #1, perform task planning for task #1 (i.e., the first task) to obtain one or more target sub-tasks of task #1.

[0113] The scene information of digital scene #1 can be determined based on the orthogonal projection of digital scene #1 onto plane #1 (i.e., the first plane).

[0114] Task #1 is the specific task to be processed. For example, Task #1 can be a long-order task, meaning that method 300 can be used to implement task planning for long-order tasks.

[0115] Furthermore, method 300 may also include: receiving task description #1, which is used to indicate task #1.

[0116] For example, task #1 may be instructed by a user. For instance, task description #1 may be determined based on user instructions. For example, the user's instructions may be the Chinese instruction "Set up a formal dinner table" or the English instruction "setaformaldinnertable". In this case, method 300 may further include: receiving the user's instruction #1, which instructs task #1.

[0117] The numbers "Task #1", "Task Description #1", and "Instruction #1" are for descriptive convenience only and have no limiting effect. Any specific task to be processed can be considered Task #1. Any instruction used to indicate Task #1 can be considered Instruction #1. Any task description used to indicate Task #1 can be considered Task Description #1.

[0118] The real scene can also be replaced with other descriptive methods such as real scene, real environment or real environment.

[0119] Real-world scenario #1 refers to the scenario in which task #1 takes place.

[0120] The designation "Real Scenario #1" is for descriptive convenience only and has no limiting effect. The actual execution environment of Task #1 can be considered Real Scenario #1.

[0121] For example, real-world scenario #1 can be the initial scenario or the current scenario of task #1. The scenario information of the current scenario can be understood as the scenario information of the real-world scenario acquired in real time during the processing of task #1.

[0122] Scene information from real-world scenarios can also be replaced with descriptions such as observational information from real-world scenarios.

[0123] As one possible implementation, scene information of the real scene can include visual information of the real scene, such as image data, and / or point cloud data.

[0124] Image data may include one or more of the following: color images or depth maps, etc.

[0125] Optionally, the scene information of real scene #1 may include image data of real scene #1 and / or point cloud data of real scene #1.

[0126] For example, the sensor may include any one or more of the following: a camera or a lidar, etc.

[0127] Scene information in a real-world scenario may also include information other than visual information, and this application embodiment does not limit this. For ease of description, this application embodiment mainly uses visual information as an example for illustration.

[0128] Digital scenes can also be replaced with other descriptive methods such as virtual scenes, simulated scenes, or simulation scenes.

[0129] Digital scene #1 is the digital scene corresponding to real scene #1, or in other words, digital scene #1 is a digital scene used to simulate real scene #1. Step 320 is the real-to-sim process of digitizing the real scene.

[0130] The designation "Digital Scenario #1" is for descriptive convenience only and has no limiting effect. The digital scenario corresponding to the real scenario #1 is called Digital Scenario #1.

[0131] The following is an example of the generation process of digital scenario #1.

[0132] Alternatively, step 320 can be implemented by the following steps (not shown in the figure).

[0133] 321. Obtain the object instance and pose of one or more objects in real scene #1 based on the scene information of real scene #1.

[0134] 322, Generate a digital scene #1 based on the object instances and poses of one or more objects.

[0135] For example, step 321 can be performed by Figure 2 The object instantiation module 210 in the middle is executed.

[0136] Object instances and poses can also be collectively referred to as object instantiation information. That is, step 321 enables the acquisition of object-centric object instantiation information. The process of acquiring object instantiation information can also be called the object instantiation process.

[0137] Taking the scene information of real scene #1, which includes the visual information of real scene #1 (such as image data or point cloud data), as an example, one or more objects in real scene #1 can also be understood as one or more objects in the visual information of real scene #1.

[0138] One or more objects in the visual information of real scene #1 can be some or all of the objects in the visual information of real scene #1.

[0139] The pose of an object can be determined using a computer vision (CV) model.

[0140] Taking the scene information of real scene #1, including the image data of real scene #1, as an example, in step 321, the pose estimation of the image data of real scene #1 can be performed by the CV model to obtain the pose of the object in the image data.

[0141] For example, the pose of an object can be the 6D pose of the object.

[0142] For example, the pose of an object can be determined by... Figure 4 The 6D pose estimation tool was determined.

[0143] Taking the scene information of real scene #1, including the point cloud data of real scene #1, as an example, in step 321, the point cloud data of real scene #1 can be subjected to three-dimensional (3D) detection or 3D segmentation by a CV model to obtain the pose of the objects in the point cloud data.

[0144] The object instance can also be replaced with a description such as the object's virtual model, CAD model, or 3D model.

[0145] An object instance can be obtained in several ways.

[0146] Taking the scene information of real scene #1, including the image data of real scene #1, as an example, in step 321, the image data of real scene #1 can be 2D detected or 2D segmented by a CV model to obtain the attribute information of objects in the image data; and the object instance is obtained based on the attribute information of the object.

[0147] For example, 2D detection can be performed by Figure 4 The 2D inspection tool in the program is executed.

[0148] For example, 2D segmentation can be performed by Figure 4 The 2D segmentation tool in the program is executed.

[0149] For example, an object's attribute information may include at least one of the following: the object's category, the object's size, or the object's shape.

[0150] For example, obtaining an object instance based on an object's attribute information may include searching for a matching CAD model from an object asset library based on the object's attribute information to obtain the object instance. The object asset library can be used to store CAD models of multiple objects. For example, the object asset library could be... Figure 4 The object asset library in the system.

[0151] For example, obtaining an object instance based on the object's attribute information may include generating an object instance based on the object's attribute information.

[0152] Taking the scene information of real scene #1, including the point cloud data of real scene #1, as an example, in step 321, the point cloud data of real scene #1 can be 3D detected or 3D segmented by a CV model; based on the 3D detection result or 3D segmentation result, the point cloud data is converted into the object mesh of the object through 3D reconstruction, thereby obtaining the object instance of the object.

[0153] For example, 3D detection can be performed by Figure 4 The 3D inspection tool in the program is executed.

[0154] For example, 3D segmentation can be performed by Figure 4 The 3D segmentation tool in the program is executed.

[0155] For example, an object mesh can be composed of Figure 4 The 3D reconstruction tools in the software are derived from 3D detection results or 3D segmentation results.

[0156] Object instances can also be obtained in other ways, and this application does not limit this method.

[0157] For example, step 322 can be performed by Figure 2 The scene instantiation module 220 in the middle is executed.

[0158] Step 322 enables the generation of an object-centric digital scene. Digital scene #1 is an object-centric digital scene. Object-centric digital scenes focus more on the attributes, categories, and states of individual objects, while scene-centric digital scenes emphasize the understanding of the overall scene, such as the overall layout of the environment.

[0159] For example, step 322 may include: generating a digital scene #1 based on the background, object instances of the one or more objects, and their poses.

[0160] For example, step 322 can be performed by Figure 2 The scene instantiation module 220 is executed. For example, the background can be generated from... Figure 4 The background texture is obtained from the background texture library in the scene instantiation tool shown.

[0161] For example, through Figure 4 The rendering engine in the scene instantiation tool shown renders the digitized scene #1 based on the background, the object instances of the one or more objects, and their poses.

[0162] It should be understood that steps 321 and 322 are only one possible way to generate a digital scene, and the process of step 320 can also be implemented by other real2sim solutions. This application embodiment does not limit this.

[0163] In the solution of this application embodiment, a digital scene can be generated based on the object instance and pose of the object, that is, the generation of a digital scene centered on the object is realized. This helps to avoid the adverse effects of background and lighting factors in the real scene on the task planning results, thereby helping to ensure the reliability and correctness of the task planning results, and thus ensuring the successful execution of the task.

[0164] The scene information of a digital scene can also be replaced with the digital representation of the digital scene.

[0165] For example, the scene information of digital scene #1 can be obtained through... Figure 2 The scene instantiation module 220 obtained it.

[0166] For ease of description, in the embodiments of this application, the scene information of the digital scene is determined based on the orthogonal projection of the digital scene onto a certain plane. Alternatively, it can be described as the scene information of the digital scene being determined based on a certain viewpoint. This plane is the projection plane of that viewpoint.

[0167] Taking the scene information of digital scene #1 as an example, the scene information of digital scene #1 is determined based on the orthogonal projection of digital scene #1 onto plane #1. It can also be described as the scene information of digital scene #1 being obtained based on viewpoint #1. Plane #1 and viewpoint #1 are corresponding.

[0168] The designations of plane #1 and viewpoint #1 are for ease of description only and do not constitute a limitation on the embodiments of this application. Any plane can be regarded as plane #1, and correspondingly, any viewpoint can be regarded as viewpoint #1, as long as the plane used for projection remains consistent when acquiring scene information of the digital scene, that is, the viewpoint remains consistent.

[0169] Optionally, plane #1 is a horizontal plane. That is, viewpoint #1 can be BEV.

[0170] Plane #1 can also be other planes, and correspondingly, viewing angle #1 can also be other viewing angles; this application embodiment does not limit this. For example, viewing angle #1 can be set according to task requirements. For instance, for embodied tasks involving horizontal manipulation, plane #1 can be a horizontal plane, i.e., viewing angle #1 can be a BEV (Browser-Electronic View). As another example, for embodied tasks involving vertical manipulation, plane #1 can be a vertical plane, and viewing angle #1 can also be a side view or a front view, etc.

[0171] Optionally, the scene information of digital scene #1 may include an image of the orthogonal projection of digital scene #1 onto plane #1, and / or descriptive information of the orthogonal projection of digital scene #1 onto plane #1.

[0172] In other words, the scene information of digital scene #1 may include an image of digital scene #1 based on viewpoint #1, and / or descriptive information of digital scene #1 based on viewpoint #1.

[0173] The image of the digital scene #1 based on viewpoint #1 can also be replaced with observation information or observation from viewpoint #1.

[0174] Taking viewpoint #1 as BEV as an example, the image of digital scene #1 based on viewpoint #1 can be the BEV map of digital scene #1, or in other words, the BEV observation of digital scene #1.

[0175] For example, the descriptive information of the orthogonal projection of the digital scene #1 onto plane #1 can be a textual description of the image of the orthogonal projection of the digital scene #1 onto plane #1.

[0176] For example, the descriptive information of the orthogonal projection of digital scene #1 onto plane #1 can be used to indicate at least one of the following: the center point of one or more objects in digital scene #1, or the bounding box of one or more objects in digital scene #1.

[0177] One or more objects in digital scene #1 can also be understood as one or more objects in the image of digital scene #1 orthogonally projected onto plane #1.

[0178] For example, the scene information of digital scene #1 may include a multimodal description based on perspective #1, such as a BEV diagram of digital scene #1 and a text description based on the BEV perspective.

[0179] In the scheme of this application embodiment, the input information for task planning may include an image of the orthogonal projection of the digital scene #1 onto plane #1 (i.e., an image of the digital scene #1 based on viewpoint #1) and descriptive information of the orthogonal projection of the digital scene #1 onto plane #1 (i.e., descriptive information of the image of the digital scene #1 based on viewpoint #1). This is beneficial for improving processing efficiency while ensuring the successful execution of the task. Specifically, the descriptive information helps reduce the processing difficulty of task planning, thereby improving processing efficiency; the image can provide richer information, thereby helping to obtain better task planning results.

[0180] In method 300, scene information of a digital scene can be obtained based on a unified perspective (i.e., perspective #1).

[0181] A unified perspective can be understood as follows: During the execution of an embodied task, different digital scenarios may be involved. The scenario information of these different digital scenarios is determined based on the same perspective, for example, based on the BEV perspective. Alternatively, for any digital scenario, its scenario information is obtained by mapping the digital scenario onto the same plane. For example, for any digital scenario, its scenario information is obtained by projecting the digital scenario onto the same plane using orthogonal projection.

[0182] As an example scenario, during the execution of an embodied task (such as task #1), multiple digital scenes may be involved. The scene information of these digital scenes is determined based on the same perspective, for example, based on the BEV perspective. That is, the scene information of these digital scenes is determined based on the orthogonal projection of these digital scenes onto the same plane. Specific examples can be found in the description below.

[0183] As another example, during multiple executions of the same embodied task, multiple digital scenes may be involved, and the scene information of these digital scenes is determined based on the same perspective. That is, the scene information of these digital scenes is determined based on the orthogonal projection of these digital scenes onto the same plane.

[0184] As another example, the execution of different embodied tasks may involve multiple digital scenes, and the scene information of these digital scenes is determined based on the same perspective. That is, the scene information of these digital scenes is determined based on the orthogonal projection of these digital scenes onto the same plane.

[0185] Optionally, method 300 may also include the following steps (not shown in the figure).

[0186] 340. Obtain scene information of real scene #2 (i.e., the second real scene) collected by the sensor.

[0187] 350. Generate digital scene #2 (i.e., the third digital scene) based on the scene information of real scene #2.

[0188] 360. Based on the scene information of digital scene #2, perform task planning for task #2 (i.e., the second task) to obtain one or more target sub-tasks of task #2.

[0189] The scene information of digital scene #2 is determined based on the orthogonal projection of digital scene #2 onto plane #1. Alternatively, it can be understood that the scene information of digital scene #2 is obtained based on viewpoint #1.

[0190] Task #2 is the embodied task to be processed.

[0191] The designation "Task #2" is for descriptive convenience only and has no limiting effect. For a detailed description of Task #2, please refer to Task #1; it will not be repeated here.

[0192] Real-world scenario #2 is the scenario in which task #2 takes place.

[0193] The designation "Real Scenario #2" is for descriptive convenience only and has no limiting effect. The actual execution environment of Task #2 can be considered Real Scenario #2. For a detailed description of Real Scenario #2, please refer to Real Scenario #1.

[0194] The designation "Digital Scene #2" is for descriptive convenience only and has no limiting effect. The digital scene corresponding to the real scene #2 is designated as Digital Scene #2. For a detailed description of Digital Scene #2 and its scene information, please refer to the description of Digital Scene #1; it will not be repeated here.

[0195] The tasks of Task #1 and Task #2 can be the same or different.

[0196] Real-world scenario #1 and real-world scenario #2 can be the same scenario or different scenarios.

[0197] The scene information for real scene #1 is different from that for real scene #2. For example, real scene #1 and real scene #2 can be the same scene, but the scene information for real scene #1 and real scene #2 are obtained from different perspectives.

[0198] Taking Task #1 and Task #2 as examples where the task content is the same, that is, the same embodied task is executed multiple times. In the execution of these two tasks, task planning is based on scene information of the digital scene obtained from the same perspective (such as perspective #1).

[0199] Taking the different task content of Task #1 and Task #2 as an example, that is, executing different embodied tasks. In the execution of these two tasks, task planning is based on scene information of the digital scene obtained from the same perspective (such as perspective #1).

[0200] It should be understood that the embodiments of this application only use one projection plane (i.e., one viewpoint) as an example for illustration, and do not limit the number of projection planes (i.e., viewpoints) in the embodiments of this application. In other implementations, the number of projection planes (or viewpoints) can also be multiple. For example, the projection plane may include a horizontal plane and a vertical plane, and the unified viewpoint may be the BEV and side viewpoints. For any digital scene, its scene information is obtained based on the BEV and side viewpoints. For example, for any digital scene, its BEV diagram and side view are obtained as its scene information.

[0201] Step 330 will be explained below.

[0202] Subtasks can also be replaced with descriptions such as plans.

[0203] For example, task planning can be achieved through a task planning model. The task planning model can be a fine-tuned, spatially geometrically aligned, multimodal large model, such as a vision language model (VLM).

[0204] A spatially oriented, geometry-centered multimodal large model refers to a multimodal large model whose inputs and outputs both include geometric text data. The geometric text data is used to describe geometric locations.

[0205] For example, the scene information of digital scene #1 is input into VLM, which performs task planning and outputs subtasks. The descriptive information of digital scene #1 based on viewpoint #1 is geometric text data. The geometric text data in the output of VLM can be used to describe the target position of the target object in the subtask.

[0206] For example, step 330 can be performed by Figure 2 The task planning module 230 is executed. For example, a task planning model can be deployed in the task planning module 230.

[0207] The task planning module 230 can also be replaced with other descriptions such as embodied brain or embodied task planning module.

[0208] Furthermore, method 300 may also include step 370.

[0209] 370, Output one or more target subtasks of task #1.

[0210] Method 300 can be used for single-step task planning. For example, during the execution of task #1, a subtask is output each time, that is, in step 370, a target subtask is output.

[0211] Alternatively, method 300 can also be used for multi-step task planning, for example, during the execution of task #1, outputting multiple subtasks at once, such as outputting a sequence of subtasks (i.e., multiple target subtasks).

[0212] These multiple subtasks can be all subtasks of task #1, or only some subtasks of task #1.

[0213] In step 370, the one or more target subtasks can be output to the real-world scene, or in other words, output to the embodied cerebellum, so that the execution results of the one or more target subtasks affect the real-world scene. The embodied cerebellum can be used to implement action planning.

[0214] In the solution of this application embodiment, a digital scene is obtained by digitizing a real scene, and then task planning is performed based on the scene information of the digital scene obtained from a unified perspective (i.e., scene information determined based on the orthogonal projection of the digital scene on the same plane). This helps to avoid the influence of the sensor's perspective on task planning, thereby helping to ensure the reliability and correctness of the task planning results, and thus ensuring the successful execution of the task.

[0215] Meanwhile, in the solution of this application embodiment, task planning is performed based on scene information of the digital scene obtained from a unified perspective. This helps to avoid the influence of the sensor's perspective on task planning, and helps to ensure the successful execution of tasks in any scene, thereby improving the generalization of the embodied system. The generalization of the embodied system, that is, the generalization ability of the embodied system, refers to the ability of the intelligent agent to effectively perform tasks in new environments, new tasks, or new objects without special training.

[0216] For example, the solution of this application helps to avoid a variety of problems caused by different camera perspectives, such as changes in object size, object shadows, and background changes, thereby improving the generalization of the embodied system.

[0217] Meanwhile, the solution in this application reduces the reliance on data required for generalization of perspectives, thereby improving the generalization of embodied systems without increasing costs (such as the cost of data acquisition and model training).

[0218] Furthermore, subtasks can be optimized before outputting them.

[0219] Specifically, the digital scene can be edited based on the sub-tasks generated by the task planning, and the scene editing results can be used to determine whether the sub-tasks need to be optimized and how to optimize them.

[0220] Alternatively, step 330 can be implemented by the following steps (not shown in the figure).

[0221] 331. Based on the scenario information of digital scenario #4, perform task planning for task #1 to generate subtask #1 (i.e., the first subtask). Digital scenario #4 can be digital scenario #1, or it can be based on digital scenario #1.

[0222] 332. Perform scene editing on digital scene #4 to obtain digital scene #3 (i.e., the second digital scene), which is based on the expected execution result of subtask #1.

[0223] One or more target subtasks of Task #1 are determined based on scenario information of Digital Scenario #3.

[0224] For example, step 331 can be performed by Figure 2 The task planning module 230 is executed.

[0225] For example, step 332 can be performed by Figure 2 The scene editing module 240 is executed.

[0226] The description of a digital scene can be replaced with other descriptions such as imaginative editing of the digital scene.

[0227] For example, scene editing can be done through Figure 4 The scene editing tools shown implement object editing and physics engine tools. By calling these tools, closed-loop interaction of object instances in the digital scene is achieved, thus completing the imagined editing.

[0228] Step 332 can also be understood as executing subtask #1 in digital scenario #4 to obtain digital scenario #3. Alternatively, editing digital scenario #4 based on subtask #1 to obtain digital scenario #3. Or, editing digital scenario #4 based on the expected execution result of subtask #1 to obtain digital scenario #3. The scenario editing process can also be understood as the simulation execution process of subtasks.

[0229] For example, the expected execution result of a subtask can be the target position of the target object in the subtask. The target position of the target object can be its position on a two-dimensional plane or its position in three-dimensional space. The two-dimensional plane can be the projection plane corresponding to viewpoint #1, i.e., plane #1. The three-dimensional space can be the space where the digital scene is located. In digital scene #3, the position of the target object can be the target position of the target object.

[0230] For example, subtask #1 could be "move object 1 from position 1 to position 2". Object 1 is the target object, and position 1 can be the target position of subtask #1. In digital scene #4, the position of object 1 is position 1. In digital scene #3, the position of object 1 is position 2. Executing subtask #1 in digital scene #4 can be understood as moving object 1 from position 1 to position 2 in digital scene #4, and the resulting digital scene is digital scene #3. For example, in actual implementation, the position of object 1 in digital scene #4 is edited from position 1 to position 2 through scene editing module 240 to obtain digital scene #3.

[0231] For example, the expected execution result of a subtask can be the target pose of the target object in the subtask. In digital scenario #3, the pose of the target object can be the target pose of the target object.

[0232] The only difference between "target pose" and "target position" is the addition of rotation. That is, when editing a scene, in addition to adjusting the position of the target object, you may also need to adjust the rotation angle of the target object. For other descriptions, please refer to the previous text.

[0233] The expected result of a subtask can be indicated by the subtask itself. For example, if the subtask is "move object 1 from position 1 to position 2", then the expected result can be the target position of the target object, i.e., position 2.

[0234] Alternatively, the expected execution result of a subtask can be determined based on the subtask. For example, if the subtask "moves object 1 from position 1 to position 2" is expected to result in the target 6D pose of the target object, then the target position of the target object in the subtask can be converted into the corresponding 6D pose, which is the target 6D pose of the target object.

[0235] The target object in a subtask can be one or more.

[0236] The numbering of Digital Scenario #4, Digital Scenario #3, and Subtask #1 is for descriptive convenience only and has no limiting effect. Subtask #1 can be any subtask of Task #1; for example, Subtask #1 can be the result of an intermediate step planning. The digital scenario before scene editing based on Subtask #1 can be considered Digital Scenario #4, and the digital scenario after scene editing based on Subtask #1 can be considered Digital Scenario #3.

[0237] For a detailed description of the scenario information for Digital Scenario #4, please refer to the scenario information for Digital Scenario #1. It will not be repeated here.

[0238] The following is an example of digital scenario #4.

[0239] In some implementations, digital scenario #4 can be digital scenario #1.

[0240] For example, method 300 can be used for single-step task planning. In this case, digitization scenario #4 can be digitization scenario #1.

[0241] For example, subtask #1 can be the first subtask of task #1, that is, the first subtask to be executed. In this case, digitization scenario #4 can be digitization scenario #1.

[0242] For example, subtask #1 can be any subtask other than the first subtask, and digitization scene #3 can be obtained by editing digitization scene #4 based on the expected execution results of multiple subtasks. These multiple subtasks include subtask #1 and the first subtask of task #1. In this case, digitization scene #4 can be digitization scene #1. In other words, a new digitization scene can be obtained by editing a digitization scene based on multiple subtasks.

[0243] In some implementations, digital scene #4 can be obtained based on digital scene #1. Specifically, digital scene #4 can be obtained by performing one or more scene edits on digital scene #1.

[0244] For example, method 300 can be used for multi-step task planning. For instance, after generating a new subtask, the current digitization scene (such as digitization scene #4) can be edited based on the subtask to obtain a new digitization scene. If the subtask is not the first subtask of task #1, digitization scene #4 can be a digitization scene obtained by performing one or more scene edits on digitization scene #1.

[0245] After obtaining the digital scenario #3, one or more target subtasks of task #1 can be determined based on the scenario information of the digital scenario #3.

[0246] Specifically, the scene information of digital scene #3 can be determined based on the orthogonal projection of digital scene #3 onto plane #1. Alternatively, the scene information of digital scene #3 can be obtained based on viewpoint #1.

[0247] As mentioned earlier, during the execution of Task #1, multiple digital scenarios may be involved (such as Digital Scenario #1 and Digital Scenario #3), and the scenario information of these digital scenarios can be determined based on the same perspective.

[0248] For a detailed description of the scenario information for Digital Scenario #3, please refer to the scenario information for Digital Scenario #1. It will not be repeated here.

[0249] The following describes one or more target subtasks of task #1.

[0250] Optionally, step 330 may also include: determining, based on the scenario information of the digital scenario #3, that the one or more target subtasks include subtask #1.

[0251] The scenario information of digital scenario #3 can be used to determine whether subtask #1 needs optimization, or in other words, whether subtask #1 is feasible. For example, if the scenario information of digital scenario #3 determines that subtask #1 does not need optimization, i.e., subtask #1 is feasible, then the one or more target subtasks may include subtask #1. If the scenario information of digital scenario #3 determines that subtask #1 is not feasible, then the one or more target subtasks do not include subtask #1.

[0252] For example, tools such as task constraint libraries and / or task discriminators can be used to determine whether a subtask needs optimization.

[0253] The task constraint library includes descriptions of one or more task constraints. Taking a desktop control scenario as an example, task constraints may include: each subtask must conform to physical consistency, meaning there cannot be unreasonable collisions between objects. Whether there is a collision between objects can be determined by whether their bounding boxes overlap. For example, if the scene information of digital scene #3 determines that the bounding boxes of two objects in digital scene #3 overlap, then subtask #1 is not feasible.

[0254] Task discriminators can be used to determine whether a subtask needs optimization. For example, a task discriminator can determine whether a subtask needs optimization based on task constraints in a task constraint library. Alternatively, a task discriminator can also determine whether a subtask needs optimization based on a neural network model, such as a large language model (LLM). Furthermore, a task discriminator can also determine whether a subtask needs optimization based on both task constraints and LLM.

[0255] As a possible example, when planning task #1, the task planning model can output multiple candidate subtasks for the same step. These multiple candidate subtasks include subtask #1. Based on these multiple candidate subtasks, the digitization scenario #4 is edited separately. Based on the edited digitization scenario, a feasible candidate subtask (e.g., subtask #1) is selected from these multiple candidate subtasks, that is, it is determined that the one or more target subtasks include the candidate subtask.

[0256] Multiple candidate subtasks for the same step refer to multiple candidate subtasks that are in the same position in the subtask sequence. For example, suppose a task is broken down into multiple subtasks, meaning that the task requires multiple steps to complete, then the subtask sequence is the sequence of these multiple subtasks arranged in the order of execution.

[0257] Taking the first step as an example, the task planning model performs task planning for task #1, first outputting the sub-tasks of the first step. For instance, the task planning model can output multiple sub-tasks for the first step of task #1 based on the scene information of digital scene #1. These multiple sub-tasks are the multiple candidate sub-tasks for the first step. The digital scene #1 is then edited based on these multiple candidate sub-tasks, resulting in multiple edited digital scenes. Based on the edited digital scenes, feasible candidate sub-tasks are selected from these multiple candidate sub-tasks, and the selected candidate sub-tasks can be used as sub-task #1.

[0258] Optionally, step 330 may further include: optimizing subtask #1 based on the scenario information of digital scenario #3 to obtain subtask #2 (i.e., the second subtask). The one or more target subtasks are determined based on subtask #2.

[0259] In other words, the scenario information of digital scenario #3 can be used to determine the optimized subtask #1.

[0260] For example, subtasks can be optimized using tools such as optimization algorithm libraries. The optimization algorithm library includes one or more optimization algorithms. These algorithms can be used to optimize subtasks.

[0261] Taking the desktop control scenario as an example, if it is determined from the scenario information of digital scenario #3 that the target object of subtask #1 in digital scenario #3 overlaps with the bounding boxes of other objects, then subtask #1 is adjusted so that the target object of subtask #1 does not overlap with the bounding boxes of other objects. The adjusted subtask #1 is subtask #2.

[0262] As an example, the one or more target subtasks may include subtask #2.

[0263] As one possible implementation, in method 300, subtasks can be deduced based on the scene editing results, thereby optimizing the subtask sequence.

[0264] Furthermore, the optimization and derivation of subtasks can be combined. For example, if a subtask is determined to be feasible based on the scene editing results, the next (i.e., the next step) subtask of that subtask can be generated based on the scene editing results.

[0265] Optionally, step 330 may further include: performing task planning on task #1 based on the scenario information of digital scenario #3 to obtain subtask #3 (i.e., the third subtask). The one or more target subtasks are determined based on subtask #3.

[0266] Subtask #3 is a subtask that is executed after subtask #1.

[0267] In other words, the scene information of digital scene #3 can be used to generate the next subtask.

[0268] As mentioned earlier, in some implementations, the scenario information of digital scenario #3 determines that the one or more target subtasks include subtask #1, i.e., subtask #1 is feasible. Furthermore, the scenario information of digital scenario #3 can be used to generate the next subtask.

[0269] For example, task planning can be performed on task #1 based on the scenario information of digital scenario #1 and the scenario information of digital scenario #3 to obtain subtask #3.

[0270] As an example, the one or more target subtasks may include subtask #3.

[0271] As another example, step 332 can be repeated, where subtask #3 can be subtask #1 in the next execution of step 332, and digital scenario #3 can be digital scenario #4 in the next execution of step 332.

[0272] In other words, after generating the next subtask, you can continue to edit the scene based on the newly generated subtask, and determine whether the newly generated subtask needs to be optimized and / or whether to continue generating the next subtask based on the editing results.

[0273] For example, if subtask #3 is deemed feasible based on the current scene editing results, then the one or more target subtasks may include subtask #3. If subtask #3 is not the final step, then the next subtask can be generated based on the current scene editing results. As another example, if subtask #3 needs optimization based on the current scene editing results, subtask #3 can be optimized to obtain an optimized subtask #3, which belongs to the one or more target subtasks.

[0274] Furthermore, as mentioned earlier, in some implementations, subtask #1 may be optimized to obtain subtask #2. In this case, the digital scene #4 can be edited based on subtask #2 to obtain digital scene #5. Step 330 may also include: performing task planning on task #1 based on the scene information of digital scene #5 to obtain subtask #4. The one or more target subtasks are determined based on subtask #4. Subtask #4 is a subtask executed after the optimized subtask #1 (i.e., subtask #2).

[0275] The numbering of Digital Scene #5 and Subtask #4 is for convenience of description only and does not constitute a limitation on the solution of the embodiments of this application. For a detailed description of Subtask #4, please refer to the relevant description of Subtask #3, and replace Digital Scene #3 with Digital Scene #5. It will not be repeated here.

[0276] Furthermore, the above description of the subtask optimization process is only an example of using the edited digital scene for the optimization of a single subtask, and does not constitute a limitation on the solutions of the embodiments of this application. For example, in other implementations, the edited digital scene can also be used for the optimization of a sequence of subtasks, and the embodiments of this application do not limit this.

[0277] In the scenario of multi-step task planning using method 300, the digital scene can be edited based on the sub-task, and the sub-task can be optimized based on the new digital scene obtained after scene editing, and / or, subsequent sub-tasks can be generated. This process is repeated to complete the optimization and deduction of the sub-tasks and obtain the target sub-task sequence of task #1. This helps to ensure the reliability and correctness of the target sub-task sequence, and thus helps to ensure the successful execution of task #1.

[0278] In the solution of this application embodiment, during the execution of the task, the digital scene can be edited according to the sub-tasks, and the sub-tasks can be optimized according to the editing results, which helps to ensure the reliability and correctness of the task planning results and helps to ensure the successful execution of the task.

[0279] In the solution of this application embodiment, during the execution of the task, the digital scene can be edited according to the sub-tasks, and the sub-tasks can be deduced according to the editing results to realize multi-step task planning. This helps to avoid cumulative errors, thereby further ensuring the reliability and correctness of the task planning results and ensuring the successful execution of the task.

[0280] Furthermore, the methods in the embodiments of this application can be executed in parallel.

[0281] Optionally, step 330 may include: performing parallel processing of task planning for task #1 based on the scenario information of digital scenario #1 through multiple task instances to obtain one or more target subtasks of task #1.

[0282] In other words, each task instance is used to plan the task of task #1 based on the scenario information of digital scenario #1, and finally obtain one or more target sub-tasks of task #1.

[0283] For example, parallel processing can be performed by the task instantiation module in system 200. For instance, each task instance can be executed by a processing unit within the task instantiation module. Multiple processing units can perform task planning in parallel. Specifically, each processing unit optimizes and extrapolates subtasks in parallel. For example, each processing unit may include a task planning module 230 and a scene editing module 240, which respectively execute steps 331 and 332 described above, and optimize the subtasks and / or continue to generate the next subtask based on the scene editing results, until one of the processing units generates a satisfactory subtask sequence as the target subtask sequence, which is then output to the real environment.

[0284] In the scheme of this application embodiment, the optimization and / or deduction of subtasks can be performed in parallel, which is beneficial to improving the efficiency of task planning.

[0285] Furthermore, in the solutions of this application embodiment, some or all of the steps can be executed by calling a tool. For example, the corresponding steps can be executed by calling any one or more of the following: an object instantiation tool, a scene instantiation tool, a scene editing tool, or a task instantiation tool, etc. The steps executed by each tool can be referred to the description of the corresponding module above, or to the following... Figure 4 or Figure 5 To avoid repetition, this will not be elaborated upon here.

[0286] In the solution of this application embodiment, task planning can be achieved by making reasonable use of tools (e.g., object instantiation tools, scene instantiation tools, scene editing tools, task instantiation tools, etc.), maximizing the use of existing tools, which is conducive to achieving plug-and-play, improving flexibility, and improving overall efficiency and applicability.

[0287] Figure 4 A schematic diagram of another task planning system according to an embodiment of this application is shown. Figure 4 The system shown can be considered as Figure 2 This is one specific implementation of the system shown. Figure 4 The system shown can be used to execute method 300. To avoid unnecessary repetition, in the description... Figure 4 The system shown has some descriptions omitted appropriately.

[0288] like Figure 4 As shown, the system may include: toolkits, a task planning module, and a task optimization module.

[0289] The toolkit includes: object instantiation tools, scene instantiation tools, scene editing tools, and task instantiation tools.

[0290] Figure 4 The object instantiation tool in the corresponding Figure 2 The object instantiation module 210 in the middle. Figure 4 The scene instantiation tool in the corresponding Figure 2 The scene instantiation module 220 in the middle. Figure 4 The task planning module in the middle corresponds to Figure 2 Task planning module 230. Figure 4 The scene editing tools in the corresponding Figure 2 Scene editing module 240.

[0291] Object instantiation tools can be used to obtain object instances and poses of objects in a real-world scene based on scene information. The scene information of the real-world scene is... Figure 4 Sensor input in the middle.

[0292] For example, such as Figure 4 As shown, object instantiation tools include any one or more of the following: 2D detection / segmentation tools, 6D pose estimation tools, object asset libraries, 3D detection / segmentation tools, or 3D reconstruction tools.

[0293] 2D detection / segmentation tools can include 2D detection tools and / or 2D segmentation tools. 2D detection tools can be used to perform 2D detection on image data to obtain attribute information of objects within the image data. 2D segmentation tools can be used to perform 2D segmentation on image data to obtain attribute information of objects within the image data.

[0294] 6D pose estimation tools can be used to estimate the pose of image data to obtain the 6D pose of objects in the image data.

[0295] An object asset library can be used to store CAD models of multiple objects.

[0296] 3D inspection / segmentation tools can include 3D inspection tools and / or 3D segmentation tools. 3D inspection tools can be used to perform 3D inspection on point cloud data. 3D segmentation tools can be used to segment point cloud data in 3D.

[0297] 3D reconstruction tools can be used to convert point cloud data into object meshes.

[0298] Scene instantiation tools can be used to generate digital scenes based on object instances and poses.

[0299] For example, such as Figure 4 As shown, scene instantiation tools include one or more of the following: background texture library or rendering engine.

[0300] Scene instantiation tools can also be used to obtain scene information of digital scenes.

[0301] Background texture libraries can be used to store backgrounds.

[0302] Rendering engines can be used to generate digital scenes.

[0303] The task planning module is used to plan tasks (e.g., long-sequence tasks in the diagram) based on scene information from the digital scenario, in order to generate subtasks. The scene information of this digital scenario is obtained based on a unified perspective (e.g., perspective #1).

[0304] For example, such as Figure 4 As shown, the task planning module can plan long-sequence tasks based on the scenario information of the digital scene to generate sub-tasks.

[0305] Scene editing tools can be used to edit digital scenes based on sub-tasks to obtain edited digital scenes.

[0306] like Figure 4 As shown, the scene editing tool is used on digital scenes.

[0307] For example, scene editing tools may include object editing and a physics engine.

[0308] For example, the scene information of the edited digital scene can be provided to the task planning module. The task planning module can also be used to generate the next subtask based on the scene information of the edited digital scene, and finally output the task planning result. The task planning result includes one or more target subtasks.

[0309] For example, the scene information of the digital scene after scene editing can be provided to the task optimization module. The task optimization module can be used to determine whether a subtask is feasible based on the scene information of the digital scene after scene editing. Furthermore, the task optimization module can also be used to optimize the subtask based on the scene information of the digital scene after scene editing. The optimized subtask can also be used for scene editing, or in other words, the optimized subtask can also be applied to the digital scene. For a detailed description, please refer to step 330 in method 300, which will not be repeated here.

[0310] The arrows between the digital and real-world scenarios indicate that the task planning results obtained from optimization and deduction based on the digital scenario can be output to the real-world scenario for execution.

[0311] Task instantiation tools can be used to provide parallelization services. Specifically, task instantiation tools can be used to enable parallel processing of task instances, so that task optimization and derivation can be performed in parallel, for example, generating subtasks in parallel and optimizing subtasks and / or generating the next subtask.

[0312] For example, a task instantiation tool may include: a task constraint library, an optimization algorithm library, and a task discriminator. The task constraint library includes descriptions of one or more task constraints. The optimization algorithm library includes one or more optimization algorithms.

[0313] For example, such as Figure 4 As shown, task derivation and optimization can be accomplished by calling the task instantiation tool.

[0314] Figure 5 A schematic diagram of a task planning process according to an embodiment of this application is shown. Figure 5 The process shown can be regarded as Figure 3 An example of the method shown. Figure 5 The scheme shown can be derived from Figure 2 or Figure 4 The system execution is shown. To avoid unnecessary repetition, in the description... Figure 5 The flowchart shown has omitted some descriptions appropriately. The following is in conjunction with... Figure 4 The system shown is for Figure 5 The process shown is explained below. It should be understood that... Figure 5 The illustration uses the application of embodied tasks to robot control as an example only and does not constitute a limitation on the solutions of the embodiments of this application. For example, the solutions of the embodiments of this application can also be applied to other embodied tasks such as navigation.

[0315] like Figure 5 As shown, the process includes the following steps.

[0316] S1, acquire scene information of the real scene collected by the sensor.

[0317] like Figure 5 As shown, the scene information of the real scene collected by the sensor includes the scene information of the initial scene of the task, that is, the initial observation information.

[0318] The above is just an example. Step S1 corresponds to step 310 in method 300. The relevant description of step S1 can be found in step 310. The initial observation information can be used as an example of the scene information of real scene #1.

[0319] S2 uses an object instantiation tool to obtain object instances and object poses from scene information in the real scene.

[0320] like Figure 5 As shown, the pose of an object can be its 6D pose.

[0321] For example, by calling Figure 4 The 6D pose estimation tool in the software obtains the 6D pose of the object from the initial observation information.

[0322] For example, by calling Figure 4 The 2D detection / segmentation tool in the system obtains 2D detection or 2D segmentation results from the initial observation information, and retrieves the corresponding CAD model of the object from the object asset library based on the 2D detection or 2D segmentation results, and uses it as the object instance.

[0323] The above is just an example. Step S2 corresponds to step 321 in method 300. The relevant description of step S2 can be found in step 321.

[0324] S3 uses a scene instantiation tool to obtain scene information of a digital scene based on the object instance and the object's pose.

[0325] like Figure 5As shown, the scene information of a digital scene can include the BEV diagram of the digital scene and the scene description information. The scene description information can be obtained based on the BEV diagram, and can also be called the description information of the BEV diagram, i.e. Figure 5 The scene description in the text. For example, such as... Figure 5 As shown, scene descriptions can be used to describe the position and size of various objects in a BEV diagram. For example, in the diagram, “'knife_0':[0.216,0.527,0.269,0.700]” represents the position and size of the knife. Here, “0.216,0.527,0.269,0.700” represent the minimum value of the horizontal coordinate, the minimum value of the vertical coordinate, the maximum value of the horizontal coordinate, and the maximum value of the vertical coordinate of the image pixel corresponding to the knife in the image pixel space, respectively.

[0326] For example, by calling Figure 4 The rendering engine generates a digital scene based on the background texture library, object instances, and object poses, and obtains scene information of the digital scene.

[0327] The above is just an example. Step S3 corresponds to step 322 in method 300. The relevant description of step S3 can be found in step 322. The scene information of the digital scene in step S3 can be used as an example of the scene information of digital scene #1.

[0328] S4, through the task planning module, performs task planning based on the scenario information of the digital scene.

[0329] For example, the task can be instructed by user commands. For instance, such as... Figure 5 As shown, the user command can be "setaformaldinnertable".

[0330] For example, the task planning module can output the target geometric position of the target object in the subtask, that is, the position of the target object on the two-dimensional plane.

[0331] like Figure 5As shown, the system response of the task planning module is the output subtask, namely the plan in the diagram: "Pick up the plate from [0.452, 0.567, 0.547, 0.694] and place it on the target space [0.464, 0.719, 0.559, 0.846]". [0.464, 0.719, 0.559, 0.846] represents the target's geometric location. The plate is the target object.

[0332] The above is just an example. Step S4 corresponds to step 330 in method 300. This subtask can be regarded as an example of subtask #1. The relevant description of step S4 can be found in step 330.

[0333] S5 uses a scene editing tool to edit digital scenes based on sub-tasks output by the task planning module.

[0334] For example, assuming the task planning module outputs the target geometric position of the target object in the subtask, the scene editing tool can convert the target geometric position of the target object into the target 6D pose of the target object, and based on the target 6D pose, call the object editing tool and the physics engine tool to perform imaginative editing in the digital scene so that the pose of the target object in the edited digital scene is the target 6D pose.

[0335] Furthermore, scene information of the digital scene after scene editing can be provided through scene editing tools.

[0336] like Figure 5 As shown, the scene editing tool can output the BEV diagram and scene description information of the digital scene after scene editing.

[0337] The above is merely an example. Step S5 corresponds to step 332 in method 300, and the relevant description of step S5 can be found in step 332. This subtask can be considered as an example of subtask #1, the digitized scene before editing can be considered as an example of digitized scene #4, and the digitized scene after editing can be considered as an example of digitized scene #3.

[0338] like Figure 5As shown, the scene information of the edited digital scene can be provided to the task planning module again. Based on the scene information of the edited digital scene, the task planning module generates the next subtask, realizes task deduction, and obtains multi-step planning results (i.e., multiple target subtasks). The multi-step planning results can be output to the real environment for execution, thereby realizing a synchronous closed loop between the digital scene and the real scene.

[0339] In addition, Figure 5 The proposed solution also allows for optimization of the subtasks output by the task planning module. For a detailed description of subtask optimization, please refer to [reference needed]. Figure 3 or Figure 6 The proposed solution is shown below. Further details will not be provided here.

[0340] Figure 6 The diagram illustrates two task planning processes according to embodiments of this application. Figure 6 The two schemes shown can be regarded as Figure 3 The diagram shows two possible implementations of method 300. For specific implementation details, please refer to the description of method 300. To avoid repetition, the description will only briefly mention... Figure 6 The description of the scheme shown is appropriately omitted.

[0341] like Figure 6 As shown in (a), the task planning process may include the following steps.

[0342] S1-1, Digitalization of Real-World Scenarios.

[0343] In step S1-1, a digital scene a is generated based on the scene information of the real scene collected by the sensor.

[0344] S1-2, Task Planning.

[0345] For example, the scene information of the digital scene a obtained in S1-1 is input into the task planning model for task planning, resulting in multiple candidate subtasks for the first step, such as subtask a, subtask b and subtask c in the figure.

[0346] These multiple candidate subtasks can be output all at once or through multiple inferences.

[0347] S1-3, Scene Editing.

[0348] In steps S1-3, the digital scene a is edited according to the multiple candidate sub-tasks obtained in step S1-2.

[0349] S1-4, Task Optimization.

[0350] Based on the scene information of the digital scene after scene editing, feasible subtasks are determined from the multiple candidate subtasks obtained in steps S1-2.

[0351] Scene editing and task optimization based on multiple candidate subtasks can be performed in parallel. For example, scene editing of digital scene a can be performed simultaneously based on subtask a, subtask b, and subtask c, resulting in three edited digital scenes. The feasibility of subtask a, subtask b, and subtask c can then be determined based on these three digital scenes.

[0352] Alternatively, scene editing and task optimization based on these multiple candidate subtasks can be performed sequentially. In this case, scene editing and task optimization can be performed based on all candidate subtasks, or only on a portion of the candidate subtasks. For example, scene editing and task optimization can be performed sequentially based on each candidate subtask until a feasible subtask is determined.

[0353] After executing step S1-4, you can return to execute step S1-2.

[0354] In this case, in step S1-2, task planning can continue based on the feasible subtasks determined in step S1-4 to output multiple candidate subtasks for the next step. For example, the scene information of the digital scene after scene editing can be input into the task planning model for task planning to obtain multiple candidate subtasks for the second step. Assuming the feasible subtask is subtask b, the digital scene after scene editing is obtained by editing digital scene a based on subtask b.

[0355] Steps S1-2 to S1-4 are executed iteratively until the task planning result is obtained. This task planning result is a multi-step planning result (i.e., multiple target sub-tasks). Multi-step optimization is implemented during this task planning process. The multi-step planning result is then output to the real environment for execution, thereby achieving a synchronous closed loop between the digital scenario and the real scenario.

[0356] It should be understood that after executing steps S1-4, the single-step planning result can also be used as the task planning result (this option is not shown in the figure). This single-step planning result represents the feasible sub-tasks determined in steps S1-4. Single-step optimization is implemented during this task planning process. The single-step planning result is then output to the real-world environment for execution, thereby achieving a synchronous closed loop between the digital and real-world scenarios.

[0357] like Figure 6 As shown in (b), the task planning process may include the following steps.

[0358] S2-1, Digitalization of Real-World Scenarios.

[0359] In step S2-1, a digital scene a is generated based on the scene information of the real scene collected by the sensor.

[0360] S2-2, Task Planning.

[0361] For example, the scene information of the digital scene a obtained in S2-1 is input into the task planning model for task planning, and the sub-task d of the first step is obtained.

[0362] S2-3, Scene Editing.

[0363] In step S2-3, the digital scene a is edited according to the sub-task obtained in step S2-2.

[0364] S2-4, Task Judgment.

[0365] Based on the scene information of the digital scene after scene editing, determine whether the subtask output in step S2-2 needs optimization.

[0366] If no optimization is needed, after executing step S2-4, you can return to executing step S2-2.

[0367] In this case, in step S2-2, task planning can continue based on subtask d to output the subtask for the next step. For example, the scene information of the digital scene after scene editing can be input into the task planning model for task planning to obtain the subtask for the second step.

[0368] If optimization is needed, proceed to step S2-5.

[0369] S2-5, Task Optimization.

[0370] Optimize subtask d to obtain the optimized subtask, such as subtask e.

[0371] After executing step S2-5, return to execute step S2-2.

[0372] In this case, in step S2-2, task planning can continue based on subtask e to output the subtask for the next step. For example, the scene information of the digital scene after scene editing can be input into the task planning model for task planning to obtain the subtask for the second step. The digital scene after scene editing can be obtained by editing digital scene a based on subtask e.

[0373] Steps S2-2 to S2-5 are executed repeatedly until the task planning result is obtained. This task planning result is a multi-step planning result (i.e., multiple target sub-tasks). Multi-step optimization is implemented during this task planning process. The multi-step planning result is then output to the real environment for execution, thereby achieving a synchronous closed loop between the digital scenario and the real scenario.

[0374] It should be understood that after executing steps S2-4 (and S2-5), the single-step planning result can also be used as the task planning result (this option is not shown in the figure). This single-step planning result is either a feasible subtask determined in step S2-4 or an optimized subtask in step S2-5. Single-step optimization is achieved during this task planning process. The single-step planning result is then output to the real-world environment for execution, thereby achieving a synchronous closed loop between the digital and real-world scenarios.

[0375] Figure 7 A schematic diagram of another task planning process according to an embodiment of this application is shown. Figure 7 The process shown and Figure 5 The main difference in the process shown is that, Figure 7 The scheme shown implements online optimization and derivation of subtasks in parallel through task instances. For example, as... Figure 7 As shown, these multiple task instances can be deployed on a cloud server.

[0376] like Figure 7 As shown, the cloud server can receive scene information and task descriptions of the real scene collected by the sensors from the local device, and return the task planning results (such as single-step planning results or multi-step planning results) to the local device, which will then execute the task planning results.

[0377] like Figure 7 As shown, a task instance can include a task planning module and a scene editing tool. For example, Figure 7 Task instance 1 includes task planning module 1, scene editing tool 1, and task optimization module 1; task instance 2 includes task planning module 2, scene editing tool 2, and task optimization module 2. For example, the task planning modules in different task instances can deploy the same VLM. Different task planning modules can output different subtasks (such as...). Figure 7 (See Plan 1 and Plan 2 in the original text). Each task instance can execute online optimization and deduction of subtasks in parallel until the task planning result is obtained. For example, each task instance can complete online optimization and deduction of model-based subtasks, and finally output to the digital scene, and synchronously close the loop in the real scene, that is, return the task planning result obtained by each task instance to the local device, and execute it in the real scene. For example, the task planning result finally returned to the local device can be the task planning result obtained fastest among the multiple task instances.

[0378] It should be understood that Figure 7 The solutions shown are merely examples and do not constitute a limitation on the solutions implemented in this application. For example, Figure 7Multiple task instances can also be implemented through other modules, and this application embodiment does not limit this.

[0379] The solution adopted in this application can effectively improve the generalization ability of the embodied system. By using scene editing, it helps the embodied system achieve closed-loop iteration, which is conducive to ensuring the successful execution of tasks. In the functional placement task of tableware on a table (a total of 9 categories of objects, 14 types of tasks), the single-step task planning success rate of the solution adopted in this application exceeds 95%, and the multi-step task planning success rate exceeds 92%.

[0380] The following is combined with Figures 8 to 9 The apparatus of the embodiments of this application will be described below. It should be understood that the apparatus described below is capable of performing the methods of the foregoing embodiments of this application. To avoid unnecessary repetition, repeated descriptions will be appropriately omitted when describing the apparatus of the embodiments of this application below.

[0381] The system architecture of the embodiments of this application will be described below.

[0382] Figure 8 A schematic block diagram of the apparatus provided in an embodiment of this application is shown. Figure 8 The apparatus 1800 shown can be used to perform the methods of the embodiments of this application, for example, Figure 3 The method shown.

[0383] like Figure 8 As shown, the device 1800 may include: an acquisition module 1810, a processing module 1820, and an output module 1830.

[0384] For example, the device 1800 may include Figure 2 or Figure 4 The system shown. For example, the processing module 1820 in device 1800 may include Figure 2 The system 200 shown.

[0385] As one possible implementation, device 1800 can be used to perform Figure 3 Method 300 is shown.

[0386] The acquisition module 1810 is used to acquire scene information of the first real scene collected by the sensor.

[0387] The processing module 1820 is used to: generate a first digital scene based on the scene information of a first real scene; and perform task planning on a first task based on the scene information of the first digital scene to obtain one or more target sub-tasks of the first task, wherein the scene information of the first digital scene is determined based on the orthogonal projection of the first digital scene onto a first plane.

[0388] Output module 1830 is used to output one or more target subtasks of the first task.

[0389] Optionally, the processing module 1820 is specifically used to: obtain object instances and poses of one or more objects in the first real scene based on scene information of the first real scene; and generate a first digital scene based on the object instances and poses of one or more objects.

[0390] Optionally, the first plane is a horizontal plane.

[0391] Optionally, the scene information of the first digital scene includes an image of the first digital scene projected orthogonally onto the first plane, and / or descriptive information of the first digital scene projected orthogonally onto the first plane.

[0392] Optionally, the processing module 1820 is specifically used to: perform task planning for the first task based on the scene information of the first digital scene to generate a first sub-task; perform scene editing on the first digital scene to obtain a second digital scene, wherein the second digital scene is based on the expected execution result of the first sub-task, and one or more target sub-tasks of the first task are determined based on the scene information of the second digital scene.

[0393] Optionally, the processing module 1820 is further configured to: optimize the first subtask based on the scene information of the second digital scene to obtain the second subtask, wherein one or more target subtasks of the first task are further determined based on the second subtask.

[0394] Optionally, the processing module 1820 is further configured to: perform task planning on the first task based on the scene information of the second digital scene to obtain a third subtask, wherein the third subtask is a subtask executed after the first subtask, and one or more target subtasks of the first task are further determined based on the third subtask.

[0395] Optionally, the processing module 1820 is specifically used to: perform parallel processing of task planning for the first task based on the scene information of the first digital scene through multiple task instances, so as to obtain one or more target sub-tasks of the first task.

[0396] Optionally, the scene information of the first real scene includes: image data of the first real scene, and / or point cloud data of the first real scene.

[0397] Optionally, the acquisition module 1810 is further configured to acquire scene information of the second real scene collected by the sensor; the processing module 1820 is further configured to: generate a third digital scene based on the scene information of the second real scene; perform task planning on the second task based on the scene information of the third digital scene to obtain one or more target sub-tasks of the second task, wherein the scene information of the third digital scene is determined based on the orthogonal projection of the third digital scene onto the first plane; and the output module 1830 is further configured to: output one or more target sub-tasks of the second task.

[0398] For a detailed description, please refer to Method 300 above; it will not be repeated here.

[0399] Each module in device 1800 can be implemented in software or in hardware.

[0400] It should be noted that the division of units in the above device is only a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. In other embodiments, the processing module 1820 can be used to execute any step in the method described above, and other modules can be used to implement any step described above. The steps that each module is responsible for implementing can be specified as needed. By having each module implement different steps described above, all functions of the device 1800 can be achieved.

[0401] Figure 9 This is a schematic block diagram of the apparatus provided in an embodiment of this application. The apparatus 1900 may include a processor 1910, a transceiver 1920, and a memory 1930. The processor 1910, transceiver 1920, and memory 1930 are connected via internal interconnection paths. The memory 1930 is used to store instructions, and the processor 1910 is used to execute the instructions stored in the memory 1930 to receive / send data via the transceiver 1920. Optionally, the memory 1930 may be coupled to the processor 1910 via an interface or integrated with the processor 1910.

[0402] It should be noted that the transceiver 1920 mentioned above may include, but is not limited to, transceiver devices such as input / output interfaces, to enable communication between device 1900 and other devices or communication networks.

[0403] The memory 1930 can be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM).

[0404] In one implementation, the processor 1910 can be a circuit with instruction read and execute capabilities, such as a central processing unit (CPU), microprocessor, or digital signal processor (DSP). In another implementation, the processor 1910 can implement certain functions through the logical relationships of hardware circuits. These logical relationships can be fixed or reconfigurable. For example, the processor 1910 can be a hardware circuit implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), such as a field-programmable gate array (FPGA). In a reconfigurable hardware circuit, the process of the processor loading a configuration document and configuring the hardware circuit can be understood as the processor loading instructions to implement some or all of the functions of the aforementioned units.

[0405] This application also provides an electronic device, which may include the above-described device 1800 or device 1900.

[0406] This application also provides a computer program product, which includes computer program code that, when run on a computer, causes the computer to perform the methods described in the above embodiments.

[0407] This application also provides a computer-readable medium storing program code that, when run on a computer, causes the computer to perform the methods described in the above embodiments.

[0408] This application also provides a chip, including circuitry, for performing the methods described in the above embodiments.

[0409] In implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software. The method disclosed in the embodiments of this application can be directly implemented by a hardware processor, or by a combination of hardware and software modules within the processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, power-on erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are omitted here.

[0410] It should also be understood that, in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0411] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0412] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0413] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0414] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0415] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0416] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be covered.

Claims

1. A method for task planning, characterized in that, include: Acquire scene information of the first real scene collected by sensors; A first digital scene is generated based on the scene information of the first real scene; Based on the scene information of the first digital scene, task planning is performed on the first task to obtain one or more target sub-tasks of the first task, wherein the scene information of the first digital scene is determined based on the orthogonal projection of the first digital scene onto a first plane. Output one or more target subtasks of the first task.

2. The method according to claim 1, characterized in that, The step of generating the first digital scene based on the scene information of the first real scene includes: Based on the scene information of the first real scene, obtain the object instance and pose of one or more objects in the first real scene; The first digital scene is generated based on the object instances and poses of the one or more objects.

3. The method according to claim 1 or 2, characterized in that, The first plane is a horizontal plane.

4. The method according to any one of claims 1 to 3, characterized in that, The scene information of the first digital scene includes an image of the orthogonal projection of the first digital scene onto the first plane, and / or descriptive information of the orthogonal projection of the first digital scene onto the first plane.

5. The method according to any one of claims 1 to 4, characterized in that, The step of planning the first task based on the scene information of the first digital scene to obtain one or more target sub-tasks of the first task includes: Based on the scene information of the first digital scene, the first task is planned to generate the first sub-task; The first digital scene is edited to obtain a second digital scene. The second digital scene is based on the expected execution result of the first sub-task. One or more target sub-tasks of the first task are determined based on the scene information of the second digital scene.

6. The method according to claim 5, characterized in that, The step of planning the first task based on the scene information of the first digital scene to obtain one or more target sub-tasks of the first task further includes: The first subtask is optimized based on the scene information of the second digital scene to obtain the second subtask, and one or more target subtasks of the first task are further determined based on the second subtask.

7. The method according to claim 5, characterized in that, The step of planning the first task based on the scene information of the first digital scene to obtain one or more target sub-tasks of the first task further includes: Based on the scene information of the second digital scene, the first task is planned to obtain a third subtask. The third subtask is a subtask executed after the first subtask. One or more target subtasks of the first task are further determined based on the third subtask.

8. The method according to any one of claims 1 to 7, characterized in that, The step of planning the first task based on the scene information of the first digital scene to obtain one or more target sub-tasks of the first task includes: By performing parallel processing of task planning on the first task using multiple task instances based on the scene information of the first digital scene, one or more target sub-tasks of the first task can be obtained.

9. The method according to any one of claims 1 to 8, characterized in that, The scene information of the first real scene includes: image data of the first real scene, and / or point cloud data of the first real scene.

10. The method according to any one of claims 1 to 9, characterized in that, The method further includes: Acquire scene information of the second real scene collected by the sensor; A third digital scene is generated based on the scene information of the second real scene; Based on the scene information of the third digital scene, task planning is performed on the second task to obtain one or more target sub-tasks of the second task, wherein the scene information of the third digital scene is determined based on the orthogonal projection of the third digital scene onto the first plane; Output one or more target subtasks of the second task.

11. A task planning apparatus, characterized in that, include: The acquisition module is used to acquire scene information of the first real scene collected by the sensor; Processing module, used for: A first digital scene is generated based on the scene information of the first real scene; Based on the scene information of the first digital scene, task planning is performed on the first task to obtain one or more target sub-tasks of the first task, wherein the scene information of the first digital scene is determined based on the orthogonal projection of the first digital scene onto a first plane. The output module is used to output one or more target subtasks of the first task.

12. The apparatus according to claim 11, characterized in that, The processing module is specifically used for: Based on the scene information of the first real scene, obtain the object instance and pose of one or more objects in the first real scene; The first digital scene is generated based on the object instances and poses of the one or more objects.

13. The apparatus according to claim 11 or 12, characterized in that, The first plane is a horizontal plane.

14. The apparatus according to any one of claims 11 to 13, characterized in that, The scene information of the first digital scene includes an image of the orthogonal projection of the first digital scene onto the first plane, and / or descriptive information of the orthogonal projection of the first digital scene onto the first plane.

15. The apparatus according to any one of claims 11 to 14, characterized in that, The processing module is specifically used for: Based on the scene information of the first digital scene, the first task is planned to generate the first sub-task; The first digital scene is edited to obtain a second digital scene. The second digital scene is based on the expected execution result of the first sub-task. One or more target sub-tasks of the first task are determined based on the scene information of the second digital scene.

16. The apparatus according to claim 15, characterized in that, The processing module is also used for: The first subtask is optimized based on the scene information of the second digital scene to obtain the second subtask, and one or more target subtasks of the first task are further determined based on the second subtask.

17. The apparatus according to claim 15, characterized in that, The processing module is also used for: Based on the scene information of the second digital scene, the first task is planned to obtain a third subtask. The third subtask is a subtask executed after the first subtask. One or more target subtasks of the first task are further determined based on the third subtask.

18. The apparatus according to any one of claims 11 to 17, characterized in that, The processing module is specifically used for: By performing parallel processing of task planning on the first task using multiple task instances based on the scene information of the first digital scene, one or more target sub-tasks of the first task can be obtained.

19. The apparatus according to any one of claims 11 to 18, characterized in that, The scene information of the first real scene includes: image data of the first real scene, and / or point cloud data of the first real scene.

20. The apparatus according to any one of claims 11 to 19, characterized in that, The acquisition module is also used for: Acquire scene information of the second real scene collected by the sensor; The processing module is also used for: A third digital scene is generated based on the scene information of the second real scene; Based on the scene information of the third digital scene, task planning is performed on the second task to obtain one or more target sub-tasks of the second task, wherein the scene information of the third digital scene is determined based on the orthogonal projection of the third digital scene onto the first plane; and The output module is also used for: Output one or more target subtasks of the second task.

21. A computing device, characterized in that, It includes a processor and a memory, the processor being configured to execute instructions stored in the memory to cause the computing device to perform the method as described in any one of claims 1 to 10.

22. A computer program product containing instructions, characterized in that, When the instructions are executed by the computing device, the computing device performs the method as described in any one of claims 1 to 10.

23. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a computing device, cause the computing device to perform the method as described in any one of claims 1 to 10.

Images