Embodied robot, task execution method thereof, and embodied robot system

Abstract

This application discloses an embodied robot, a task execution method thereof, and an embodied robot system. The embodied robot task execution method includes: obtaining initial environmental data corresponding to a home environment; inputting the initial environmental data into a data processing model to obtain target environmental data; inputting the target environmental data into a task reasoning model to obtain a target home task; and sending the target home task to a target embodied robot, causing the target embodied robot to perform the target home task on a target object associated with the target home task. This method sequentially processes the environmental data of the home environment based on the data processing model and the task reasoning model to obtain the home task, which is beneficial for improving the accuracy of the embodied robot's execution of home tasks. Furthermore, by obtaining the target home task from the environmental data in the home environment, the embodied robot can be made more intelligent in its execution of home tasks.

Description

Technical Field

[0001] The present application belongs to the field of smart home technology, and in particular relates to an embodied robot, a task execution method thereof, and an embodied robot system. Background Art

[0002] With the rapid development of science and technology, smart home systems have become popular in people's lives. Compared with traditional homes, more and more people choose to use smart home devices in smart home systems, for example, using embodied robots to perform household tasks.

[0003] Embodied robots can be divided into many types according to their functions and application scenarios, including lock-opening robots, cleaning robots, curtain robots, companion robots, and humanoid robots.

[0004] Currently, embodied robots perform household tasks based on instructions received from users. However, when the instructions are ambiguous, the embodied robot cannot determine which household task to perform, resulting in low control accuracy. Summary of the Invention

[0005] In view of this, the embodiments of the present application provide an embodied robot, a task execution method thereof, and an embodied robot system to overcome the above problems of the prior art.

[0006] In a first aspect, an embodiment of the present application provides a task execution method of an embodied robot, comprising:

[0007] Obtaining initial environmental data corresponding to the home environment;

[0008] Inputting the initial environmental data into a data processing model to obtain target environmental data;

[0009] Inputting the target environment data into the task reasoning model to obtain the target home task;

[0010] The target home task is sent to a target embodied robot, so that the target embodied robot performs the target home task on a target object associated with the target home task.

[0011] In some optional embodiments, the model parameter quantity of the task reasoning model is greater than the model parameter quantity of the data processing model, and the parameter quantity difference between the model parameter quantity of the task reasoning model and the model parameter quantity of the data processing model is at least not less than 3 orders of magnitude.

[0012] In some optional embodiments, before sending the target home task to the target embodied robot so that the target embodied robot performs the target home task on the target object associated with the target home task, the task execution method further includes:

[0013] Acquiring a current object state of the target object and a current robot state of the target embodied robot;

[0014] Inputting the target household task, the current object state, and the current robot state into the action generation model to obtain target action information;

[0015] The sending of the target household task to the target embodied robot, so that the target embodied robot performs the target household task on a target object associated with the target household task, includes:

[0016] The target action information is sent to the target embodied robot, so that the target embodied robot performs a corresponding target action on the target object according to the target action information to perform the target household task.

[0017] Among them, in some optional embodiments, the model parameter amount of the action generation model is smaller than the model parameter amount of the task reasoning model, and the model parameter amount of the action generation model is greater than the model parameter amount of the data processing model, and the parameter difference between the model parameter amount of the action generation model and the model parameter amount of the task reasoning model is at least not less than 2 orders of magnitude.

[0018] In some optional embodiments, obtaining initial environment data corresponding to the home environment includes:

[0019] The initial environment data is acquired based on a first sensor and a second sensor, where the first sensor is installed on the target embodied robot and the second sensor is installed in the home environment.

[0020] In some optional embodiments, the target home task includes multiple subtasks, and inputting the target environment data into the task reasoning model to obtain the target home task includes:

[0021] The target environment data is input into the task reasoning model to obtain the multiple subtasks.

[0022] In some optional embodiments, the target embodied robot includes a plurality of sub-embodied robots, the target object includes a plurality of sub-objects, each subtask is associated with a sub-object, and before sending the target household task to the target embodied robot so that the target embodied robot performs the target household task on the target object associated with the target household task, the task execution method further includes:

[0023] Determining a sub-embodied robot corresponding to each subtask;

[0024] The sending of the target household task to the target embodied robot, so that the target embodied robot performs the target household task on a target object associated with the target household task, includes:

[0025] Each of the subtasks is sent to the corresponding one of the sub-embodied robots, so that each sub-embodied robot performs an associated subtask on the corresponding one of the sub-objects.

[0026] In some optional embodiments, sending the target home task to the target embodied robot so that the target embodied robot performs the target home task on a target object associated with the target home task includes:

[0027] determining a target area of ​​the target object in the home environment;

[0028] The target household task and the target area are sent to the target embodied robot, so that the target embodied robot travels to the target area and performs the target household task on the target object.

[0029] In some optional embodiments, before inputting the target environment data into the task reasoning model to obtain the target home task, the task execution method further includes:

[0030] Get initial home task instructions;

[0031] Inputting the target environment data into the task reasoning model to obtain the target home task includes:

[0032] The initial home task instruction and the target environment data are input into the task reasoning model to obtain the target home task.

[0033] In some optional embodiments, before sending the target home task to the target embodied robot so that the target embodied robot performs the target home task on the target object associated with the target home task, the task execution method further includes:

[0034] Inputting the target home task into a task simulation model so that a virtual embodied robot in the task simulation model performs the target home task in a three-dimensional model corresponding to the home environment;

[0035] determining whether the virtual embodied robot successfully performs the target household task;

[0036] The sending of the target household task to the target embodied robot, so that the target embodied robot performs the target household task on a target object associated with the target household task, includes:

[0037] When it is determined that the virtual embodied robot successfully performs the target home task, the target home task is sent to the target embodied robot, so that the target embodied robot performs the target home task on the target object associated with the target home task.

[0038] In some optional embodiments, after determining whether the virtual embodied robot successfully performs the target household task, the task execution method further includes:

[0039] When it is determined that the virtual embodied robot fails to perform the target home task, updating the target environment data according to the data of the virtual embodied robot failing to perform the target home task, the updated target environment data including the data of the virtual embodied robot failing to perform the target home task and the target environment data;

[0040] Return to executing the step of inputting the target environment data into the task reasoning model to obtain the target home task and subsequent steps, until it is determined that the virtual embodied robot successfully performs the target home task, and then send the target home task to the target embodied robot.

[0041] In some optional embodiments, after sending the target household task to the target embodied robot, the task execution method further includes:

[0042] Determining whether the current posture of the target object matches the target posture of the target object; wherein the target posture is the posture of the virtual object when the virtual embodied robot successfully performs the target household task, and the virtual object corresponds to the target object;

[0043] When the current posture matches the target posture, determining that the target embodied robot successfully performs the target home task;

[0044] When the current posture does not match the target posture, it is determined that the target embodied robot has failed to perform the target household task.

[0045] In a second aspect, an embodiment of the present application provides a task execution device of an embodied robot, comprising:

[0046] A first acquisition module is used to obtain initial environment data corresponding to the home environment;

[0047] A first input module is used to input the initial environment data into the data processing model to obtain target environment data;

[0048] A second input module is used to input the target environment data into the task reasoning model to obtain a target home task;

[0049] A sending module is used to send the target home task to a target embodied robot, so that the target embodied robot performs the target home task on a target object associated with the target home task.

[0050] In a third aspect, an embodiment of the present application provides an embodied robot, comprising:

[0051] Memory;

[0052] one or more processors coupled to the memory;

[0053] One or more applications, wherein the one or more applications are stored in the memory and are configured to be executed by the one or more processors, and the one or more applications are configured to execute the task execution method of the embodied robot provided in the first aspect above.

[0054] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which program code is stored. The program code can be called by a processor to execute the task execution method of the embodied robot provided in the first aspect above.

[0055] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when running on a computer device, enables the computer device to execute the task execution method of the embodied robot provided in the first aspect above.

[0056] In a sixth aspect, an embodiment of the present application provides an embodied robot system, comprising a sensor and a controller, wherein the sensor is connected to the controller;

[0057] The sensor is used to collect initial environmental data corresponding to the home environment;

[0058] The controller is used to execute the task execution method of the embodied robot provided in the first aspect above.

[0059] The solution provided by the present application obtains initial environmental data corresponding to the home environment, inputs the initial environmental data into a data processing model to obtain target environmental data, inputs the target environmental data into a task reasoning model to obtain a target home task, and sends the target home task to a target embodied robot, so that the target embodied robot performs the target home task on the target object associated with the target home task, thereby processing the environmental data of the home environment in sequence based on the data processing model and the task reasoning model to obtain the home task, which is beneficial to improving the accuracy of the embodied robot in performing home tasks. Moreover, by obtaining the target home task by obtaining the environmental data in the home environment, the intelligence level of the embodied robot in performing home tasks can be improved.

[0060] Furthermore, compared with the high computing resources and storage space occupied by using the traditional end-to-end large model to obtain the target home task by acquiring the environmental data in the home environment, the computing resources and storage space occupied by processing the environmental data of the home environment in sequence based on the data processing model and the task reasoning model to obtain the target home task are low, which can speed up the response time of controlling the embodied robot and help improve the control efficiency of the embodied robot. In addition, the environmental perception error of the traditional end-to-end model will be directly transmitted to the task decision layer. This application focuses on solving different problems through different models, and can also improve the accuracy of obtaining the target home task. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0062] Figure 1 A schematic diagram of a scenario of an embodied robot control system provided in an embodiment of the present application is shown.

[0063] Figure 2 A flow chart of a task execution method of an embodied robot provided in an embodiment of the present application is shown.

[0064] Figure 3 Another flowchart of the task execution method of the embodied robot provided in an embodiment of the present application is shown.

[0065] Figure 4 Another flowchart of the task execution method of the embodied robot provided in an embodiment of the present application is shown.

[0066] Figure 5Another flowchart of the task execution method of the embodied robot provided in an embodiment of the present application is shown.

[0067] Figure 6 Another flowchart of the task execution method of the embodied robot provided in an embodiment of the present application is shown.

[0068] Figure 7 A structural block diagram of a task execution device of an embodied robot provided in an embodiment of the present application is shown.

[0069] Figure 8 A functional block diagram of the embodied robot provided in an embodiment of the present application is shown.

[0070] Figure 9 A functional block diagram of the embodied robot system provided in an embodiment of the present application is shown.

[0071] Figure 10 A computer-readable storage medium provided in an embodiment of the present application is shown for storing or carrying program code for implementing a task execution method of an embodied robot provided in an embodiment of the present application.

[0072] Figure 11 A computer program product provided in an embodiment of the present application is shown for storing or carrying program codes for implementing a task execution method of an embodied robot provided in an embodiment of the present application. DETAILED DESCRIPTION

[0073] In order to make the purpose, features, and advantages of the invention of this application more obvious and easy to understand, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the embodiments described below are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0074] It will be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0075] It should also be understood that the terms used in this specification are for the purpose of describing specific embodiments only and are not intended to limit the present application. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0076] It should be further understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0077] In addition, in the description of the present application, the terms "first", "second", "third", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0078] With the rapid development of science and technology, smart home systems have become popular in people's lives. Compared with traditional homes, more and more people choose to use smart home devices in smart home systems, for example, using embodied robots to perform household tasks.

[0079] Embodied robots can be divided into many types according to their functions and application scenarios, including lock-opening robots, cleaning robots, curtain robots, companion robots, and humanoid robots.

[0080] Currently, embodied robots perform household tasks based on instructions received from users. However, when the instructions are ambiguous, the embodied robot cannot determine which household task to perform, resulting in low control accuracy.

[0081] In response to the above problems, the embodied robot and its task execution method and embodied robot system provided in the embodiments of the present application obtain initial environmental data corresponding to the home environment, input the initial environmental data into a data processing model to obtain target environmental data, input the target environmental data into a task reasoning model to obtain a target home task, and send the target home task to a target embodied robot, so that the target embodied robot performs the target home task on the target object associated with the target home task, thereby realizing the processing of the environmental data of the home environment in sequence based on the data processing model and the task reasoning model to obtain the home task, which is conducive to improving the accuracy of the embodied robot in performing home tasks. In addition, by obtaining the target home task by obtaining the environmental data in the home environment, the intelligence level of the embodied robot in performing home tasks can be improved.

[0082] Furthermore, compared with the high computing resources and storage space occupied by using the traditional end-to-end large model to obtain the target home task by acquiring the environmental data in the home environment, the computing resources and storage space occupied by processing the environmental data of the home environment in sequence based on the data processing model and the task reasoning model to obtain the target home task are low, which can speed up the response time of controlling the embodied robot and help improve the control efficiency of the embodied robot. In addition, the environmental perception error of the traditional end-to-end model will be directly transmitted to the task decision layer. This application focuses on solving different problems through different models, and can also improve the accuracy of obtaining the target home task.

[0083] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application.

[0084] See also Figure 1 , which shows a schematic diagram of an application scenario of the embodied robot control system provided in an embodiment of the present application. The embodied robot control system may include an embodied robot 100, a sensor 200, an object 300, and a control device 400, etc. The control device 400 can be connected to the embodied robot 100 and the sensor 200 through a network, and exchange data with the embodied robot 100 and the sensor 200 through the network.

[0085] The number of embodied robots 100 may be one or more, and the embodied robots 100 may be mobile home robots or fixed home robots. Mobile home robots can move around in a home environment. For example, mobile home robots may include at least one of cleaning robots (such as sweeping robots, mopping robots, and sweeping-and-mopping robots), robotic arms, companion robots, and humanoid robots, without limitation herein.

[0086] A fixed home robot is a robot that cannot move in a home environment, that is, it is fixedly installed in a home environment. For example, a fixed home robot may include at least any one of a curtain robot, a switch robot, and a lock-opening robot, etc., which is not limited here.

[0087] Sensor 200 can be used to collect environmental data of the home environment. Sensor 200 may include at least any one of a visual sensor, a temperature sensor, a humidity sensor, a light sensor, and a sound sensor. The environmental data may include at least any one of image data, video data, temperature data, humidity data, light intensity data, and semantic data.

[0088] Object 300 may include at least any one of a user, a water cup, a sofa, a refrigerator, an air conditioner, a table and chairs, a television, and a trash can, etc., which is not limited here.

[0089] The control device 400 may include any one of a terminal device and a server, etc. The type of the control device 400 is not limited here and can be specifically set according to actual needs.

[0090] The terminal device can be a mobile terminal device (for example, any one of a mobile phone, a personal digital assistant (PDA), a tablet personal computer (Tablet PC), a laptop computer, a smart watch, a smart bracelet or a wearable device), or a fixed terminal device (smart gateway device, desktop computer, smart panel, etc.), etc., and is not limited here.

[0091] The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be any one of the cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), big data or artificial intelligence platforms, etc., without limitation here.

[0092] The network may be any one of a ZigBee network, a Bluetooth (BT) network, a Wireless Fidelity (Wi-Fi) network, a Thread network, a Long Range Radio (LoRa) network, a Low-Power Wide-Area Network (LPWAN), an infrared network, a Narrow Band Internet of Things (NB-IoT), a Controller Area Network (CAN), a Digital Living Network Alliance (DLNA) network, a Wide Area Network (WAN), a Local Area Network (LAN), a Metropolitan Area Network (MAN), or a Wireless Personal Area Network (WPAN), etc., without limitation herein.

[0093] In some embodiments, there may be multiple embodied robots 100, and the sensor 200 may include a first sensor and a second sensor. The first sensor may be installed on multiple embodied robots, and the second sensor may be installed in a home environment.

[0094] There are multiple first sensors, one of which is installed on each embodied robot body. The types of the first sensors installed on the multiple embodied robot bodies can be the same or different. The first sensor and the second sensor can each include at least one of a visual sensor, a temperature sensor, a humidity sensor, a light sensor, and a sound sensor.

[0095] See also Figure 2 , which shows a flowchart of a task execution method of an embodied robot provided by an embodiment of the present application. In a specific embodiment, the task execution method of an embodied robot can be applied to the embodied robot 100, and can also be applied to Figure 1 The control device 400 in the embodied robot control system shown in FIG. 1 is used as an example to describe the control device 400. Figure 2 The process shown in FIG. 1 is described in detail. The control method of the embodied robot may include the following steps 110 to 140.

[0096] Step 110: Acquire initial environment data corresponding to the home environment.

[0097] In an embodiment of the present application, the number of embodied robots may be multiple, the sensors may include a first sensor and a second sensor, the detection range of the first sensor may be a first home area environment in a home environment, the detection range of the second sensor may be a second home area environment in a home environment, and the initial environment data may include first environment data and second environment data.

[0098] Optionally, the control device can send a collection instruction to the first sensor and the second sensor through the network. The first sensor receives and responds to the collection instruction, collects data for the first home area environment, and sends the collected first environment data to the control device through the network. The second sensor receives and responds to the collection instruction, collects data for the second home area environment, and sends the collected second environment data to the control device through the network. The control device respectively receives the first environment data returned by the first sensor and the second environment data returned by the second sensor to obtain initial environment data.

[0099] Optionally, the first sensor may collect data on the first home area environment in real time or at a preset frequency, and the second sensor may collect data on the second home area environment in real time or at a preset frequency.

[0100] In this embodiment, environmental data of the home environment is collected based on a first sensor installed on the embodied robot and a second sensor installed in the home environment. This can prevent incomplete environmental data collected by a single sensor due to the limitation of the collection range, and is conducive to improving the integrity of the environmental data.

[0101] The first environment data and the second environment data may both include at least any one of image data, video data, temperature data, humidity data, light intensity data, and semantic data.

[0102] Step 120: Input the initial environment data into the data processing model to obtain the target environment data.

[0103] In an embodiment of the present application, the control device can input initial environmental data into the data processing model, and the data processing model receives and responds to the initial environmental data and outputs target environmental data.

[0104] Optionally, the target environment data is structured data, and the data processing model can be used to process the initial environment data to obtain the target environment data. Because the initial environment data may contain noise (e.g., blurred images, invalid sensor values) and the data formats of different modalities vary significantly (e.g., image pixel matrices, audio waveforms, sensor value sequences), the data processing model is used to convert the chaotic unstructured data into structured data in a unified format.

[0105] The data processing model can be obtained by training a deep learning neural network model based on historical environmental data annotated with historical structured data. The historical structured data is obtained by processing the historical environmental data, which can include at least any one of image data, video data, temperature data, humidity data, light intensity data, and semantic data.

[0106] The deep learning neural network model can be any of the following: a convolutional neural network (CNN) model, a deep belief network (DBN) model, a stacked auto encoder network (SAE) model, a recurrent neural network (RNN) model, a deep neural network (DNN) model, a long short-term memory (LSTM) network model, or a gated recurring unit (GRU) model. The type of deep learning neural network model is not limited here and can be set according to actual needs.

[0107] Optionally, the data processing model can be used to analyze the initial environmental data (including simple evaluation, classification, filtering, or feature extraction) to obtain target environmental data for preliminary judgment or prediction of certain situations or trends. For example, the initial environmental data includes temperature, humidity, and air pressure data. The data processing model analyzes the initial environmental data to obtain target environmental data such as "80% probability of rain within 1 hour."

[0108] Optionally, the data processing model can be a target detection model (such as CNN, YOLO, SSD, etc.).

[0109] Step 130: Input the target environment data into the task reasoning model to obtain the target home task.

[0110] In an embodiment of the present application, the control device may input target environment data into a task reasoning model. The task reasoning model receives and responds to the target environment data, outputting a target home task to the control device, which then receives the target home task output by the task reasoning model. For example, the target environment data may indicate the presence of an object at a certain location on the ground. The task reasoning model may imagine that the object will block a person walking at that location, and thus output a target home task of "remove the object."

[0111] Among them, the number of target home tasks can be one or more, the task reasoning model can be obtained by training a deep learning neural network model based on historical environmental data marked with historical home tasks, and the historical home tasks can be obtained by reasoning about historical environmental data.

[0112] As an example, the deep learning neural network model can be any one of a large language model (LLM) or a multimodal visual language model (VLM). The type of deep learning neural network model is not limited here and can be set according to actual needs.

[0113] In some embodiments, the model parameter quantity of the task reasoning model may be greater than the model parameter quantity of the data processing model, and the parameter quantity difference between the model parameter quantity of the task reasoning model and the model parameter quantity of the data processing model is at least not less than 3 orders of magnitude.

[0114] Among them, the number of model parameters refers to the total number of parameters that need to be learned and adjusted in the model. The size of the model parameters will affect the capacity, complexity and computing requirements of the model.

[0115] For example, a data processing model has fewer than 100M model parameters (less than 1×10^8), making it a lightweight model. A task reasoning model has more than 500M or 1000M model parameters (at least more than 5×10^11). The order of magnitude refers to the difference in exponents expressed in scientific notation. The exponents for the two are 8 and 11, respectively, resulting in a difference of three orders of magnitude.

[0116] Since the model parameters of the data processing model are relatively small, the computational complexity is low. Based on a model parameter volume of 100M, the response time of the data processing model is less than 100ms, which can process or analyze environmental data more quickly, and is conducive to improving the efficiency of the data processing model in processing or analyzing environmental data. The model parameters of the task reasoning model are relatively large. Based on a model parameter volume of 500M or 1000M, the response time of the task reasoning model is less than 1min, which makes the task reasoning model have stronger learning and reasoning capabilities, and is conducive to improving the reasoning accuracy of the task reasoning model, so that the target home task can be inferred more accurately. Through two models with different parameter volumes, both processing efficiency and accuracy can be taken into account.

[0117] Step 140: Send the target home task to the target embodied robot, so that the target embodied robot performs the target home task on the target object associated with the target home task.

[0118] In an embodiment of the present application, the control device can send the target home task to the target embodied robot through the network. The target embodied robot receives and responds to the target home task, and performs the target home task on the target object associated with the target home task, thereby realizing the sequential processing of the environmental data of the home environment based on the data processing model and the task reasoning model to obtain the home task, which is beneficial to improving the accuracy of the embodied robot in performing home tasks. Moreover, by obtaining the target home task by acquiring the environmental data in the home environment, the intelligence level of the embodied robot in performing home tasks can be improved.

[0119] Furthermore, compared with the high computing resources and storage space occupied by using the traditional end-to-end large model to obtain the target home task by acquiring the environmental data in the home environment, the computing resources and storage space occupied by processing the environmental data of the home environment in sequence based on the data processing model and the task reasoning model to obtain the target home task are low, which can speed up the response time of controlling the embodied robot and help improve the control efficiency of the embodied robot. In addition, the environmental perception error of the traditional end-to-end model will be directly transmitted to the task decision layer. This application focuses on solving different problems through different models, and can also improve the accuracy of obtaining the target home task.

[0120] The target object may be any one of a user, a water cup, a sofa, a refrigerator, an air conditioner, a table and chairs, a television, and a trash can, etc., and is not limited here.

[0121] In some embodiments, after step 130, the control device can obtain the current object state of the target object and the current robot state of the target embodied robot, and input the target home task, the current object state and the current robot state into the action generation model. The action generation model receives and responds to the target home task, the current object state and the current robot state, and outputs the target action information to the control device. The control device receives the target action information output by the action generation model, and sends the target action information to the embodied robot through the network. The embodied robot receives and responds to the target action information, and performs the corresponding target action on the target object. Based on the action information generated by the action generation model, the embodied robot is controlled to perform the corresponding action to perform the target home task, which is conducive to further improving the control accuracy of the embodied robot.

[0122] The current object state may include the position and angle of the target object, and the current robot state may include the position and angle of the target embodied robot, which are not limited here.

[0123] The control device can obtain the current object state of the target object and the current robot state of the target embodied robot based on the initial environment data. For example, the initial environment data may include a home environment image, and the home environment image can be analyzed.

[0124] The action generation model can be a diffusion model or other deep learning model obtained by training a deep learning neural network model based on historical home information annotated with historical action information. The historical home information may include historical home tasks, historical object states, and historical robot states.

[0125] In some embodiments, the model parameter amount of the action generation model may be smaller than the model parameter amount of the task reasoning model, and the model parameter amount of the action generation model is greater than the model parameter amount of the data processing model, and the parameter difference between the model parameter amount of the action generation model and the model parameter amount of the task reasoning model is at least not less than 2 orders of magnitude.

[0126] Optionally, the difference in parameter quantities between the data processing model and the action generation model is at least one order of magnitude. Compared to using an end-to-end large model, where perception errors in environmental data are directly transmitted to the task decision layer, sequentially performing different processing based on the data processing model, action generation model, and task reasoning model is beneficial for improving the accuracy of executing target household tasks based on environmental data.

[0127] As an example, the model parameter amount of the action generation model can be 1B, 1B=1×10^9. The exponents between the action generation model and the task reasoning model are 9 and 11 respectively, so there is a difference of 2 orders of magnitude. Based on the model parameter amount of 1B, the response time of the action generation model is less than 3s.

[0128] See also Figure 3 , which shows a flowchart of a task execution method of an embodied robot provided by another embodiment of the present application. In a specific embodiment, the task execution method of an embodied robot can be applied to the embodied robot 100, and can also be applied to Figure 1 The control device 400 in the embodied robot control system shown in FIG. 1 is used as an example to describe the control device 400. Figure 3 The process shown in FIG. 1 is described in detail, and the task execution method of the embodied robot may include the following steps 210 to 250.

[0129] Step 210: Acquire initial environment data corresponding to the home environment.

[0130] Step 220: Input the initial environment data into the data processing model to obtain the target environment data.

[0131] Step 230: Input the target environment data into the task reasoning model to obtain multiple subtasks.

[0132] In this embodiment, the target home task may include multiple subtasks. The control device may input the target environment data into the task reasoning model. The task reasoning model receives and responds to the target environment data and outputs multiple subtasks to the control device. The control device receives the multiple subtasks output by the task reasoning model.

[0133] For example, if the target environment data includes the information that "the water cup in the living room has fallen over", the task reasoning model will obtain the following based on the target environment data: hold the water cup upright (task 1), wipe the table clean (task 2), and clean the floor (task 3).

[0134] For example, if the target environment data includes the information that "the laundry basket is full of clothes", the task reasoning model can obtain the following information based on the target environment data: open the washing machine door (task 1), put the clothes in the laundry basket into the washing machine (task 2), add laundry beads (task 3), close the washing machine door (task 4), and start washing and drying (task 5).

[0135] In some embodiments, after obtaining the multiple subtasks in step 230, the multiple subtasks may be sent to a target embodied robot, so that the target embodied robot performs each subtask in sequence.

[0136] In this embodiment, the task reasoning model obtains multiple subtasks based on the target environment data, so that the embodied robot can understand each subtask more clearly, thereby improving the accuracy of the embodied robot in executing the task.

[0137] In some implementations, after obtaining multiple subtasks in step 230, the following steps 240 to 250 may be performed:

[0138] Step 240: Determine a sub-embodied robot corresponding to each subtask.

[0139] In this embodiment, the target embodied robot may include multiple sub-embodied robots, the target object may include multiple sub-objects, and each subtask may be associated with a sub-object.

[0140] The control device can search the robot table according to each subtask to obtain a corresponding sub-embodied robot, and determine a sub-embodied robot corresponding to each subtask based on the correspondence between the task and the robot, thereby improving the accuracy of determining the sub-embodied robot.

[0141] The robot table can be used to represent the correspondence between tasks and embodied robots. For example, tasks can include Task 1, Task 2, Task 3, Task 4, and Task 5, and embodied robots can include Embodied Robot 1, Embodied Robot 2, Embodied Robot 3, Embodied Robot 4, and Embodied Robot 5.

[0142] The correspondence between tasks and embodied robots can be shown in Table 1, that is, the robot table. According to the correspondence, a sub-embodied robot corresponding to each subtask can be obtained.

[0143] Table 1

[0144]

[0145] It should be noted that the correspondence between tasks and embodied robots is not limited to that shown in Table 1. It can be set according to actual needs. It can also be that one sub-embodied robot corresponds to multiple sub-tasks, for example, embodied robot 1 corresponds to task 1, and embodied robot 3 corresponds to tasks 2 and 3.

[0146] Step 250: Send each subtask to a corresponding sub-embodied robot, so that each sub-embodied robot performs an associated subtask on a corresponding sub-object.

[0147] In this embodiment, the control device can send each subtask to a corresponding sub-embodied robot through the network. Each sub-embodied robot receives and responds to a subtask, and performs an associated subtask on a corresponding sub-object. This realizes the control of each embodied robot to perform one or more corresponding tasks based on the correspondence between the task and the robot, which is conducive to improving the efficiency of the overall task execution.

[0148] See also Figure 4 , which shows a flowchart of a task execution method of an embodied robot provided by another embodiment of the present application. In a specific embodiment, the task execution method of an embodied robot can be applied to the embodied robot 100, and can also be applied to Figure 1 The control device 400 in the embodied robot control system shown in FIG. 1 is used as an example to describe the control device 400. Figure 4 The process shown in FIG. 1 is described in detail, and the task execution method of the embodied robot may include the following steps 310 to 350.

[0149] Step 310: Obtain initial environment data corresponding to the home environment.

[0150] Step 320: Input the initial environment data into the data processing model to obtain the target environment data.

[0151] Step 330: Input the target environment data into the task reasoning model to obtain the target home task.

[0152] Step 340: Determine the target area of ​​the target object in the home environment.

[0153] In this embodiment, the control device can determine the target area of ​​the target object in the home environment based on a pre-built environment map corresponding to the home environment and the location of the target object in the environment map. If the environment map does not contain the target object's location information, the control device can also infer the target area of ​​the target object in the home environment based on the target object's attributes using a large language model. For example, if the target object is fruit, the target area of ​​the fruit in the home environment can be inferred to be the kitchen.

[0154] Step 350: Send the target household task and the target area to the target embodied robot, so that the target embodied robot drives to the target area and performs the target household task on the target object.

[0155] In this embodiment, the control device can send the target home task and target area to the target embodied robot. The target embodied robot receives and responds to the target home task and target area, drives to the target area, and performs the target home task on the target object. This realizes the control of the embodied robot to drive to the target area and perform the target home task on the target object based on the perception of the target area and the corresponding home task of the target object in the home environment. The embodied robot does not need to perceive the area where the target object is located in the home environment again, which is conducive to improving the control efficiency of the embodied robot.

[0156] See also Figure 5 , which shows a flowchart of a task execution method of an embodied robot provided by another embodiment of the present application. In a specific embodiment, the task execution method of an embodied robot can be applied to the embodied robot 100, and can also be applied to Figure 1 The control device 400 in the embodied robot control system shown in FIG. 1 is used as an example to describe the control device 400. Figure 5 The process shown in FIG. 1 is described in detail, and the task execution method of the embodied robot may include the following steps 410 to 450.

[0157] Step 410: Obtain initial home task instructions.

[0158] In this embodiment, the control device can obtain an initial home task instruction, which can be a voice instruction or text instruction sent by the user.

[0159] In some embodiments, the control device can be configured with a voice recognition module, and the user can send voice commands within the voice collection range of the voice recognition module. The voice recognition module collects the voice commands issued by the user, performs voice recognition on the collected voice commands, and determines the recognized voice commands as initial home task commands.

[0160] In some embodiments, the control device may be configured with an operation panel, and the user may touch the operation panel to input text instructions. The control device receives the text instructions through the operation panel and determines the text instructions as initial home task instructions.

[0161] In some embodiments, the embodied robot control system may further include a client, which is connected to the control device via a network and exchanges data with the control device via the network.

[0162] The user can send text instructions to the client, the client receives and responds to the text instructions, and forwards the text instructions to the control device through the network. The control device receives the text instructions forwarded by the client.

[0163] Step 420: Obtain initial environment data corresponding to the home environment.

[0164] Step 430: Input the initial environment data into the data processing model to obtain the target environment data.

[0165] Step 440: Input the initial home task instruction and target environment data into the task reasoning model to obtain the target home task.

[0166] In this embodiment, the control device can input the initial home task instructions and target environment data into the task reasoning model. The task reasoning model receives and responds to the initial home task and target environment data, understands the initial home task instructions, identifies the status of each object in the home environment based on the target environment data, and outputs the target home task to the control device. For example, if the initial home task instruction is to tidy up the house, and the target environment data is the image data corresponding to the home environment, the task reasoning model can output target home tasks such as straightening the pillows on the sofa, placing the trash can in the corridor into the kitchen, and throwing paper towels on the ground into the trash can based on the status of each object in the image data.

[0167] In this embodiment, the control device receives the target home task output by the task reasoning model, and infers the corresponding home task based on the initial home task instruction sent by the user and the target environment data, so that the inferred home task is more in line with the user's expectations, which is conducive to improving the intelligence level of the embodied robot in performing home tasks and enhancing the user experience.

[0168] Step 450: Send the target home task to the target embodied robot, so that the target embodied robot performs the target home task on the target object associated with the target home task.

[0169] In this embodiment, step 450 may refer to the contents of the corresponding steps in the aforementioned embodiments, which will not be repeated here.

[0170] See also Figure 6, which shows a flowchart of a task execution method of an embodied robot provided by another embodiment of the present application. In a specific embodiment, the task execution method of an embodied robot can be applied to the embodied robot 100, and can also be applied to Figure 1 The control device 400 in the embodied robot control system shown in FIG. 1 is used as an example to describe the control device 400. Figure 6 The process shown in FIG. 1 is described in detail, and the task execution method of the embodied robot may include the following steps 510 to 560.

[0171] Step 510: Acquire initial environment data corresponding to the home environment.

[0172] Step 520: Input the initial environment data into the data processing model to obtain the target environment data.

[0173] Step 530: Input the target environment data into the task reasoning model to obtain the target home task.

[0174] In this embodiment, step 510, step 520 and step 530 may refer to the contents of the corresponding steps in the aforementioned embodiment, and will not be repeated here.

[0175] Step 540: Input the target home task into the task simulation model, so that the virtual embodied robot in the task simulation model performs the target home task in the three-dimensional model corresponding to the home environment.

[0176] The task simulation model pre-builds a virtual embodied robot and a three-dimensional model corresponding to the home environment. Optionally, environmental data from the home environment can be collected in advance using sensors on the embodied robot (such as lidar, cameras, depth cameras, encoders, and IMUs). Based on this environmental data, a three-dimensional model is generated using technologies such as SLAM (Simultaneous Localization and Mapping) and VLM (Visual Localization and Mapping). Furthermore, a virtual robot is deployed within the constructed three-dimensional model to generate the task simulation model.

[0177] In this embodiment, the control device can input the target home task into the task simulation model, and the virtual embodied robot in the task simulation model receives and responds to the target home task, and performs the target home task on the virtual object in the three-dimensional model based on the virtual embodied robot, so as to simulate the process of the target embodied robot performing the target home task on the target object in the real home environment, wherein the virtual object corresponds to the target object (for example, the target object is the door of the bedroom, and correspondingly, the virtual object is the door of the bedroom in the three-dimensional model).

[0178] Step 550: Determine whether the virtual embodied robot successfully performs the target household task.

[0179] Optionally, after the virtual embodied robot completes the target household task, the control device may determine whether the virtual embodied robot successfully performs the target household task through an evaluation model.

[0180] In this embodiment, the control device can obtain a scene image after the virtual embodied robot performs the target household task. Specifically, the control device can capture a scene image of a three-dimensional model after the virtual embodied robot performs the target household task. The scene image may include a virtual object corresponding to the target object. Furthermore, an evaluation model (e.g., a VLM) compares the scene image with the expected target image and outputs a similarity. The control device then determines whether the virtual embodied robot successfully performs the target household task based on the relationship between the similarity and a similarity threshold. For example, if the similarity threshold is set to 90%, when the similarity is greater than 90%, the virtual embodied robot is determined to have successfully performed the target household task. The evaluation model pre-establishes a correspondence between the target household task and the expected target image.

[0181] Step 560: When it is determined that the virtual embodied robot successfully performs the target home task, the target home task is sent to the target embodied robot, so that the target embodied robot performs the target home task on the target object associated with the target home task.

[0182] Optionally, when the control device determines that the virtual embodied robot successfully performs the target household task on the virtual object, the current virtual object posture can be recorded as the target posture. The target posture can be used to represent the position and orientation of the virtual object in the three-dimensional model when the virtual embodied robot successfully performs the target household task.

[0183] In this embodiment, step 560 may refer to the contents of the corresponding steps in the aforementioned embodiments, which will not be repeated here.

[0184] In this embodiment, the control device can determine whether the task simulation is successful after the virtual embodied robot performs the task, so as to control the target embodied robot when the target home task is successfully executed based on the simulation results, which is conducive to further improving the accuracy of the embodied robot in performing home tasks.

[0185] In some embodiments, after step 550, when the control device determines that the virtual embodied robot has failed to perform the target home task, the target environment data can be updated based on the data of the virtual embodied robot's failure to perform the target home task, and the updated target environment data includes the data of the virtual embodied robot's failure to perform the target home task and the target environment data output by the data processing model in step 520, and returns to execute step 530 and subsequent steps, so that the task reasoning model outputs a new target home task based on the updated target environment data until step 560 is executed.

[0186] The data indicating that the virtual embodied robot fails to perform the target household task may be image data of a scene image captured from a three-dimensional model.

[0187] Optionally, if the number of times the virtual embodied robot fails to perform the target home task reaches a preset threshold (set based on manual experience), it may be that the target home task cannot be achieved or is beyond the ability of the embodied robot to perform the task. At this time, the control device can report an error to the user.

[0188] In this embodiment, if the virtual embodied robot fails to perform the target home task in the task simulation model, it means that the target home task output by the task reasoning model is unreasonable. Therefore, the task reasoning model outputs a new target home task based on the updated target environment data, which is conducive to further improving the accuracy of the embodied robot in performing home tasks in a real home environment.

[0189] In some embodiments, after step 560, the control device may determine whether the current posture of the target object matches the target posture of the target object; wherein the target posture is the posture of the virtual object when the virtual embodied robot successfully performs the target home task, and the virtual object corresponds to the target object; if the current posture matches the target posture, it is determined that the target embodied robot successfully performs the target home task; if the current posture does not match the target posture, it is determined that the target embodied robot fails to perform the target home task.

[0190] The current posture of the target object may be the position and direction of the target object in the home environment after the target embodied robot completes the target home task, that is, the posture of the target object in the home environment.

[0191] Optionally, if it is determined that the target embodied robot fails to perform the target home task, the process returns to step 560 so that the target robot performs the target home task again until the current posture of the target object matches the target posture, and it is determined that the target embodied robot succeeds in performing the target home task.

[0192] For example, if the target home task is to open the door, the virtual object is the "door" in the three-dimensional model. When it is determined that the virtual embodied robot has successfully performed the door opening task in the three-dimensional model, the current posture of the "door" is recorded as the target posture. After the target embodied robot completes the door opening task on the "door" in the real home environment, it is determined whether the current posture of the "door" and the target posture match. If they match, it means that the "door" in the real home environment has reached the opening and closing degree of the "door" in the three-dimensional model, and it is determined that the target embodied robot has successfully performed the target home task; if they do not match, it means that the "door" in the real home environment has not reached the opening and closing degree of the "door" in the three-dimensional model, and it is determined that the target embodied robot has successfully performed the target home task.

[0193] In this embodiment, the result of the target embodied robot performing the target home task in a real home environment is judged based on the posture of the virtual object when the virtual embodied robot successfully performs the target home task, and the current posture of the target object after the target embodied robot completes the target home task, thereby improving the accuracy of the target embodied robot in performing the target home task.

[0194] See also Figure 7 , which shows a task execution device 600 of an embodied robot provided by an embodiment of the present application. In a specific embodiment, the task execution device 600 of the embodied robot can be applied to the embodied robot 100, and can also be applied to Figure 1 The control device 400 in the embodied robot control system shown in FIG. 1 is used as an example to describe the control device 400. Figure 7 The task execution device 600 of the embodied robot shown in FIG. 6 is described in detail. The task execution device 600 of the embodied robot may include a first acquisition module 610 , a first input module 620 , a second input module 630 and a sending module 640 .

[0195] The first acquisition module 610 can be used to obtain initial environmental data corresponding to the home environment; the first input module 620 can be used to input the initial environmental data into the data processing model to obtain target environmental data; the second input module 630 can be used to input the target environmental data into the task reasoning model to obtain the target home task; the sending module 640 can be used to send the target home task to the target embodied robot, so that the target embodied robot performs the target home task on the target object associated with the target home task.

[0196] In some embodiments, the model parameter quantity of the task reasoning model may be greater than the model parameter quantity of the data processing model, and the parameter quantity difference between the model parameter quantity of the task reasoning model and the model parameter quantity of the data processing model is at least not less than 3 orders of magnitude.

[0197] In some embodiments, the task execution device 600 of the embodied robot may further include a second acquisition module and a third input module.

[0198] The second acquisition module can be used to send module 640 to send the target home task to the target embodied robot, so that before the target embodied robot performs the target home task on the target object associated with the target home task, it obtains the current object state of the target object and the current robot state of the target embodied robot; the third input module can be used to input the target home task, the current object state and the current robot state into the action generation model to obtain the target action information.

[0199] In some embodiments, the sending module 640 may include a first sending unit.

[0200] The first sending unit can be used to send target action information to the target embodied robot, so that the target embodied robot performs the corresponding target action on the target object according to the target action information to perform the target household task.

[0201] In some embodiments, the model parameter amount of the action generation model may be smaller than the model parameter amount of the task reasoning model, and the model parameter amount of the action generation model is greater than the model parameter amount of the data processing model, and the parameter difference between the model parameter amount of the action generation model and the model parameter amount of the task reasoning model is at least not less than 2 orders of magnitude.

[0202] In some implementations, the first acquisition module 610 may acquire a unit.

[0203] The acquisition unit can be used to acquire initial environmental data based on a first sensor and a second sensor; wherein the first sensor can be installed on the target embodied robot, and the second sensor can be installed in a home environment.

[0204] In some implementations, the target home task may include multiple subtasks, and the second input module 630 may include a first input unit.

[0205] The first input unit can be used to input target environment data into the task reasoning model to obtain multiple subtasks.

[0206] In some embodiments, the target embodied robot may include multiple sub-embodied robots, the target object may include multiple sub-objects, each subtask may be associated with a sub-object, and the task execution device 600 of the embodied robot may further include a first determination module.

[0207] The first determination module can be used for the sending module 640 to send the target home task to the target embodied robot, so that the target embodied robot determines a sub-embodied robot corresponding to each subtask before performing the target home task on the target object associated with the target home task.

[0208] In some implementations, the sending module 640 may further include a second sending unit.

[0209] The second sending unit may be configured to send each subtask to a corresponding sub-embodied robot, so that each sub-embodied robot performs an associated subtask on a corresponding sub-object.

[0210] In some implementations, the sending module 640 may further include a first determining unit and a third sending unit.

[0211] The first determination unit can be used to determine the target area of ​​the target object in the home environment; the third sending unit can be used to send the target home task and the target area to the target embodied robot, so that the target embodied robot drives to the target area to perform the target home task on the target object.

[0212] In some embodiments, the task execution device 600 of the embodied robot may further include a third acquisition module.

[0213] The third acquisition module can be used for the second input module 630 to input the target environment data into the task reasoning model, and obtain the initial home task instructions before obtaining the target home task.

[0214] In some embodiments, the second input module 630 may further include a second input unit.

[0215] The second input unit can be used to input initial home task instructions and target environment data into the task reasoning model to obtain the target home task.

[0216] In some embodiments, the task execution device 600 of the embodied robot may further include a fourth input module and a second determination module.

[0217] The fourth input module can be used to send module 640 to send the target home task to the target embodied robot, so that before the target embodied robot performs the target home task on the target object associated with the target home task, the target home task is input into the task simulation model, so that the virtual embodied robot in the task simulation model performs the target home task in the three-dimensional model corresponding to the home environment; the second determination module can be used to determine whether the virtual embodied robot successfully performs the target home task.

[0218] In some implementations, the sending module 640 may further include a fourth sending unit.

[0219] The fourth sending unit can be used to send the target home task to the target embodied robot when it is determined that the virtual embodied robot has successfully performed the target home task, so that the target embodied robot performs the target home task on the target object associated with the target home task.

[0220] In some embodiments, the task execution device 600 of the embodied robot may further include an update module and a return module.

[0221] The update module can be used after the second determination module determines whether the virtual embodied robot successfully performs the target home task. When it is determined that the virtual embodied robot fails to perform the target home task, the target environment data is updated according to the data of the virtual embodied robot's failure to perform the target home task. The updated target environment data may include the data of the virtual embodied robot's failure to perform the target home task and the target environment data; the return module can be used to return the execution input target environment data to the task reasoning model to obtain the steps and subsequent steps of the target home task, until it is determined that the virtual embodied robot succeeds in performing the target home task, and the target home task is sent to the target embodied robot.

[0222] In some embodiments, the task execution device 600 of the embodied robot may further include a third determination module, a fourth determination module, and a fifth determination module.

[0223] The third determination module can be used to determine whether the current posture of the target object matches the target posture of the target object; wherein the target posture can be the posture of the virtual object when the virtual embodied robot successfully performs the target home task, and the virtual object can correspond to the target object; the fourth determination module can be used to determine that the target embodied robot successfully performs the target home task when the current posture matches the target posture; the fifth determination module can be used to determine that the target embodied robot fails to perform the target home task when the current posture does not match the target posture.

[0224] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to in detail. For the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple. For the relevant parts, refer to the partial description of the method embodiments. Any processing method described in the method embodiment can be implemented by the corresponding processing module in the device embodiment, and will not be repeated in detail in the device embodiment.

[0225] In addition, the functional modules in the various embodiments of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into a single module. The above-mentioned integrated modules may be implemented in the form of hardware or software functional modules.

[0226] See also Figure 8 , which shows a functional block diagram of an embodied robot 700 provided in one embodiment of the present application. The embodied robot 700 may include one or more of the following components: a memory 710, a processor 720, and one or more applications, wherein the one or more applications may be stored in the memory 710 and configured to be executed by the one or more processors 720, and the one or more applications are configured to execute the method described in the aforementioned method embodiment.

[0227] The memory 710 may include random access memory (RAM) or read-only memory (ROM). The memory 710 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 710 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as obtaining initial environment data, inputting initial environment data, obtaining target environment data, inputting target environment data, obtaining a target home task, sending a target home task, executing a target home task, obtaining a current object state, obtaining a current robot state, inputting a target home task, inputting a current object state, inputting a current robot state, obtaining target action information, training a deep learning neural network model, sending target action information, labeling historical home tasks, obtaining a training set, inferring historical home tasks, obtaining a task inference model, determining a sub-embodied robot, determining a target area, sending a target area, driving to a target area, obtaining an initial home task instruction, and inputting an initial home task instruction), and instructions for implementing the following method embodiments. The data storage area can also store data created by the embodied robot 700 during use (such as home environment, initial environment data, data processing model, target environment data, task reasoning model, target home task, target embodied robot, target object, model parameter quantity, current object state, current robot state, action generation model, target action information, historical action information, historical home information, deep learning neural network model, historical home tasks, historical object state, historical robot state, target action, first sensor, second sensor, historical environment data, historical home tasks, training set, sub-task, sub-embodied robot, sub-object, target area and initial home task instructions), etc.

[0228] The processor 720 may include one or more processing cores. The processor 720 utilizes various interfaces and circuits to connect various components within the embodied robot 700. It executes instructions, programs, code sets, or instruction sets stored in the memory 710, as well as accesses data stored in the memory 710, to perform various functions and process data for the embodied robot 700. Optionally, the processor 720 may be implemented using at least one of the following hardware forms: a digital signal processing (DSP), a field-programmable gate array (FPGA), or a programmable logic array (PLA). The processor 720 may integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. The CPU primarily processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem handles wireless communications. It is understood that the modem may also be implemented independently of the processor 720 via a separate communications chip.

[0229] Please refer to Figure 9 , which shows a functional block diagram of an embodied robot system 800 provided in one embodiment of the present application. The embodied robot system 800 may include a sensor 810 and a controller 820.

[0230] The sensor 810 may be used to collect initial environmental data corresponding to the home environment, and the controller 820 may be used to execute the method described in the aforementioned method embodiment.

[0231] Please refer to Figure 10 , which shows a block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable storage medium 900 stores program code 910, which can be called by a processor to execute the method described in the above method embodiment.

[0232] Computer-readable storage medium 900 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, a hard disk, or ROM. Alternatively, computer-readable storage medium 900 may include non-transitory computer-readable storage medium. Computer-readable storage medium 900 has storage space for program code 910 for executing any of the method steps described above. This program code can be read from or written to one or more computer program products. Program code 910 may be compressed, for example, in a suitable format.

[0233] Please refer to Figure 11 , which shows a block diagram of the structure of a computer program product 1000 provided in an embodiment of the present application. Computer program product 1000 includes a computer program / instructions 1010, which is stored in a computer-readable storage medium of a computer device. When computer program product 1000 is executed on a computer device, the computer device's processor reads computer program / instructions 1010 from the computer-readable storage medium and executes computer program / instructions 1010, causing the computer device to perform the method described in the above method embodiment.

[0234] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A task execution method of an embodied robot, characterized in that: include: Obtaining initial environmental data corresponding to the home environment; The initial environmental data includes image data, video data, temperature data, humidity data, light intensity data and semantic data; Inputting the initial environmental data into a data processing model to obtain target environmental data; the data processing model is used to analyze the initial environmental data to obtain the target environmental data; the target environmental data includes environmental data predicted based on the initial environmental data; Inputting the target environment data into a task reasoning model to obtain a target home task; the task reasoning model is obtained by training a deep learning neural network model based on historical environment data labeled with historical home tasks; the model parameters of the task reasoning model are greater than the model parameters of the data processing model, and the parameter difference between the model parameters of the task reasoning model and the model parameters of the data processing model is no less than three orders of magnitude; Inputting the target home task into a task simulation model so that a virtual embodied robot in the task simulation model performs the target home task in a three-dimensional model corresponding to the home environment, thereby simulating a process in which the target embodied robot performs the target home task on a target object in a real home environment; Determining whether the virtual embodied robot successfully performs the target household task through an evaluation model; the evaluation model pre-constructs a correspondence between the target household task and the expected target image; When it is determined that the virtual embodied robot successfully performs the target home task, sending the target home task to the target embodied robot, so that the target embodied robot performs the target home task on the target object associated with the target home task; When it is determined that the virtual embodied robot fails to perform the target home task, the target environment data is updated according to the data of the virtual embodied robot failing to perform the target home task, where the updated target environment data includes the data of the virtual embodied robot failing to perform the target home task and the target environment data before the update; the data of the virtual embodied robot failing to perform the target home task includes image data of a scene image captured from a three-dimensional model; Return to executing the step of inputting the target environment data into the task reasoning model to obtain the target home task and subsequent steps, until it is determined that the virtual embodied robot successfully performs the target home task, and then send the target home task to the target embodied robot.

2. The task execution method according to claim 1, characterized in that: Before sending the target home task to the target embodied robot so that the target embodied robot performs the target home task on the target object associated with the target home task, the task execution method further includes: Acquiring a current object state of the target object and a current robot state of the target embodied robot; Inputting the target household task, the current object state, and the current robot state into the action generation model to obtain target action information; The sending of the target household task to the target embodied robot, so that the target embodied robot performs the target household task on a target object associated with the target household task, includes: The target action information is sent to the target embodied robot, so that the target embodied robot performs a corresponding target action on the target object according to the target action information to perform the target household task.

3. The task execution method according to claim 2, characterized in that: The model parameter amount of the action generation model is smaller than the model parameter amount of the task reasoning model, and the model parameter amount of the action generation model is larger than the model parameter amount of the data processing model, and the parameter difference between the model parameter amount of the action generation model and the model parameter amount of the task reasoning model is not less than 2 orders of magnitude.

4. The task execution method according to claim 1, characterized in that: The obtaining of initial environment data corresponding to the home environment includes: The initial environment data is acquired based on a first sensor and a second sensor; wherein the first sensor is installed on the target embodied robot, and the second sensor is installed in the home environment.

5. The task execution method according to any one of claims 1 to 4, characterized in that: The target home task includes multiple subtasks. Inputting the target environment data into the task reasoning model to obtain the target home task includes: The target environment data is input into the task reasoning model to obtain the multiple subtasks.

6. The task execution method according to claim 5, characterized in that: The target embodied robot includes a plurality of sub-embodied robots, the target object includes a plurality of sub-objects, each sub-task is associated with a sub-object, and before sending the target household task to the target embodied robot so that the target embodied robot performs the target household task on the target object associated with the target household task, the task execution method further includes: Determining a sub-embodied robot corresponding to each subtask; The sending of the target household task to the target embodied robot, so that the target embodied robot performs the target household task on a target object associated with the target household task, includes: Each of the subtasks is sent to the corresponding one of the sub-embodied robots, so that each sub-embodied robot performs an associated subtask on the corresponding one of the sub-objects.

7. The task execution method according to any one of claims 1 to 4, characterized in that: The sending of the target household task to the target embodied robot, so that the target embodied robot performs the target household task on a target object associated with the target household task, includes: determining a target area of ​​the target object in the home environment; The target household task and the target area are sent to the target embodied robot, so that the target embodied robot travels to the target area and performs the target household task on the target object.

8. The task execution method according to any one of claims 1 to 4, characterized in that: Before inputting the target environment data into the task reasoning model to obtain the target home task, the task execution method further includes: Get initial home task instructions; Inputting the target environment data into the task reasoning model to obtain the target home task includes: The initial home task instruction and the target environment data are input into the task reasoning model to obtain the target home task.

9. The task execution method according to claim 1, characterized in that: After sending the target household task to the target embodied robot, the task execution method further includes: Determining whether the current posture of the target object matches the target posture of the target object; wherein the target posture is the posture of the virtual object when the virtual embodied robot successfully performs the target household task, and the virtual object corresponds to the target object; When the current posture matches the target posture, determining that the target embodied robot successfully performs the target home task; When the current posture does not match the target posture, it is determined that the target embodied robot has failed to perform the target household task.

10. An embodied robot, characterized in that: include: Memory; one or more processors coupled to the memory; One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more application programs are configured to execute the task execution method according to any one of claims 1 to 9.

11. An embodied robotic system, characterized in that: It includes a sensor and a controller, wherein the sensor is connected to the controller; The sensor is used to collect initial environmental data corresponding to the home environment; The controller is used to execute the task execution method according to any one of claims 1 to 9.

Images