Body-equipped robot, task execution method thereof and body-equipped robot system

Abstract

The invention discloses a robot with a body, a task execution method of the robot with the body and a system of the robot with the body, and the task execution method of the robot with the body comprises the steps: obtaining initial environment data corresponding to a home environment; inputting the initial environment data to a data processing model to obtain target environment data; inputting the target environment data to a task reasoning model to obtain a target home task; and sending the target home task to the target body-equipped robot, so that the target body-equipped robot executes the target home task on a target object associated with the target home task. According to the method, the environment data of the home environment is sequentially processed based on the data processing model and the task reasoning model to obtain the home task, the accuracy of the home task executed by the robot is improved, the target home task is obtained by obtaining the environment data in the home environment, and the user experience is improved. The intelligent degree of home task execution of the robot can be improved.

Description

Technical Field

[0001] This application belongs to the technical field of smart home, and particularly relates to an embodied robot, a task execution method thereof, and an embodied robot system. Background Art

[0002] With the rapid development of technology, smart home systems have also become popular in people's lives. Compared with traditional homes, more and more people choose to use smart home devices in smart home systems. For example, they use embodied robots to perform home tasks.

[0003] Embodied robots can be classified into various types according to their functions and application scenarios, including unlocking robots, cleaning robots, curtain robots, companion robots, and humanoid robots, etc.

[0004] Currently, an embodied robot executes corresponding home tasks according to the home task instructions sent by the user. However, when the home task instruction is a fuzzy instruction, the embodied robot cannot determine which home task needs to be executed, resulting in low control accuracy for controlling the embodied robot. Summary of the Invention

[0005] In view of this, embodiments of this application provide an embodied robot, a task execution method thereof, and an embodied robot system to overcome the above problems of the prior art.

[0006] In a first aspect, embodiments of this application provide a task execution method for an embodied robot, including: Obtaining initial environment data corresponding to a home environment; Inputting the initial environment data into a data processing model to obtain target environment data; Inputting the target environment data into a task inference model to obtain a target home task; Sending the target home task to a target embodied robot, so that the target embodied robot executes the target home task on a target object associated with the target home task.

[0007] Wherein, in some optional embodiments, the number of model parameters of the task inference model is greater than the number of model parameters of the data processing model, and the difference in the number of parameters between the number of model parameters of the task inference model and the number of model parameters of the data processing model is at least not less than three orders of magnitude.

[0008] Wherein, in some optional embodiments, before sending the target home task to a target embodied robot, so that the target embodied robot executes the target home task on a target object associated with the target home task, the task execution method further includes: Obtain the current object state of the target object and the current robot state of the target embodied robot; Input the target home task, the current object state, and the current robot state into an action generation model to obtain target action information; Sending the target home task to the target embodied robot, such that the target embodied robot executes the target home task on the target object associated with the target home task, includes: Send the target action information to the target embodied robot, such that the target embodied robot executes corresponding target actions on the target object according to the target action information to execute the target home task.

[0009] Wherein, in some alternative embodiments, the number of model parameters of the action generation model is less than the number of model parameters of the task inference model, and the number of model parameters of the action generation model is greater than the number of model parameters of the data processing model, and the difference in the number of parameters between the number of model parameters of the action generation model and the number of model parameters of the task inference model is at least not less than two orders of magnitude.

[0010] Wherein, in some alternative embodiments, the obtaining the initial environmental data corresponding to the home environment includes: Obtain the initial environmental data based on a first sensor and a second sensor, the first sensor being installed on the target embodied robot, and the second sensor being installed in the home environment.

[0011] Wherein, in some alternative embodiments, the target home task includes multiple subtasks, and the inputting the target environmental data into the task inference model to obtain the target home task includes: Input the target environmental data into the task inference model to obtain the multiple subtasks.

[0012] Wherein, in some alternative embodiments, the target embodied robot includes multiple sub-embodied robots, the target object includes multiple sub-objects, each subtask is associated with a sub-object, and before sending the target home task to the target embodied robot, such that the target embodied robot executes the target home task on the target object associated with the target home task, the task execution method further includes: Determine a sub-embodied robot corresponding to each subtask; Sending the target home task to the target embodied robot, such that the target embodied robot executes the target home task on the target object associated with the target home task, includes: Send each of the subtasks to the corresponding one of the embodied robots, so that each embodied robot performs an associated subtask on the corresponding one of the sub-objects.

[0013] Wherein, in some alternative embodiments, the sending the target home task to the target embodied robot, so that the target embodied robot performs the target home task on the target object associated with the target home task, includes: Determine the target area of the target object in the home environment; Send the target home task and the target area to the target embodied robot, so that the target embodied robot travels to the target area to perform the target home task on the target object.

[0014] Wherein, in some alternative embodiments, before the inputting the target environment data into the task inference model to obtain the target home task, the task execution method further includes: Obtain an initial home task instruction; The inputting the target environment data into the task inference model to obtain the target home task includes: Input the initial home task instruction and the target environment data into the task inference model to obtain the target home task.

[0015] Wherein, in some alternative embodiments, before the sending the target home task to the target embodied robot, so that the target embodied robot performs the target home task on the target object associated with the target home task, the task execution method further includes: Input the target home task into the task simulation model, so that the virtual embodied robot in the task simulation model performs the target home task in the three-dimensional model corresponding to the home environment; Determine whether the virtual embodied robot successfully performs the target home task; The sending the target home task to the target embodied robot, so that the target embodied robot performs the target home task on the target object associated with the target home task, includes: When it is determined that the virtual embodied robot successfully performs the target home task, send the target home task to the target embodied robot, so that the target embodied robot performs the target home task on the target object associated with the target home task.

[0016] Wherein, in some alternative embodiments, after the determining whether the virtual embodied robot successfully performs the target home task, the task execution method further includes: When it is determined that the virtual embodied robot fails to execute the target home task, update the target environment data according to the data of the failure of the virtual embodied robot to execute the target home task. The updated target environment data includes the data of the failure of the virtual embodied robot to execute the target home task and the target environment data; Return to execute the step of inputting the target environment data into the task inference model to obtain the steps and subsequent steps of the target home task, until it is determined that the virtual embodied robot successfully executes the target home task, and then send the target home task to the target embodied robot.

[0017] Wherein, in some alternative embodiments, after sending the target home task to the target embodied robot, the task execution method further includes: Determine whether the current pose of the target object matches the target pose of the target object; wherein, the target pose is the pose of the virtual object when the virtual embodied robot successfully executes the target home task, and the virtual object corresponds to the target object; When the current pose matches the target pose, determine that the target embodied robot successfully executes the target home task; When the current pose does not match the target pose, determine that the target embodied robot fails to execute the target home task.

[0018] In a second aspect, an embodiment of the present application provides a task execution device for an embodied robot, including: A first acquisition module, configured to acquire initial environment data corresponding to a home environment; A first input module, configured to input the initial environment data into a data processing model to obtain target environment data; A second input module, configured to input the target environment data into a task inference model to obtain a target home task; A sending module, configured to send the target home task to a target embodied robot, so that the target embodied robot executes the target home task on a target object associated with the target home task.

[0019] In a third aspect, an embodiment of the present application provides an embodied robot, including: A memory; One or more processors, coupled to the memory; One or more application programs, wherein the one or more application programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more application programs are configured to execute the task execution method of the embodied robot provided in the first aspect as described above.

[0020] Fourthly, an embodiment of the present application provides a computer-readable storage medium, in which program code is stored, and the program code can be called by a processor to execute the task execution method of the embodied robot provided in the first aspect above.

[0021] Fifthly, an embodiment of the present application provides a computer program product, which, when running on a computer device, causes the computer device to execute the task execution method of the embodied robot provided in the first aspect above.

[0022] Sixthly, an embodiment of the present application provides an embodied robot system, including a sensor and a controller, and the sensor is connected to the controller; The sensor is used to collect initial environmental data corresponding to the home environment; The controller is used to execute the task execution method of the embodied robot provided in the first aspect above.

[0023] The solution provided by the present application obtains the initial environmental data corresponding to the home environment, inputs the initial environmental data into the data processing model to obtain the target environmental data, inputs the target environmental data into the task inference model to obtain the target home task, and sends the target home task to the target embodied robot, so that the target embodied robot executes the target home task on the target object associated with the target home task, realizing the sequential processing of the environmental data of the home environment based on the data processing model and the task inference model respectively to obtain the home task, which is beneficial to improving the accuracy of the embodied robot in executing the home task. Moreover, by obtaining the environmental data in the home environment to obtain the target home task, the intelligence level of the embodied robot in executing the home task can be improved.

[0024] Furthermore, compared with using a traditional end-to-end large model to obtain the target home task through the environmental data in the home environment, which has high computational resources and storage space occupancy, the computational resources and storage space occupancy of obtaining the target home task by sequentially processing the environmental data of the home environment based on the data processing model and the task inference model are low, which can speed up the response time of controlling the embodied robot, is beneficial to improving the control efficiency of controlling the embodied robot, and the environmental perception error of the traditional end-to-end model will be directly transmitted to the task decision layer. By using different models to focus on solving different problems in the present application, the accuracy of obtaining the target home task can also be improved. Description of the Drawings

[0025] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the accompanying drawings required for use in the embodiments or the description of the prior art. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0026] Figure 1 Fig. shows a schematic diagram of a scenario of an embodied robot control system provided by an embodiment of the present application.

[0027] Figure 2 Fig. shows a schematic flowchart of a method for task execution of an embodied robot provided by an embodiment of the present application.

[0028] Figure 3 Fig. shows another schematic flowchart of a method for task execution of an embodied robot provided by an embodiment of the present application.

[0029] Figure 4 Fig. shows yet another schematic flowchart of a method for task execution of an embodied robot provided by an embodiment of the present application.

[0030] Figure 5 Fig. shows yet another schematic flowchart of a method for task execution of an embodied robot provided by an embodiment of the present application.

[0031] Figure 6 Fig. shows yet another schematic flowchart of a method for task execution of an embodied robot provided by an embodiment of the present application.

[0032] Figure 7 Fig. shows a structural block diagram of a task execution device of an embodied robot provided by an embodiment of the present application.

[0033] Figure 8 Fig. shows a functional block diagram of an embodied robot provided by an embodiment of the present application.

[0034] Figure 9 Fig. shows a functional block diagram of an embodied robot system provided by an embodiment of the present application.

[0035] Figure 10 Fig. shows a computer-readable storage medium for storing or carrying program codes for implementing the method for task execution of an embodied robot according to an embodiment of the present application.

[0036] Figure 11 Fig. shows a computer program product for storing or carrying program codes for implementing the method for task execution of an embodied robot according to an embodiment of the present application. Detailed implementation manners

[0037] To make the objectives, features, and advantages of the present application more obvious and understandable, the following will clearly and completely describe the technical solutions in the embodiments of the present application in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the embodiments described below are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0038] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, wholes, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0039] It should also be understood that the terms used in this specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in this specification of the present application and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" are intended to include the plural forms.

[0040] It should be further understood that the term "and / or" used in this specification of the present application and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0041] In addition, in the description of the present application, the terms "first", "second", "third", etc. are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance.

[0042] With the rapid development of technology, smart home systems have also become popular in people's lives. Compared with traditional homes, more and more people choose to use smart home devices in smart home systems. For example, embodied robots are used to perform home tasks.

[0043] Embodied robots can be divided into various types according to different functions and application scenarios, including unlocking robots, cleaning robots, curtain robots, companion robots, and humanoid robots, etc.

[0044] Currently, embodied robots execute corresponding home tasks according to the home task instructions sent by users. However, when the home task instructions are ambiguous instructions, the embodied robots cannot determine what kind of home tasks need to be executed, resulting in relatively low control accuracy in controlling the embodied robots.

[0045] In view of the above problems, the embodiment of the present application provides an embodied robot, its task execution method, and an embodied robot system. By obtaining the initial environmental data corresponding to the home environment, inputting the initial environmental data into a data processing model to obtain target environmental data, inputting the target environmental data into a task inference model to obtain a target home task, and sending the target home task to a target embodied robot, the target embodied robot executes the target home task on the target object associated with the target home task, realizing the sequential processing of the environmental data of the home environment based on the data processing model and the task inference model to obtain the home task, which is beneficial to improving the accuracy of the embodied robot in executing the home task. Moreover, by obtaining the environmental data in the home environment to obtain the target home task, the intelligence level of the embodied robot in executing the home task can be improved.

[0046] Furthermore, compared with using a traditional end-to-end large model, which has a high occupancy of computing resources and storage space for obtaining the target home task by acquiring the environmental data in the home environment, the method of sequentially processing the environmental data of the home environment based on the data processing model and the task inference model has a low occupancy of computing resources and storage space, can accelerate the response time for controlling the embodied robot, is beneficial to improving the control efficiency of the embodied robot, and the environmental perception error of the traditional end-to-end model will directly be transmitted to the task decision layer. By using different models to focus on solving different problems in the present application, the accuracy of obtaining the target home task can also be improved.

[0047] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application.

[0048] Please refer to Figure 1 , which shows a schematic diagram of an application scenario of an embodied robot control system provided by an embodiment of the present application. The embodied robot control system may include an embodied robot 100, a sensor 200, an object 300, a control device 400, etc. The control device 400 can be connected to the embodied robot 100 and the sensor 200 through a network and perform data interaction with the embodied robot 100 and the sensor 200 through the network.

[0049] Among them, the number of embodied robots 100 can be one or more, and the embodied robot 100 is a mobile home robot or a fixed home robot. The mobile home robot can move in the home environment. For example, the mobile home robot can include at least any one of a cleaning robot (such as a sweeping robot, a mopping robot, and a sweeping and mopping integrated robot), a robotic arm, a companion robot, and a humanoid robot, etc., which is not limited here.

[0050] A stationary home robot is a robot that cannot move in a home environment, i.e., it is fixedly installed in a home environment. For example, the stationary home robot can include at least any one of a curtain robot, a switch robot, a lock-opening robot, etc., and no limitation is made here.

[0051] The sensor 200 can be used to collect environmental data of the home environment. The sensor 200 can include at least any one of a vision sensor, a temperature sensor, a humidity sensor, a light sensor, a sound sensor, etc. The environmental data can include at least any one of image data, video data, temperature data, humidity data, light intensity data, semantic data, etc.

[0052] The object 300 can include at least any one of a user, a water cup, a sofa, a refrigerator, an air conditioner, a table and chair, a TV, a trash can, etc., and no limitation is made here.

[0053] The control device 400 can include any one of a terminal device and a server, etc. The type of the control device 400 is not limited here and can be specifically set according to actual needs.

[0054] The terminal device can be a mobile terminal device (for example, any one of a mobile phone, a personal digital assistant (PDA), a tablet personal computer (Tablet PC), a laptop computer, a smart watch, a smart bracelet, or a wearable device, etc.), or a fixed terminal device (a smart gateway device, a desktop computer, a smart panel, etc.), etc., and no limitation is made here.

[0055] The server can be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery network (CDN), big data, or artificial intelligence platforms, etc., and no limitation is made here.

[0056] The network can be any one of a ZigBee network, a Bluetooth (BT) network, a Wireless Fidelity (Wi-Fi) network, a Thread network, a Long Range Radio (LoRa) network, a Low-Power Wide-Area Network (LPWAN), an infrared network, a Narrow Band Internet of Things (NB-IoT), a Controller Area Network (CAN), a Digital Living Network Alliance (DLNA) network, a Wide Area Network (WAN), a Local Area Network (LAN), a Metropolitan Area Network (MAN), or a Wireless Personal Area Network (WPAN), etc., which is not limited herein.

[0057] In some embodiments, the number of embodied robots 100 can be multiple. The sensor 200 can include a first sensor and a second sensor. The first sensor can be installed on multiple embodied robots, and the second sensor can be installed in a home environment.

[0058] Among them, the number of the first sensors is multiple, and one first sensor is installed on each embodied robot body. The types of the first sensors installed on multiple embodied robot bodies can be the same or different. Both the first sensor and the second sensor can include at least any one of a visual sensor, a temperature sensor, a humidity sensor, a light sensor, a sound sensor, etc.

[0059] Please refer to Figure 2 , which shows a flowchart of a task execution method of an embodied robot provided by an embodiment of the present application. In a specific embodiment, the task execution method of the embodied robot can be applied to the embodied robot 100, or can also be applied to a control device 400 in an embodied robot control system as shown in Figure 1 . Taking the application to the control device 400 as an example, the process shown in Figure 2 will be elaborated in detail below. The control method of the embodied robot can include the following steps 110 to 140.

[0060] Step 110: Obtain initial environment data corresponding to the home environment.

[0061] In the embodiments of the present application, the number of embodied robots can be multiple. The sensors can include a first sensor and a second sensor. The detection range of the first sensor can be the first home area environment in the home environment, and the detection range of the second sensor can be the second home area environment in the home environment. The initial environment data can include first environment data and second environment data.

[0062] Optionally, the control device can send acquisition instructions to the first sensor and the second sensor through a network. The first sensor receives and responds to the acquisition instructions, collects data on the first home area environment, and sends the collected first environment data to the control device through the network. The second sensor receives and responds to the acquisition instructions, collects data on the second home area environment, and sends the collected second environment data to the control device through the network. The control device respectively receives the first environment data returned by the first sensor and the second environment data returned by the second sensor to obtain the initial environment data.

[0063] Optionally, the first sensor can collect data on the first home area environment in real time or at a preset frequency, and the second sensor can collect data on the second home area environment in real time or at a preset frequency.

[0064] In this embodiment, based on the first sensor installed on the embodied robot and the second sensor installed in the home environment to collect environmental data of the home environment, it can prevent the environmental data collected by a single sensor from being incomplete due to the limitation of the detection range, which is beneficial to improving the integrity of the environmental data.

[0065] Among them, both the first environment data and the second environment data can include at least any one of image data, video data, temperature data, humidity data, light intensity data, and semantic data, etc.

[0066] Step 120: Input the initial environment data into the data processing model to obtain the target environment data.

[0067] In the embodiments of the present application, the control device can input the initial environment data into the data processing model. The data processing model receives and responds to the initial environment data and outputs the target environment data.

[0068] Optionally, the target environment data is structured data. The data processing model can be used to process the initial environment data to obtain the target environment data. Here, since the initial environment data may have noise (such as blurred images, invalid sensor values), and the data formats of different modalities vary greatly (such as the pixel matrix of images, the waveform of audio, the numerical sequence of sensors), the data processing model is used to convert the messy unstructured data into structured data in a unified format.

[0069] Among them, the data processing model can be obtained by training a deep learning neural network model based on historical environmental data labeled with historical structured data. The historical structured data is obtained by processing the historical environmental data, and the historical environmental data can include at least any one of image data, video data, temperature data, humidity data, light intensity data, and semantic data, etc.

[0070] The deep learning neural network model can be any one of a Convolutional Neural Networks (CNN) model, a Deep Belief Networks (DBN) model, a Stacked Auto Encoder Networks (SAE) model, a Recurrent Neural Networks (RNN) model, a Deep Neural Networks (DNN) model, a Long Short-Term Memory (LSTM) network model, or a Gated Recurring Units (GRU) model, etc. The type of the deep learning neural network model is not limited here, and it can be specifically set according to actual needs.

[0071] Optionally, the data processing model can be used to analyze the initial environmental data (including simple evaluation, classification, filtering, or feature extraction, etc.) to obtain target environmental data for making a preliminary judgment or prediction on certain situations or trends. For example, the initial environmental data includes temperature data, humidity data, and air pressure data, and the data processing model obtains target environmental data of "80% probability of rain within 1 hour" through the analysis of the initial environmental data.

[0072] Optionally, the data processing model can be an object detection model (such as CNN, YOLO, SSD, etc.).

[0073] Step 130: Input the target environmental data into the task inference model to obtain the target home task.

[0074] In the embodiment of the present application, the control device can input the target environmental data into the task inference model. The task inference model receives and responds to the target environmental data, outputs the target home task to the control device, and the control device receives the target home task output by the task inference model. For example, the target environmental data can be used to represent that there is an item at a certain position on the ground, and the task inference model can imagine that the item will block people's walking at that position, thus outputting a target home task of "move the item away".

[0075] Among them, the number of target home tasks can be one or more. The task inference model can be obtained by training a deep learning neural network model based on historical environmental data annotated with historical home tasks. The historical home tasks can be obtained by reasoning about the historical environmental data.

[0076] As an example, the deep learning neural network model can be any one of a large language model (LLM) or a multimodal visual language model (VLM), etc. The type of the deep learning neural network model is not limited here and can be specifically set according to actual needs.

[0077] In some embodiments, the number of model parameters of the task inference model can be greater than the number of model parameters of the data processing model, and the difference in the number of model parameters between the task inference model and the data processing model is at least not less than three orders of magnitude.

[0078] Among them, the number of model parameters refers to the total number of parameters that need to be learned and adjusted in the model. The size of the number of model parameters will affect the capacity, complexity, and computational requirements of the model.

[0079] As an example, the number of model parameters of the data processing model is less than 100M (i.e., less than 1×10^8), which is a lightweight model; the number of model parameters of the task inference model is greater than 500M or 1000M (i.e., at least greater than 5×10^11). The order of magnitude refers to the exponential difference expressed in scientific notation. The exponents between the two are 8 and 11 respectively, so the difference is three orders of magnitude.

[0080] Since the number of model parameters of the data processing model is relatively small, the computational load is low. Based on the number of model parameters of 100M, the response time of the data processing model is less than 100ms, and it can process or analyze environmental data more quickly, which is beneficial to improving the efficiency of the data processing model in processing or analyzing environmental data. The model parameters of the task inference model are relatively large. Based on the number of model parameters of 500M or 1000M, the response time of the task inference model is less than 1min, making the task inference model have strong learning and inference capabilities, which is beneficial to improving the inference accuracy of the task inference model, so that it can more accurately infer the target home tasks. Through two models with different numbers of parameters, both processing efficiency and accuracy can be taken into account.

[0081] Step 140: Send the target home task to the target embodied robot, so that the target embodied robot executes the target home task on the target object associated with the target home task.

[0082] In the embodiments of the present application, the control device can send a target home task to the target embodied robot through a network. The target embodied robot receives and responds to the target home task, and executes the target home task on the target object associated with the target home task, realizing the processing of the environmental data of the home environment based on the data processing model and the task inference model in sequence to obtain the home task, which is beneficial to improving the accuracy of the embodied robot in executing the home task. Moreover, by obtaining the environmental data in the home environment to obtain the target home task, the intelligence level of the embodied robot in executing the home task can be improved.

[0083] Furthermore, compared with using a traditional end-to-end large model to obtain the target home task by acquiring the environmental data in the home environment, which has high computational resources and storage space occupancy, the method of processing the environmental data of the home environment based on the data processing model and the task inference model in sequence to obtain the target home task has low computational resources and storage space occupancy, can accelerate the response time of controlling the embodied robot, is beneficial to improving the control efficiency of controlling the embodied robot. And the environmental perception error of the traditional end-to-end model will be directly transmitted to the task decision layer. By using different models to focus on solving different problems in this application, the accuracy of obtaining the target home task can also be improved.

[0084] Among them, the target object can be any one of a user, a water cup, a sofa, a refrigerator, an air conditioner, a table and chair, a TV, a trash can, etc., and no limitation is made here.

[0085] In some embodiments, after step 130, the control device can obtain the current object state of the target object and the current robot state of the target embodied robot, and input the target home task, the current object state and the current robot state into the action generation model. The action generation model receives and responds to the target home task, the current object state and the current robot state, outputs the target action information to the control device. The control device receives the target action information output by the action generation model, and sends the target action information to the embodied robot through the network. The embodied robot receives and responds to the target action information, and executes the corresponding target action on the target object, and controls the embodied robot to execute the corresponding action based on the action information generated by the action generation model to execute the target home task, which is beneficial to further improving the control accuracy of controlling the embodied robot.

[0086] Among them, the current object state can include the position and angle of the target object, and the current robot state can include the position and angle of the target embodied robot, and no limitation is made here.

[0087] The control device can obtain the current object state of the target object and the current robot state of the target embodied robot according to the initial environmental data. For example, the initial environmental data can include a home environment image, and the home environment image can be analyzed.

[0088] The action generation model can be a diffusion model, or it can be other deep learning models obtained by training a deep learning neural network model based on historical home information annotated with historical action information. The historical home information can include historical home tasks, historical object states, and historical robot states.

[0089] In some embodiments, the number of model parameters of the action generation model can be less than that of the task inference model, and the number of model parameters of the action generation model is greater than that of the data processing model. The difference in the number of model parameters between the action generation model and the task inference model is at least not less than two orders of magnitude.

[0090] Optionally, the difference in the number of model parameters between the data processing model and the action generation model is at least not less than one order of magnitude. Compared with the case where the perception error of environmental data using an end-to-end large model is directly transmitted to the task decision layer, performing different processes in sequence based on the data processing model, the action generation model, and the task inference model is beneficial to improving the accuracy of executing target home tasks according to environmental data.

[0091] As an example, the number of model parameters of the action generation model can be 1B, where 1B = 1×10^9. The exponents between the action generation model and the task inference model are 9 and 11 respectively, so the difference is two orders of magnitude. Based on the model parameter of 1B, the response time of the action generation model is less than 3s.

[0092] Please refer to Figure 3 , which shows a flowchart of a task execution method for an embodied robot provided by another embodiment of the present application. In a specific embodiment, the task execution method of the embodied robot can be applied to the embodied robot 100, or can also be applied to a control device 400 in an embodied robot control system as shown in Figure 1 . Taking the application to the control device 400 as an example, the process shown in Figure 3 will be elaborated in detail below. The task execution method of the embodied robot can include the following steps 210 to step 250.

[0093] Step 210: Obtain initial environmental data corresponding to the home environment.

[0094] Step 220: Input the initial environmental data into the data processing model to obtain target environmental data.

[0095] Step 230: Input the target environmental data into the task inference model to obtain multiple subtasks.

[0096] In this embodiment, the target home task may include multiple subtasks. The control device may input target environment data into the task inference model. The task inference model receives and responds to the target environment data, outputs multiple subtasks to the control device, and the control device receives the multiple subtasks output by the task inference model.

[0097] For example, if the target environment data includes the information that "the water cup in the living room has fallen over", the task inference model obtains according to the target environment data: right the water cup (task 1), wipe the table clean (task 2), and clean the floor (task 3).

[0098] Another example: If the target environment data includes the information that "the laundry basket is full of clothes", the task inference model obtains according to the target environment data: open the washing machine door (task 1), put the clothes in the laundry basket into the washing machine (task 2), add laundry beads (task 3), close the washing machine door (task 4), and start the laundry and drying process (task 5).

[0099] In some embodiments, after obtaining multiple subtasks in step 230, the multiple subtasks may be sent to a target embodied robot, so that the target embodied robot executes each subtask in sequence.

[0100] In this embodiment, the task inference model obtains multiple subtasks according to the target environment data, enabling the embodied robot to more clearly understand each subtask, thereby improving the accuracy of the embodied robot in executing tasks.

[0101] In some embodiments, after obtaining multiple subtasks in step 230, the following steps 240 - step 250 may also be performed: Step 240: Determine a sub - embodied robot corresponding to each subtask.

[0102] In this embodiment, the target embodied robot may include multiple sub - embodied robots, the target object may include multiple sub - objects, and each subtask may be associated with a sub - object.

[0103] The control device may look up the robot table according to each subtask to obtain a corresponding sub - embodied robot, and determine a sub - embodied robot corresponding to each subtask based on the correspondence between the task and the robot, improving the accuracy of determining the sub - embodied robot.

[0104] Among them, the robot table can be used to represent the correspondence between the task and the embodied robot. For example, the tasks may include task 1, task 2, task 3, task 4, and task 5, and the embodied robots may include embodied robot 1, embodied robot 2, embodied robot 3, embodied robot 4, and embodied robot 5.

[0105] The correspondence between tasks and embodied robots can be shown in Table 1, i.e., the robot table. According to the correspondence, a sub-embodied robot corresponding to each sub-task can be obtained.

[0106] Table 1

[0107] It should be noted that the correspondence between tasks and embodied robots is not limited to that shown in Table 1. Specifically, it can be set according to actual needs. It can also be that one sub-embodied robot corresponds to multiple sub-tasks. For example, embodied robot 1 corresponds to task 1, and embodied robot 3 corresponds to task 2 and task 3.

[0108] Step 250: Send each sub-task to a corresponding sub-embodied robot, so that each sub-embodied robot executes an associated sub-task on a corresponding sub-object.

[0109] In this embodiment, the control device can send each sub-task to a corresponding sub-embodied robot through the network. Each sub-embodied robot receives and responds to a sub-task, and executes an associated sub-task on a corresponding sub-object, realizing the control of each embodied robot to execute one or more corresponding tasks according to the correspondence between tasks and robots, which is beneficial to improving the overall task execution efficiency.

[0110] Please refer to Figure 4 , which shows a flowchart of the task execution method of the embodied robot provided in another embodiment of the present application. In a specific embodiment, the task execution method of the embodied robot can be applied to the embodied robot 100, or can also be applied to the control device 400 in the embodied robot control system as shown in Figure 1 . Taking the application to the control device 400 as an example, the process shown in Figure 4 will be elaborated in detail. The task execution method of the embodied robot can include the following steps 310 to step 350.

[0111] Step 310: Obtain the initial environmental data corresponding to the home environment.

[0112] Step 320: Input the initial environmental data into the data processing model to obtain the target environmental data.

[0113] Step 330: Input the target environmental data into the task inference model to obtain the target home task.

[0114] Step 340: Determine the target area of the target object in the home environment.

[0115] In this embodiment, the control device can obtain the target area of the target object in the home environment according to the pre-constructed environmental map corresponding to the home environment and the position of the target object in the environmental map. If there is no position information of the target object in the environmental map, the target area of the target object in the home environment can also be inferred based on the attributes of the target object using a large language model. For example, if the target object is fruit, the inferred target area of the fruit in the home environment is the kitchen.

[0116] Step 350: Send the target home task and the target area to the target embodied robot, so that the target embodied robot travels to the target area to execute the target home task on the target object.

[0117] In this embodiment, the control device can send the target home task and the target area to the target embodied robot. The target embodied robot receives and responds to the target home task and the target area, travels to the target area, and executes the target home task on the target object, realizing the control of the embodied robot to travel to the target area to execute the target home task based on the perceived target area of the target object in the home environment and the corresponding home task, without the need for the embodied robot to perceive the area where the target object in the home environment is located again, which is beneficial to improving the control efficiency of the embodied robot.

[0118] Please refer to Figure 5 , which shows a flowchart of a task execution method of an embodied robot provided in another embodiment of the present application. In a specific embodiment, the task execution method of the embodied robot can be applied to the embodied robot 100, or can also be applied to the control device 400 in the embodied robot control system as shown in Figure 1 . Taking the application to the control device 400 as an example, the process shown in Figure 5 will be elaborated in detail below. The task execution method of the embodied robot can include the following steps 410 to step 450.

[0119] Step 410: Obtain an initial home task instruction.

[0120] In this embodiment, the control device can obtain an initial home task instruction, and the initial home task instruction can be a voice instruction or a text instruction sent by the user, etc.

[0121] In some implementation manners, the control device can be configured with a voice recognition module. The user can send a voice instruction within the voice collection range of the voice recognition module. The voice recognition module collects the voice instruction issued by the user, performs voice recognition on the collected voice instruction, and determines the recognized voice instruction as the initial home task instruction.

[0122] In some embodiments, the control device may be configured with an operation panel through which a user can input text instructions by touch. The control device receives the text instructions through the operation panel and determines the text instructions as initial home task instructions.

[0123] In some embodiments, the embodied robot control system may further include a client that is connected to the control device through a network and performs data interaction with the control device through the network.

[0124] The user can send text instructions to the client. The client receives and responds to the text instructions, forwards the text instructions to the control device through the network, and the control device receives the text instructions forwarded by the client.

[0125] Step 420: Obtain initial environmental data corresponding to the home environment.

[0126] Step 430: Input the initial environmental data into the data processing model to obtain target environmental data.

[0127] Step 440: Input the initial home task instructions and the target environmental data into the task inference model to obtain a target home task.

[0128] In this embodiment, the control device can input the initial home task instructions and the target environmental data into the task inference model. The task inference model receives and responds to the initial home task and the target environmental data, understands the initial home task instructions, and identifies the states of various objects in the home environment according to the target environmental data, so as to output the target home task to the control device. For example, if the initial home task instruction is to do housework and the target environmental data is the image data corresponding to the home environment, the task inference model can output target home tasks such as straightening the throw pillows on the sofa, placing the trash can in the corridor into the kitchen, and throwing the tissue on the ground into the trash can according to the states of various objects in the image data.

[0129] In this embodiment, the control device receives the target home task output by the task inference model, reasons the corresponding home task based on the initial home task instructions and the target environmental data sent by the user, making the inferred home task more in line with the user's expectations, which is beneficial to improving the intelligence level of the embodied robot performing home tasks and enhancing the user experience.

[0130] Step 450: Send the target home task to the target embodied robot, so that the target embodied robot performs the target home task on the target object associated with the target home task.

[0131] In this embodiment, the content of step 450 can refer to the corresponding steps in the foregoing embodiments and will not be elaborated here.

[0132] Please refer to Figure 6, which shows a flowchart of a task execution method for an embodied robot provided by another embodiment of the present application. In a specific embodiment, the task execution method of the embodied robot can be applied to the embodied robot 100, or can also be applied to a control device 400 in an embodied robot control system as shown in Figure 1 below. Taking the application to the control device 400 as an example, the process shown in Figure 6 will be elaborated in detail. The task execution method of the embodied robot may include the following steps 510 to step 560.

[0133] Step 510: Obtain initial environmental data corresponding to the home environment.

[0134] Step 520: Input the initial environmental data into the data processing model to obtain target environmental data.

[0135] Step 530: Input the target environmental data into the task inference model to obtain the target home task.

[0136] In this embodiment, steps 510, 520, and 530 can refer to the content of the corresponding steps in the foregoing embodiments, and will not be elaborated here.

[0137] Step 540: Input the target home task into the task simulation model, so that the virtual embodied robot in the task simulation model executes the target home task in the three-dimensional model corresponding to the home environment.

[0138] Among them, a virtual embodied robot and a three-dimensional model corresponding to the home environment are pre-constructed in the task simulation model. Optionally, environmental data in the home environment can be collected in advance through sensors on the body of the embodied robot (such as lidar, camera, depth camera, encoder, IMU (Inertial Measurement Unit)), and a three-dimensional model can be generated according to the environmental data using technologies such as SLAM (Simultaneous Localization and Mapping) and VLM (Visual Localization and Mapping). Further, in the constructed three-dimensional model, a virtual robot is deployed to generate the task simulation model.

[0139] In this embodiment, the control device can input a target home task into the task simulation model. The virtual embodied robot in the task simulation model receives and responds to the target home task, and performs the target home task on the virtual object in the three-dimensional model to simulate the process of the target embodied robot performing the target home task on the target object in the real home environment. Among them, the virtual object corresponds to the target object (for example, if the target object is the door of the bedroom, correspondingly, the virtual object is the door of the bedroom in the three-dimensional model).

[0140] Step 550: Determine whether the virtual embodied robot has successfully performed the target home task.

[0141] Optionally, after the virtual embodied robot has performed the target home task, the control device can determine whether the virtual embodied robot has successfully performed the target home task through the evaluation model.

[0142] In this embodiment, the control device can obtain the scene image after the virtual embodied robot has performed the target home task. Specifically, the control device can capture the scene image of the three-dimensional model after the virtual embodied robot has performed the target home task. Among them, the scene image can include the virtual object corresponding to the target object. Further, the evaluation model (such as VLM) compares the scene image with the expected target image and outputs the similarity. The control device determines whether the virtual embodied robot has successfully performed the target home task according to the size relationship between the similarity and the similarity threshold. For example, the similarity threshold is set to 90%. When the similarity is greater than 90%, it is determined that the virtual embodied robot has successfully performed the target home task. Among them, the evaluation model has a pre-established correspondence relationship between the target home task and the expected target image.

[0143] Step 560: When it is determined that the virtual embodied robot has successfully performed the target home task, send the target home task to the target embodied robot, so that the target embodied robot performs the target home task on the target object associated with the target home task.

[0144] Optionally, when the control device determines that the virtual embodied robot has successfully performed the target home task on the virtual object, the pose of the current virtual object can be recorded as the target pose. Among them, the target pose can be used to represent the position and orientation of the virtual object in the three-dimensional model when the virtual embodied robot has successfully performed the target home task. In this embodiment, step 560 can refer to the content of the corresponding step in the foregoing embodiment and will not be elaborated here.

[0145] In this embodiment, the control device can determine whether the task simulation is successful based on the virtual embodied robot after the task is executed, so as to control the target embodied robot according to the simulation result when predicting the successful execution of the target home task, which is beneficial to further improving the accuracy of the embodied robot in executing home tasks.

[0146] In some embodiments, after step 550, when the control device determines that the virtual embodied robot fails to execute the target home task, it can update the target environment data according to the data of the virtual embodied robot's failure to execute the target home task. The updated target environment data includes the data of the virtual embodied robot's failure to execute the target home task and the target environment data output by the data processing model in step 520, and then return to execute step 530 and subsequent steps, so that the task inference model outputs a new target home task according to the updated target environment data until step 560 is executed.

[0147] Among them, the data of the virtual embodied robot's failure to execute the target home task can be the image data of intercepting the scene image of the three-dimensional model.

[0148] Optionally, if the number of times the virtual embodied robot fails to execute the target home task reaches a preset number threshold (set according to manual experience), it may be that the target home task cannot be achieved or exceeds the ability range of the embodied robot to execute tasks. At this time, the control device can report an error to the user.

[0149] In this embodiment, if the virtual embodied robot fails to execute the target home task in the task simulation model, it means that the target home task output by the task inference model is unreasonable. Therefore, the task inference model outputs a new target home task according to the updated target environment data, which is beneficial to further improving the accuracy of the embodied robot in executing home tasks in the real home environment.

[0150] In some embodiments, after step 560, the control device can determine whether the current pose of the target object matches the target pose of the target object; among them, the target pose is the pose of the virtual object when the virtual embodied robot successfully executes the target home task, and the virtual object corresponds to the target object; if the current pose matches the target pose, it is determined that the target embodied robot successfully executes the target home task; if the current pose does not match the target pose, it is determined that the target embodied robot fails to execute the target home task.

[0151] Among them, the current pose of the target object can be the position and direction of the target object in the home environment after the target embodied robot executes the target home task, that is, the pose of the target object in the home environment.

[0152] Optionally, if it is determined that the target embodied robot fails to execute the target home task, return to step 560 to cause the target robot to execute the target home task again until the current pose of the target object matches the target pose, and it is determined that the target embodied robot successfully executes the target home task.

[0153] Exemplarily, if the target home task is to open a door, the virtual object is the "door" in the 3D model. When it is determined that the virtual embodied robot successfully executes the task of opening the door in the 3D model, record the current pose of the "door" as the target pose. After the target embodied robot finishes executing the task of opening the door for the "door" in the real home environment, determine whether the current pose and the target pose of the "door" match. If they match, it means that the "door" in the real home environment has reached the opening and closing degree of the "door" in the 3D model, and it is determined that the target embodied robot successfully executes the target home task; if they do not match, it means that the "door" in the real home environment has not reached the opening and closing degree of the "door" in the 3D model, and it is determined that the process of the target embodied robot executing the target home task needs to be optimized again. Therefore, the target home task can be executed again.

[0154] In this embodiment, based on the pose of the virtual object when the virtual embodied robot successfully executes the target home task and the current pose of the target object after the target embodied robot finishes executing the target home task, the result of the target embodied robot executing the target home task in the real home environment is judged, which improves the accuracy of the target embodied robot executing the target home task.

[0155] Please refer to Figure 7 , which shows a task execution device 600 of an embodied robot provided by an embodiment of the present application. In a specific embodiment, the task execution device 600 of the embodied robot can be applied to the embodied robot 100, or can be applied to a control device 400 in an embodied robot control system as shown in Figure 1 . Taking the application to the control device 400 as an example, the task execution device 600 of the embodied robot shown in Figure 7 will be elaborated in detail. The task execution device 600 of the embodied robot may include a first acquisition module 610, a first input module 620, a second input module 630, and a sending module 640.

[0156] The first acquisition module 610 can be used to acquire initial environment data corresponding to the home environment; the first input module 620 can be used to input the initial environment data into the data processing model to obtain target environment data; the second input module 630 can be used to input the target environment data into the task inference model to obtain a target home task; the sending module 640 can be used to send the target home task to the target embodied robot, so that the target embodied robot executes the target home task on the target object associated with the target home task.

[0157] In some embodiments, the number of model parameters of the task inference model may be greater than that of the data processing model, and the difference in the number of parameters between the task inference model and the data processing model is at least not less than three orders of magnitude.

[0158] In some embodiments, the task execution device 600 of the embodied robot may further include a second acquisition module and a third input module.

[0159] The second acquisition module may be configured to send a target home task to the target embodied robot by the sending module 640, so as to obtain the current object state of the target object and the current robot state of the target embodied robot before the target embodied robot executes the target home task associated with the target object; the third input module may be configured to input the target home task, the current object state, and the current robot state into the action generation model to obtain target action information.

[0160] In some embodiments, the sending module 640 may include a first sending unit.

[0161] The first sending unit may be configured to send the target action information to the target embodied robot, so that the target embodied robot executes the corresponding target action on the target object according to the target action information to execute the target home task.

[0162] In some embodiments, the number of model parameters of the action generation model may be less than that of the task inference model, and the number of model parameters of the action generation model is greater than that of the data processing model. The difference in the number of parameters between the action generation model and the task inference model is at least not less than two orders of magnitude.

[0163] In some embodiments, the first acquisition module 610 may include an acquisition unit.

[0164] The acquisition unit may be configured to acquire initial environment data based on the first sensor and the second sensor; wherein, the first sensor may be installed on the target embodied robot, and the second sensor may be installed in the home environment.

[0165] In some embodiments, the target home task may include multiple subtasks, and the second input module 630 may include a first input unit.

[0166] The first input unit may be configured to input the target environment data into the task inference model to obtain multiple subtasks.

[0167] In some embodiments, the target embodied robot may include multiple sub-embodied robots, the target object may include multiple sub-objects, each subtask may be associated with a sub-object, and the task execution device 600 of the embodied robot may further include a first determination module.

[0168] The first determination module may be configured to send, by the sending module 640, a target home task to the target embodied robot, so as to determine, before the target embodied robot executes the target home task for the target object associated with the target home task, a sub-embodied robot corresponding to each subtask.

[0169] In some embodiments, the sending module 640 may further include a second sending unit.

[0170] The second sending unit may be configured to send each subtask to a corresponding sub-embodied robot, so that each sub-embodied robot executes an associated subtask for a corresponding sub-object.

[0171] In some embodiments, the sending module 640 may further include a first determination unit and a third sending unit.

[0172] The first determination unit may be configured to determine a target area of the target object in the home environment; the third sending unit may be configured to send the target home task and the target area to the target embodied robot, so that the target embodied robot travels to the target area to execute the target home task for the target object.

[0173] In some embodiments, the task execution device 600 of the embodied robot may further include a third acquisition module.

[0174] The third acquisition module may be configured to acquire an initial home task instruction before the second input module 630 inputs target environment data to the task inference model to obtain a target home task.

[0175] In some embodiments, the second input module 630 may further include a second input unit.

[0176] The second input unit may be configured to input the initial home task instruction and the target environment data to the task inference model to obtain a target home task.

[0177] In some embodiments, the task execution device 600 of the embodied robot may further include a fourth input module and a second determination module.

[0178] The fourth input module can be used to send a target home task to a target embodied robot by the sending module 640, so that before the target embodied robot executes the target home task on the target object associated with the target home task, the target home task is input into the task simulation model, so that the virtual embodied robot in the task simulation model executes the target home task in the three-dimensional model corresponding to the home environment; the second determination module can be used to determine whether the virtual embodied robot successfully executes the target home task.

[0179] In some embodiments, the sending module 640 may further include a fourth sending unit.

[0180] The fourth sending unit can be used to send the target home task to the target embodied robot when it is determined that the virtual embodied robot successfully executes the target home task, so that the target embodied robot executes the target home task on the target object associated with the target home task.

[0181] In some embodiments, the task execution device 600 of the embodied robot may further include an update module and a return module.

[0182] The update module can be used to update the target environment data according to the data of the virtual embodied robot's failure to execute the target home task after the second determination module determines whether the virtual embodied robot successfully executes the target home task. The updated target environment data may include the data of the virtual embodied robot's failure to execute the target home task and the target environment data; the return module can be used to return to the step of executing the input target environment data into the task inference model to obtain the steps and subsequent steps of the target home task, until it is determined that the virtual embodied robot successfully executes the target home task, and then send the target home task to the target embodied robot.

[0183] In some embodiments, the task execution device 600 of the embodied robot may further include a third determination module, a fourth determination module, and a fifth determination module.

[0184] The third determination module can be used to determine whether the current pose of the target object matches the target pose of the target object; wherein, the target pose can be the pose of the virtual object when the virtual embodied robot successfully executes the target home task, and the virtual object can correspond to the target object; the fourth determination module can be used to determine that the target embodied robot successfully executes the target home task when the current pose matches the target pose; the fifth determination module can be used to determine that the target embodied robot fails to execute the target home task when the current pose does not match the target pose.

[0185] It should be noted that the embodiments in this specification are all described in a progressive manner. Each embodiment focuses on the differences from other embodiments. For the same or similar parts among the embodiments, reference can be made to each other. For the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple. For the relevant parts, reference can be made to the corresponding description in the method embodiments. For any processing method described in the method embodiments, it can be implemented by the corresponding processing module in the device embodiments, and will not be elaborated one by one in the device embodiments.

[0186] In addition, in each embodiment of the present application, the various functional modules can be integrated into one processing module, or each module can exist physically alone, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules.

[0187] Please refer to Figure 8 , which shows a functional block diagram of an embodied robot 700 provided by an embodiment of the present application. The embodied robot 700 may include one or more of the following components: a memory 710, a processor 720, and one or more application programs. One or more application programs may be stored in the memory 710 and configured to be executed by one or more processors 720. One or more application programs are configured to execute the methods described in the foregoing method embodiments.

[0188] The memory 710 may include a Random Access Memory (RAM), or may also include a Read-Only Memory. The memory 710 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 710 may include a program storage area and a data storage area. Among them, the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as obtaining initial environment data, inputting initial environment data, obtaining target environment data, inputting target environment data, obtaining a target home task, sending a target home task, executing a target home task, obtaining the current object state, obtaining the current robot state, inputting a target home task, inputting the current object state, inputting the current robot state, obtaining target action information, training a deep learning neural network model, sending target action information, annotating historical home tasks, obtaining a training set, inferring historical home tasks, obtaining a task inference model, determining a sub-embodied robot, determining a target area, sending a target area, traveling to the target area, obtaining an initial home task instruction, and inputting an initial home task instruction, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data created during the use of the embodied robot 700 (such as a home environment, initial environment data, a data processing model, target environment data, a task inference model, a target home task, a target embodied robot, a target object, the number of model parameters, the current object state, the current robot state, an action generation model, target action information, historical action information, historical home information, a deep learning neural network model, historical home tasks, historical object states, historical robot states, target actions, a first sensor, a second sensor, historical environment data, historical home tasks, a training set, subtasks, sub-embodied robots, sub-objects, target areas, and initial home task instructions).

[0189] The processor 720 may include one or more processing cores. The processor 720 connects various parts within the embodied robot 700 using various interfaces and lines, and performs various functions of the embodied robot 700 and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 710, and by invoking data stored in the memory 710. Optionally, the processor 720 may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), or programmable logic array (PLA). The processor 720 may integrate a combination of one or several of a central processing unit (CPU), a graphics processing unit (GPU), and a modem, etc. Among them, the CPU mainly processes the operating system, user interface, application programs, etc.; the GPU is responsible for rendering and drawing display content; the modem is used to process wireless communication. It can be understood that the above-mentioned modem may not be integrated into the processor 720 and may be implemented separately through a communication chip.

[0190] Please refer to Figure 9 , which shows a functional block diagram of an embodied robot system 800 provided by an embodiment of the present application. The embodied robot system 800 may include a sensor 810 and a controller 820.

[0191] Among them, the sensor 810 may be used to collect initial environmental data corresponding to the home environment. The controller 820 may be used to execute the method described in the foregoing method embodiments.

[0192] Please refer to Figure 10 , which shows a structural block diagram of a computer-readable storage medium provided by an embodiment of the present application. Program code 910 is stored in the computer-readable storage medium 900, and the program code 910 can be called by a processor to execute the method described in the foregoing method embodiments.

[0193] The computer-readable storage medium 900 can be an electronic memory such as a flash memory, an EEPROM (Electrically Erasable Programmable Read-Only Memory), an EPROM, a hard disk, or a ROM. Optionally, the computer-readable storage medium 900 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 900 has a storage space for program code 910 that executes any of the method steps in the above-described method. These program codes can be read out from or written into one or more computer program products. The program code 910 can be compressed in a suitable form, for example.

[0194] Please refer to Figure 11 , which shows a structural block diagram of a computer program product 1000 provided by an embodiment of the present application. The computer program product 1000 includes computer programs / instructions 1010, and the computer programs / instructions 1010 are stored in a computer-readable storage medium of a computer device. When the computer program product 1000 runs on the computer device, the processor of the computer device reads the computer programs / instructions 1010 from the computer-readable storage medium, and the processor executes the computer programs / instructions 1010, so that the computer device executes the method described in the above method embodiment.

[0195] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A task execution method of an embodied robot, characterized in that: include: Obtaining initial environment data corresponding to the home environment; Inputting the initial environment data into a data processing model to obtain target environment data; Inputting the target environment data into the task reasoning model to obtain the target home task; The target home task is sent to a target embodied robot, so that the target embodied robot performs the target home task on a target object associated with the target home task.

2. The task execution method according to claim 1, characterized in that: The model parameter quantity of the task reasoning model is greater than the model parameter quantity of the data processing model, and the parameter quantity difference between the model parameter quantity of the task reasoning model and the model parameter quantity of the data processing model is at least not less than 3 orders of magnitude.

3. The task execution method according to claim 1, characterized in that: Before sending the target home task to the target embodied robot so that the target embodied robot performs the target home task on the target object associated with the target home task, the task execution method further includes: Acquiring a current object state of the target object and a current robot state of the target embodied robot; Inputting the target household task, the current object state and the current robot state into the action generation model to obtain target action information; The sending the target household task to the target embodied robot so that the target embodied robot performs the target household task on the target object associated with the target household task includes: The target action information is sent to the target embodied robot, so that the target embodied robot performs a corresponding target action on the target object according to the target action information to perform the target household task.

4. The task execution method according to claim 3, characterized in that: The model parameter amount of the action generation model is smaller than the model parameter amount of the task reasoning model, and the model parameter amount of the action generation model is larger than the model parameter amount of the data processing model, and the parameter difference between the model parameter amount of the action generation model and the model parameter amount of the task reasoning model is at least not less than 2 orders of magnitude.

5. The task execution method according to claim 1, characterized in that: The obtaining of initial environment data corresponding to the home environment includes: The initial environment data is acquired based on a first sensor and a second sensor; wherein the first sensor is installed on the target embodied robot, and the second sensor is installed in the home environment.

6. The task execution method according to any one of claims 1 to 5, characterized in that: The target home task includes a plurality of subtasks. The inputting the target environment data into the task reasoning model to obtain the target home task includes: The target environment data is input into the task reasoning model to obtain the multiple subtasks.

7. The task execution method according to claim 6, characterized in that: The target embodied robot includes a plurality of sub-embodied robots, the target object includes a plurality of sub-objects, each sub-task is associated with a sub-object, and before sending the target home task to the target embodied robot so that the target embodied robot performs the target home task on the target object associated with the target home task, the task execution method further includes: Determine a sub-embodied robot corresponding to each of the subtasks; The sending the target household task to the target embodied robot so that the target embodied robot performs the target household task on the target object associated with the target household task includes: Each of the subtasks is sent to the corresponding one of the sub-embodied robots, so that each of the sub-embodied robots performs an associated subtask on the corresponding one of the sub-objects.

8. The task execution method according to any one of claims 1 to 5, characterized in that: The sending the target household task to the target embodied robot so that the target embodied robot performs the target household task on the target object associated with the target household task includes: Determine a target area of ​​the target object in the home environment; The target household task and the target area are sent to the target embodied robot, so that the target embodied robot drives to the target area to perform the target household task on the target object.

9. The task execution method according to any one of claims 1 to 5, characterized in that: Before inputting the target environment data into the task reasoning model to obtain the target home task, the task execution method further includes: Get initial home task instructions; The step of inputting the target environment data into the task reasoning model to obtain the target home task includes: The initial home task instruction and the target environment data are input into the task reasoning model to obtain the target home task.

10. The task execution method according to any one of claims 1 to 5, characterized in that: Before sending the target home task to the target embodied robot so that the target embodied robot performs the target home task on the target object associated with the target home task, the task execution method further includes: Inputting the target home task into a task simulation model so that a virtual embodied robot in the task simulation model performs the target home task in a three-dimensional model corresponding to the home environment; determining whether the virtual embodied robot successfully performs the target household task; The sending the target household task to the target embodied robot so that the target embodied robot performs the target household task on the target object associated with the target household task includes: When it is determined that the virtual embodied robot has successfully performed the target home task, the target home task is sent to the target embodied robot, so that the target embodied robot performs the target home task on the target object associated with the target home task.

11. The task execution method according to claim 10, characterized in that: After determining whether the virtual embodied robot successfully performs the target household task, the task execution method further includes: When it is determined that the virtual embodied robot fails to perform the target home task, the target environment data is updated according to the data of the failure of the virtual embodied robot to perform the target home task, and the updated target environment data includes the data of the failure of the virtual embodied robot to perform the target home task and the target environment data; Return to the step of inputting the target environment data into the task reasoning model to obtain the target home task and subsequent steps, until it is determined that the virtual embodied robot has successfully executed the target home task, and then send the target home task to the target embodied robot.

12. The task execution method according to claim 10, characterized in that: After sending the target household task to the target embodied robot, the task execution method further includes: Determine whether the current posture of the target object matches the target posture of the target object; wherein the target posture is the posture of the virtual object when the virtual embodied robot successfully performs the target household task, and the virtual object corresponds to the target object; When the current posture matches the target posture, it is determined that the target embodied robot successfully performs the target household task; When the current posture does not match the target posture, it is determined that the target embodied robot fails to perform the target home task.

13. An embodied robot, characterized in that: include: Memory; One or more processors coupled to the memory; One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more application programs are configured to execute the task execution method according to any one of claims 1 to 12.

14. An embodied robot system, characterized in that: It includes a sensor and a controller, wherein the sensor is connected to the controller; The sensor is used to collect initial environmental data corresponding to the home environment; The controller is used to execute the task execution method as described in any one of claims 1 to 12.

Images