Method and device for executing user task, equipment and medium

Abstract

The invention provides a method and device for executing a user task, equipment and a medium. In one method, a user task is received from a user, the user task instructing a robotic device to classify a plurality of objects within a first range in a physical space. A plurality of text items respectively associated with a plurality of objects are identified from an image including the plurality of objects. For a first object of the plurality of objects, a first destination location of the first object in the physical space is determined based on a first text item of the plurality of text items associated with the first object. The robot device moves a first object to a first destination location. By utilizing the example implementation mode of the invention, the type of the object can be more accurately determined from the text item, so that a more accurate destination position is determined. Therefore, the robot equipment can execute the user task in the complex physical space, the flexibility and accuracy of the robot equipment in executing the task in the complex environment can be improved, and then the expected user task is completed.

Description

Methods, apparatuses, devices, and media for performing user tasks TECHNICAL FIELD

[0001] Exemplary implementations of the present disclosure generally relate to the field of robotics, and in particular, to methods, apparatuses, devices, and computer-readable storage media for performing user tasks using a robot. BACKGROUND

[0002] Robotics technology has been rapidly developed and has been widely used in multiple technical fields. Currently, various special-purpose robotic devices have been developed, for example, in an industrial environment, robots can be used to perform various tasks such as processing, grabbing, sorting, packaging, etc. For another example, in a home environment, a sweeping robot, a glass wiping robot, etc. have been developed. However, robots can usually only perform pre-set fixed tasks and cannot perform different user tasks according to user needs.

[0003] SUMMARY

[0004] In a first aspect of the present disclosure, a method for performing a user task is provided. In the method, a user task is received from a user, the user task instructing a robotic device to sort a plurality of objects within a first range in a physical space; a plurality of text items respectively associated with the plurality of objects are identified from an image including the plurality of objects; for a first object of the plurality of objects, a first destination position of the first object in the physical space is determined based on a first text item of the plurality of text items associated with the first object; and the robotic device moves the first object to the first destination position.

[0005] In a second aspect of the present disclosure, an apparatus for performing a user task is provided. The apparatus comprises: a receiving module configured to receive a user task from a user, the user task instructing a robotic device to sort a plurality of objects within a first range in a physical space; an identifying module configured to identify a plurality of text items respectively associated with the plurality of objects from an image including the plurality of objects; a determining module configured to, for a first object of the plurality of objects, determine a first destination position of the first object in the physical space based on a first text item of the plurality of text items associated with the first object; and a performing module configured to cause the robotic device to move the first object to the first destination position.

[0006] In a third aspect of the present disclosure, an electronic device is provided. The electronic device comprises: at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, causing the electronic device to perform the method according to the first aspect of the present disclosure.

[0007] In a fourth aspect of the present disclosure, a computer readable storage medium is provided, having stored thereon a computer program which, when executed by a processor, causes the processor to implement the method according to the first aspect of the present disclosure.

[0008] In a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the method according to the first aspect of the present disclosure.

[0009] It is to be understood that the particulars shown herein are by way of example and for purposes of illustrative discussion of the various embodiments of the present disclosure only and are not intended to limit the scope of the present disclosure to the particular embodiment illustrated. Other BRIEF DESCRIPTION OF DRAWINGS

[0010] In the following detailed description, reference will be made to the accompanying drawings, of which:

[0011] FIG. 1 shows a block diagram of an application environment according to one example implementation of the present disclosure;

[0012] FIG. 2 shows a block diagram for performing a user task according to some implementations of the present disclosure;

[0013] FIG. 3 shows a block diagram of an image acquisition process according to some implementations of the present disclosure;

[0014] FIG. 4 shows a block diagram of a process for a moving object according to some implementations of the present disclosure;

[0015] FIG. 5 shows a block diagram of a process for invoking a model according to some implementations of the present disclosure;

[0016] FIG. 6 shows a block diagram of a process for invoking an action model according to some implementations of the present disclosure;

[0017] FIG. 7 shows a flowchart of a method for performing a user task according to some implementations of the present disclosure;

[0018] FIG. 8 shows a block diagram of an apparatus for performing a user task according to some implementations of the present disclosure; and

[0019] FIG. 9 shows a block diagram of a device capable of implementing various implementations of the present disclosure. DETAILED DESCRIPTION

[0020] Implementations of the present disclosure will be described more fully hereinafter with reference to the accompanying drawings. While several implementations of the present disclosure are described, it should be understood that the present disclosure can be embodied in many other forms and should not be construed as limited to the implementations set forth herein; rather, these implementations are provided so that this disclosure will be thorough and complete, and fully convey the scope of the present disclosure to those skilled in the art. It should be understood that the drawings and implementations described are for illustrative purposes only and are not intended to limit the scope of the present disclosure.

[0021] In the description of implementations of the present disclosure, the term "includes" and its derivatives mean "including but not limited to". The term "based on" means "based at least in part on". The term "one implementation" or "the implementation" means "at least one implementation". The term "some implementations" means "at least some implementations". Other explicit or implicit definitions can also be included below. As used herein, the term "model" can represent the relationship between various data. For example, the above relationship can be obtained based on various technical solutions known at present and / or to be developed in the future.

[0022] It can be understood that the data involved in the technical solutions of the present disclosure (including but not limited to the data itself, the acquisition or use of the data) should comply with the requirements of relevant laws and regulations and relevant provisions.

[0023] It can be understood that before using the technical solutions disclosed by the implementations of the present disclosure, the type of personal information involved in the present disclosure, the scope of use, the scene of use, etc. should be informed to the user and the authorization of the user should be obtained through appropriate means according to relevant laws and regulations.

[0024] For example, in response to receiving the active request of the user, the user is sent prompt information to explicitly prompt the user that the operation requested to be executed will require the acquisition and use of the personal information of the user. Thus, the user can voluntarily choose whether to provide personal information to the software or hardware such as electronic device, application program, server or storage medium, etc. that executes the operation of the technical solutions of the present disclosure according to the prompt information.

[0025] As an optional but not limited implementation, in response to receiving the active request of the user, the way of sending prompt information to the user, for example, can be the way of pop-up window, and the prompt information can be presented in the form of text in the pop-up window. In addition, the pop-up window can also carry selection controls for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0026] It can be understood that the above notification and user authorization process is only illustrative, and does not limit the implementations of the present disclosure, and other ways that meet the relevant laws and regulations can also be applied to the implementations of the present disclosure.

[0027] The term "in response to" as used herein refers to a state in which a corresponding event occurs or a condition is satisfied. It will be understood that the timing of the execution of a subsequent action performed in response to the event or condition is not necessarily strongly correlated with the time at which the event occurs or the condition is satisfied. For example, in some cases, the subsequent action can be performed immediately upon the occurrence of the event or the satisfaction of the condition, while in other cases, the subsequent action can be performed after a period of time has elapsed since the occurrence of the event or the satisfaction of the condition.

[0028] Example Environment

[0029] In recent years, robotic technology and machine learning technology have been widely applied to multiple application scenarios. However, robots are generally only capable of performing pre-set fixed tasks and are not able to perform different user tasks according to user needs. In particular, in complex application environments, it is difficult for a robotic device to determine user needs and then perform corresponding tasks.

[0030] Simple robotic devices that perform specific tasks have been developed, however, such simple robotic devices are not able to understand complex user instructions and are not able to perform desired tasks according to user instructions in complex physical spaces. At this time, it is desirable to control the operation of the robot in an effective manner and then perform the desired task.

[0031] According to one example implementation of the present disclosure, a method for performing a user task is proposed. Referring to FIG. 1, an application environment according to one example implementation of the present disclosure is described, and FIG. 1 shows a block diagram 100 of an application environment according to one example implementation of the present disclosure. As shown in FIG. 1, a robotic device 110 and a user 120 can be located in a physical space 160, and the user 120 can control the robotic device 110 to perform multiple tasks. The physical space 160 can include, but is not limited to, one or more rooms. For example, in a home environment, the physical space 160 can include, but is not limited to, a living room, a bedroom, a study, a kitchen, a bathroom, etc., or a combination of one or more of the above.

[0032] As shown in FIG. 1, the robotic device 110 can include multiple parts. For example, a control unit 111 can serve as a control center of the robotic device 110, and an application program can be loaded into the control unit 111 in order to control various parts of the robotic device. The user 120 can use an interaction unit 112 to interact with the robotic device 110, for example, to input control instructions to the robotic device 110 in order to perform a desired task using the robotic device 110. The robotic device 110 can include an arm 113 for performing actions such as grasping, releasing, etc. For example, the arm 113 can grasp an object and move the object to a desired location, etc.

[0033] Alternatively and / or additionally, the robotic device 110 can further include a collection unit 114. Here, the collection unit 114 can include various types, e.g., an image collection unit, a sound collection unit, etc. Alternatively and / or additionally, the robotic device 110 can further include a sensing unit for detecting surrounding objects, e.g., can detect the distance of the robotic device from surrounding objects based on laser, etc. The robotic device 110 can further include a driving unit 115, e.g., the robotic device 110 can be deployed on a movable base, and the driving unit 115 can drive the wheels of the base to move along a desired path.

[0034] The physical environment 160 can include one or more collection units 130, …, and 132, e.g., one or more image collection devices can be deployed in a room to collect images of the room from various angles. The physical environment 160 can include a control device 140, which can control the one or more collection units 130, …, and 132, etc. via a network (not shown). Alternatively and / or additionally, in a smart home environment, the control device 140 can control various electrical appliances in the physical space 160.

[0035] Alternatively and / or additionally, a machine learning model (e.g., the model 150) can be provided to manage the physical space 160. It should be appreciated that although FIG. 1 shows the model 150 located inside the physical space 160, alternatively and / or additionally, the model 150 can be located at a remote device outside the physical space 160, and the control device 140, the robotic device 110, or other devices can access the remote model 150 via a network.

[0036] The model 150 can include one or more models. If the model 150 includes multiple models, the multiple models can include multiple types of models. The model 150 can include at least a language model (LM) and an action model, for example. The language model can have the ability to answer questions by learning from a large amount of corpus. The action model can control the robotic device 110 to perform various actions. The model 150 can further include an image recognition model, a text recognition model, etc., for example.

[0037] As shown in FIG. 1, the user 120 can instruct the robotic device 110 to operate various objects in the physical space 110. Here, the objects can be various items in a home environment, e.g., the user 120 can instruct the robotic device 110 to find a certain object in the physical space 160; as another example, the user 120 can instruct the robotic device 110 to place the found object to a designated location, etc.

[0038] Summary of performing a task

[0039] To at least partially address the deficiencies of the prior art, according to one example implementation of the present disclosure, a method for performing a user task is proposed. An overview of one example implementation of the present disclosure is described with reference to FIG. 2, which shows a block diagram 200 for performing a user task according to some implementations of the present disclosure.

[0040] As shown in FIG. 2, the robotic device 110 in the physical space 160 can receive a user task 210 from the user 120. At this time, the user task 210 can instruct the robotic device 110 to sort a plurality of objects in a specified range (e.g., the range 230, which can also be referred to as a first range) in the physical space 160. For example, the user 120 can speak "organize the items on the table" in natural language. In the example of FIG. 2, the first range is "the table", and the plurality of objects, i.e., the plurality of objects placed on the table, includes object 222 and object 224. The robotic device 110 can acquire an image including the plurality of objects. For example, the image can be acquired via at least any of the acquisition units 114, 130, …, and 132.

[0041] A plurality of text items respectively associated with the plurality of objects can be identified from the image including the plurality of objects. The text items may, for example, include text items on the plurality of objects. For example, taking object 224 as an example, object 224 is a bottled water, and the text items associated with object 224 may, for example, include text items on the packaging of the bottled water, text items on the label of the bottled water, such as "pure water", "coffee", "tea", and the like. The text items may, for example, also include text items associated with the plurality of objects that are found. For example, taking object 222 as an example, object 222 is a book. The name and author of the book can be determined (e.g., by identifying the cover of the book to determine the name and author of the book). Alternatively and / or additionally, the relevant text items of this book can be found online, which may, for example, include the background of the book, reviews and comments, author introduction, and the like.

[0042] For a first object of the plurality of objects, a first destination location of the first object in the physical space is determined based on a first text item of the plurality of text items that is associated with the first object. The first object can be any suitable object of the plurality of objects. The first destination location can be a location where the first object should be placed. For example, in a home environment, a bottle of water (i.e., object 224) and a book (i.e., object 222) are included on a table. The object 222 should be placed in a bookshelf, a study, or other location, and thus, if the object 222 is the first object, the first destination location can be the bookshelf (e.g., destination location 250) and / or the study. The object 224 should be placed in a refrigerator, a kitchen, or other location, and thus, if the object 224 is the first object, the first destination location can be the refrigerator and / or the kitchen. At this point, the robotic device 110 can be instructed to move the first object to the first destination location. For example, the robotic device 110 can be instructed to move the object 222 to the destination location 250.

[0043] According to some implementations of the present disclosure, the above-described method can be performed at any computing device having computing capability. For example, the above-described method can be performed with an application deployed at the robotic device 110. Alternatively and / or additionally, an application can be deployed at the control device 140 in order to perform the above-described method. In particular, the powerful processing capability of the model 150 can be invoked in order to find the first range that can include the plurality of objects and the destination location of each object of the plurality of objects in the physical space. In turn, the robotic device 110 can move along the path 242 to the range 230 and move the object 222 of the plurality of objects within the range 230 to its destination location 250 in the physical space 160 along the path 244.

[0044] With the example implementations of the present disclosure, the destination location of an object can be determined based on the text information of the object, and in turn, the robotic device can be controlled to move the plurality of objects within the first range to the corresponding destination locations. The accuracy of determining the destination locations can be improved, and the flexibility and accuracy of the robotic device to perform tasks in a complex environment can be improved, and in turn, the intended user task can be completed.

[0045] Detailed procedure of performing a task

[0046] Having described an overview of some implementations according to the present disclosure, in the following, more details regarding performing a user task will be described. For ease of description, in the following, more details regarding performing a user task will be described only by way of example with the control of the robotic device 110 to organize items on a table.

[0047] According to some implementations of the present disclosure, the image can be from at least any of: a capture device at the robotic device, a capture device in the first physical space, a capture device in the second physical space. More details of image capturing are described with reference to FIG. 3, which illustrates a block diagram 300 of an image capturing process according to some implementations of the present disclosure. As shown in FIG. 3, an image (e.g., one or more images 310) of the physical space 160 can be acquired from the capture unit 114 at the robotic device 110. Since the robotic device 110 can move freely in the physical space 160, the capture unit 114 can capture images of various locations in the physical space, thereby facilitating finding the first range and the plurality of objects.

[0048] Alternatively and / or additionally, an image of the physical space 160 can be acquired from the capture units 130, …, and 132. Here, the capture units 130, …, and 132 can be pre-deployed at designated locations within the physical space 160, e.g., at corner locations of the ceiling, etc. In this way, an image of the physical space 160 taken from a top-down perspective can be obtained, thereby facilitating an overall understanding of the layout of the physical space 160. This can help determine the first range and the first destination location corresponding to the first object within the first range.

[0049] According to some implementations of the present disclosure, an image of the physical space 160 where the robotic device 110 is located can be acquired, and the first range can be located based on the image. For example, if the user 120 speaks “organize items on the table” in natural language, the user task 210 instructs the robotic device 110 to sort the plurality of objects on the table. The range 230 where the table is located can be determined based on the image of the physical space 160. For example, a prompt word for locating the first range can be acquired, e.g., “please determine the location of the table in the following image,” and a model (e.g., the model 150) can be utilized to locate the range 230. The robotic device 110 can then be instructed to move to the range 230.

[0050] According to some implementations of the present disclosure, the image of the plurality of objects within the first range can be acquired at the same time as the image of the physical space 160 where the robotic device 110 is located. Illustratively, the image of the plurality of objects within the first range can be determined based on the image of the physical space 160. For example, the image of the plurality of objects within the first range can be cropped from the image of the physical space 160. Alternatively or additionally, according to some implementations of the present disclosure, the image of the plurality of objects within the first range can also be acquired in response to the robotic device 110 moving to the first range. Illustratively, the image of the plurality of objects within the first range can be acquired by the capture unit 114 at the robotic device 110 in response to the robotic device 110 moving to the first range.

[0051] According to some implementations of the present disclosure, the image of the plurality of objects can comprise one or more images. If the image of the plurality of objects comprises a plurality of images, each image can correspond to one object. For example, the image of the plurality of objects can comprise a plurality of images each corresponding to one of the plurality of objects. For a first object, the position of the first object can be adjusted so as to obtain an image of the first object. At this time, the position of the capturing device used to capture the image can be adjusted (e.g., an angle avoiding occlusion can be found) so as to obtain the image of the first object. For example, for the plurality of objects within the range 230, the position of the object 222 can be adjusted, and / or the position of the robotic device 110 can be moved so as to obtain an image of the object 222 only by means of the capturing unit 114 of the robotic device 110. In this way, an image corresponding to each of the plurality of objects can be obtained respectively, which can improve the accuracy of the image corresponding to each of the plurality of objects, and further improve the accuracy of subsequent performance of tasks based on the image.

[0052] If the image of the plurality of objects comprises only one image, it can be determined whether there is an occlusion relationship between the plurality of objects. Any suitable manner can be used to determine whether there is an occlusion relationship between the plurality of objects. For example, the model 150 can be used to determine whether there is an occlusion relationship between the plurality of objects. For another example, any suitable rule or algorithm can also be used to determine whether there is an occlusion relationship between the plurality of objects. Alternatively or additionally, the user 120 can be prompted to determine whether there is an occlusion relationship between the plurality of objects. The determination result provided by the user 120 can be received, and based on the determination result, it can be determined whether there is an occlusion relationship between the plurality of objects. Alternatively and / or additionally, if there is an occlusion relationship, the user can be instructed to eliminate the occlusion relationship.

[0053] If there is no occlusion relationship between the plurality of objects, an image comprising the plurality of objects can be obtained directly. If there is an occlusion relationship between the plurality of objects, the robotic device 110 can be instructed to move at least one of the plurality of objects in response to determining that there is an occlusion relationship between the plurality of objects, and then an image comprising the plurality of objects can be obtained. More details are described with reference to FIG. 4, which shows a block diagram 400 of a process of moving an object according to some implementations of the present disclosure. As shown in FIG. 4, in the image 410, the object 402 is located in front of the object 401 and occludes the object 401. At this time, the robotic device can be instructed to move the object 402 from the position 430 to a position that does not occlude the object 401 (e.g., the position 430’ in the image 420).

[0054] According to some implementations of the present disclosure, a target position can be determined, and the robotic device can be instructed to move the object 402 to the target position. For example, an action can be generated with the aid of the model 150 to control the robotic device to move the object 402 from the position 430 to the position 430'. In this way, the robotic device can be supported to handle complex problems in a complex environment, so as to perform the user task in a more accurate manner.

[0055] After the images corresponding to the plurality of objects are acquired, a first destination position of the first object in the physical space can be determined based on a first text item associated with the first object in the plurality of text items. According to some implementations of the present disclosure, a first type of the first object can be determined based on the first text item. As to the specific manner of determining the first type of the first object, any appropriate manner can be employed to determine the first type of the first object. For example, the first type of the first object can be determined based on a pre-stored database. The pre-stored database can store a plurality of text items associated with a plurality of objects and respective types of the plurality of objects. The first text item can be retrieved in the pre-stored database to determine the first type of the first object.

[0056] Alternatively and / or additionally, the extracted text item can be input to the model to determine the type of the object. At this time, the prompt word can be expressed as: "The following text is extracted from the item. Please determine the type of the item: XXX, author XXX". For another example, the prompt word can be expressed as: "Please extract the text from the following image and determine the type of the item", and the like.

[0057] According to some implementations of the present disclosure, a position of a second object of the first type in the physical space can also be determined to determine the first destination position based on the position of the second object. It can be understood that any appropriate manner can be employed to determine the position of the second object. For example, the position of the second object can be determined based on the image of the physical space 160 where the robotic device 110 is located. Taking the first object as a book for example, the image of the physical space 160 where the robotic device 110 is located can be acquired, and the positions of other books in the physical space 160 can be determined based on the image. Taking the example that all the other books are placed in a bookshelf, the position of the second object can be determined as the bookshelf. For another example, the position of the second object can also be determined based on a predetermined rule or algorithm. Still taking the first object as a book for example, if the predetermined rule indicates that the book is placed in a study, the position of the second object can be determined as the study based on the predetermined rule. It can be understood that the position of the second object can also be determined with the aid of a model (e.g., the model 150), manually determined by the user 120, and the like. For example, the robotic device 110 can ask the user: "Where to put the book", and the book can be placed at the position specified by the user.

[0058] If the position of the second object is to be determined with the aid of the model, according to some implementations of the present disclosure, a prompt can be constructed based on the image of the physical space 160 and the first type of the first object, which can be used to determine the position of the object of the first type in the image of the physical space 160. Taking the first type of the first object determined to be a book as an example, the prompt can be expressed as: “Please identify the position where the ‘book’ is placed from the following image”.

[0059] The prompt can be provided to the machine learning model (e.g., the model 150) to determine the position of the second object of the first type with the aid of the machine learning model. The response of the machine learning model to the prompt, i.e., the model output of the machine learning model output for the prompt, can indicate the position of the second object (e.g., the bookshelf). Thus, the position of the second object, i.e., the first destination position of the first object, can be determined based on the response of the machine learning model to the prompt.

[0060] Specifically, the machine learning model can process the image, and in the case that the image includes the second object, the model can output the position where the second object is located (e.g., the region coordinates of the second object in the image, and / or directly output the image of the region where the second object is located, etc.). If the image does not include the second object, the model can output a response such as “not found”. According to some implementations of the present disclosure, whether the image includes the second object can be detected in a variety of ways, thereby improving the performance of the robotic device in detecting the position of the second image.

[0061] Alternatively and / or additionally, the prompt can be further obtained based on the image and the extracted text item (e.g., the book name XXX, the author XXX), which can be expressed as: “Please identify the position where the ‘book’ is placed from the following image, the book name XXX, the author XXX”. In this way, the refined type of the book can be further determined, e.g., an English book, a physical book, etc., and thus the specific placement position in the bookshelf can be determined. For example, the destination position of the English book can be square 1 in the bookshelf, the destination position of the physical book can be square 2 in the bookshelf, etc. Alternatively and / or additionally, the model can perform a further search based on the received prompt to determine more details of the object, and thus determine the destination position of each object in a more accurate manner.

[0062] Alternatively or additionally, according to some implementations of the present disclosure, the other physical space associated with the physical space 160 can also be determined in response to determining that the image represents that the second object is not included in the physical space 160 where the robotic device 110 is located. In this case, the physical space 160 where the robotic device 110 is located can be referred to as a first physical space, and the other physical space associated with the physical space 160 can be referred to as a second physical space. It should be understood that the second physical space herein is a potential physical space that can include the first object. Exemplarily, taking the book as the first object, in a home environment, since the book can be placed in a bookshelf, the second physical space can be determined as the bookshelf. Specifically, in the process of determining the second physical space associated with the first physical space, the prompt for locating the second object can be obtained based on the image and the first type of the physical space 160, and the response of the machine learning model to the prompt can be received in order to determine the second physical space. After the second physical space that can include the second object is determined, the position of the second physical space in the first physical space can be determined as the position of the second object, i.e., the first destination position of the first object.

[0063] More details about determining the second physical space are described with reference to FIG. 5, which shows a block diagram 500 of a process of invoking a model according to some implementations of the present disclosure. As shown in FIG. 5, based on the user task 210, it can be determined that a plurality of objects on the desktop (e.g., bottled water and a book) are to be moved to the corresponding positions. For the book, the corresponding prompt 510 can be obtained based on the image 310. The prompt 510 can be expressed as, for example, “Please determine the physical space that can include the ‘book’ from the following image”, or as, for example, “In the following image, where can the ‘book’ be placed”, and the like. The prompt 510 and the image 310 can be input to a language model 520, so that the language model 520 finds the second physical space (e.g., a bookshelf) that can include the book from the image 310. The language model 520 can be, for example, one of the models included in the model 150.

[0064] According to some implementations of the present disclosure, after determining the first destination location, a message associated with the first object can be provided to the user 120. The message can be used to prompt the user 120 to confirm whether to move the first object to the first destination location. As to the specific way of providing the message, the message can be provided to the user 120 via a display screen of the robotic device 110, a voice playing device, etc. The message can also be provided in response to receiving a response from the user 120 to the message, and determining whether to move the first object to the first destination location based on the response. If the response indicates to determine to move the first object to the first destination location, the robotic device 110 can be instructed to move the first object to the first destination location. In this way, the object can be moved only after obtaining the user confirmation, and the accuracy of the robotic device performing the task can be improved according to the user indication.

[0065] According to some implementations of the present disclosure, a motion trajectory from the location of the first object to the first destination location can also be determined based on the image of the physical space 160. It can be appreciated that the motion trajectory can also be determined in any suitable manner, such as by the model 150, by a human, based on predetermined rules or algorithms, etc. The motion trajectory can be used to instruct the robotic device 110 to avoid obstacles in the physical space 160, for example, and to move the first object to the first destination location in a shorter path. After determining the motion trajectory, the robotic device 110 can be instructed to move the first object to the first destination location according to the motion trajectory.

[0066] According to some implementations of the present disclosure, the actions to be performed by the robotic device 110 can be determined by the model 150, and the robotic device 110 can be controlled to perform the actions by the model 150. For example, if the first destination location is a bookshelf, the specific way of moving the first object to the bookshelf (e.g., opening the bookshelf, putting the first object in the bookshelf, etc.) can be determined by the model 150. Illustratively, a prompt can be constructed to ask the model 150 how to open the bookshelf. The prompt can be expressed as, for example, “determine the way to open the bookshelf from the following image,” and the prompt and the corresponding image (e.g., an image of the physical environment 160 including the bookshelf) can be sent to the model 150. The model 150 can return, for example, “pull the handle to open the bookshelf door.” In turn, the robotic device 110 can be instructed to pull the handle to open the bookshelf. In this way, the powerful processing capability of the model can be utilized to solve unknown problems in complex environments according to some implementations of the present disclosure, and thus determine the actions to be performed by the robotic device. In this way, the ability of the robotic device to handle complex tasks can be improved, and thus the user task can be performed in a more accurate manner.

[0067] According to some implementations of the present disclosure, a motion model can be utilized to determine the specific motion to be performed by the robotic device. More details are described with reference to FIG. 6, which illustrates a block diagram 600 of a process of invoking a motion model according to some implementations of the present disclosure. As shown in FIG. 6, a motion model 630 can be provided, which can be a pre-trained and fine-tuned model that can determine the specific motion to be performed by the robotic device based on the current state of the robotic device and the instruction. The motion model 630 can be the same as the model 150.

[0068] The current state 620 can include data of multiple aspects, such as an image of the robotic device 110, an image of the environment of the robotic device 110, pose data of the robotic arm (e.g., positions of various joints of the robotic arm (POS1, …)), and a state of a tool (e.g., a gripper, a cutter, etc.) fixed at the end of the robotic arm. For example, 0 can be used to represent a closed state of the gripper, and 1 can be used to represent an open state of the gripper. The instruction and the current state can be input to the motion model 630, and the motion model can be utilized to determine the motion to be performed by the robotic device 110 based on the instruction and the current state. Here, the motion can represent the difference between the current pose of the robotic device 110 and the next pose, and the difference between the current state of the tool and the next state, etc.

[0069] The instruction 610 (e.g., “open the bookcase”) can be input to the motion model 630, where the instruction 610 can be represented in natural language and can be determined from the response of the language model 630. Further, the current state of the robotic device 110 can be obtained, and the motion model 630 can determine the corresponding motion 640 based on the input data. For example, the orientations, positions, velocities, accelerations, etc. of various joints in the arm, and / or wheels and / or other movable devices of the robotic device at the next time point can be determined. Further, the determined motion 640 can be utilized to control the state of the robotic device 110 at the next time point.

[0070] In this way, the motion of the robotic device can be controlled by means of the model, the robotic device can be instructed to move the first object based on the model output, the motion of the robotic device can be accurately controlled, and the user task can be performed with higher efficiency.

[0071] It can be understood that a plurality of destination locations corresponding to the plurality of objects in the first range respectively can be determined. The robot device 110 can be controlled to move the plurality of objects to the corresponding destination locations in sequence in a similar manner as described above. For example, the robot device 110 can be controlled to move the books to the bookshelf in sequence, and then move the bottled water to the refrigerator, and so on. It should be noted that if there are at least two objects of the same type in the plurality of objects, the at least two objects can be moved together. For example, if the plurality of objects includes a group of objects of a first type, and the group of objects includes at least two objects, the robot device 110 can be instructed to move the group of objects at one time. In this way, a group of objects can be moved at one time according to the types of different objects in the plurality of objects, and the efficiency of the robot device 110 in moving objects can be improved.

[0072] Alternatively or additionally, according to some implementations of the present disclosure, if it is determined that the number of a group of objects of a first type in the plurality of objects satisfies a threshold condition, the robot device 110 can be instructed to obtain a third object. The threshold condition may, for example, indicate a threshold number of the group of objects. For example, if the threshold number is 4, the robot device 110 can be instructed to obtain a third object in response to the number of objects included in the group of objects of the first type reaching 4. The third object may, for example, be another object used to move the group of objects. The third object can be a predetermined object. For example, the user 120 can pre-set that the robot device 110 can move the plurality of objects by means of a specified object. For example, taking a group of books as an example, the third object can be a tray, a basket, a bag, a trailer, or any object that can help move the group of books. Further, the robot device 110 can be instructed to move the group of objects to the first destination location via the third object. For example, if the third object is a tray, the robot device 110 can be instructed to move the group of objects (e.g., a group of books) to the first destination location (e.g., a bookshelf) by means of the tray.

[0073] According to some implementations of the present disclosure, after moving the at least one object of the first type to the first destination position, a destination position of the at least one object of the second type can be determined and the at least one object of the second type can be moved to the second destination position of the object of the second type in the physical space. For example, the manner of leaving the first destination position, the movement trajectory of re-approaching the first range, the movement trajectory of moving from the first range to the second destination, etc. can be determined with the aid of the model 150. A new image can be captured and a new prompt word can be constructed to query the model 150 (e.g. the language model) for the next instruction. The prompt word can be expressed as, for example, “please determine the next instruction based on the following image”, “what to do next”, etc. Illustratively, when the robotic device has placed the book into the bookshelf, the language model can return “close the bookshelf”, at which point a corresponding action can be generated based on the instruction “close the bookshelf” and the current state of the robotic device to instruct the robotic device to close the bookshelf.

[0074] According to some implementations of the present disclosure, the priorities of the plurality of objects in the first range can also be determined and the plurality of objects can be moved in sequence based on the priorities. The priorities of the plurality of objects can be determined in any manner, for example, the priorities of the plurality of objects can be determined with the aid of the model 150 or predetermined rules. Illustratively, the priorities of objects that need to be refrigerated (e.g. ice cream, frozen meat, etc.) can be higher than the priorities of objects that can be stored at room temperature. In this way, objects corresponding to higher priorities can be moved first, improving the quality of the robotic device 110 performing the task.

[0075] It should be appreciated that although the above describes one example implementation of the present disclosure in a Chinese language environment. Alternatively and / or additionally, the technical solutions of one example implementation of the present disclosure can be performed in a plurality of language environments. For example, the robot can be controlled in a Chinese, English, Japanese, French, etc. environment. In particular, the robot can be controlled in different language application environments based on the multi-language capabilities provided by the machine learning technology. Further, although the above describes the process of performing a user task with the robotic device using the example of placing a book, alternatively and / or additionally, the robotic device can be controlled to perform other user tasks, for example, finding other items in a room, placing a certain item to a specified location, etc.

[0076] According to some implementations of the present disclosure, a user can interact with the robotic device via language, motion, gesture, etc. For example, the user can speak out a user task desired to be performed, predefine a certain motion to specify the user task, etc. Specifically, the user can make a motion of putting an object, which can be used as a trigger for the robotic device to perform a user task of sorting the object. When the motion is recognized from the captured image sequence, the robotic device can automatically ask the user whether the user task of sorting the object is needed and ask the user about the scope to be handled, and in case of a positive reply, the robotic device can perform the user task.

[0077] Alternatively and / or additionally, the user can interact with the robotic device via the interaction unit 112, e.g., the user inputs a task in text and / or image representation, and controls the robotic device to perform the task. Alternatively and / or additionally, the user can specify a condition for performing the task, e.g., to perform the task immediately, to perform the task after a predetermined time, or to perform the task upon determining that a predetermined condition is met (e.g., after the user has dinner), etc.

[0078] According to some implementations of the present disclosure, the robotic device can provide a variety of messages to the user, e.g., for a first object (e.g., a book), if multiple first destination locations (e.g., a bookshelf and a study) are included in the physical space, the robotic device can ask the user whether the book is needed to be moved to the bookshelf or to the study. For another example, assuming that the robotic device does not find the first destination location, the robotic device can ask the user where to move the first object, etc.

[0079] According to some implementations of the present disclosure, a variety of positioning algorithms can be utilized to determine the positions of the robotic device, and the objects in the physical environment. For example, a global positioning system (GPS) can be deployed at the robotic device, and satellite signals can be used to determine the precise position of the robotic device. Alternatively and / or additionally, a communication unit can be deployed at the robotic device, the position of the robotic device can be determined by means of signals between the communication unit and a base station, and utilizing a communication network. Alternatively and / or additionally, Wi-Fi access points can be deployed in the physical space, and the communication unit at the robotic device can interact with the Wi-Fi hotspots to determine the position via Wi-Fi signal strength and known positions of the Wi-Fi access points. Alternatively and / or additionally, the communication unit at the robotic device can support Bluetooth functionality, in which case Bluetooth signals and known positions of Bluetooth devices can be used to determine the positions of nearby devices. An inertial navigation system can be deployed at the robotic device, and accelerometers and gyroscopes can be used to measure and calculate the movement and orientation of the device in space, and in turn determine the position of the robotic device.

[0080] Alternatively and / or additionally, a visual positioning system can be used to determine the position of the robotic device and / or the individual objects. A map of the physical space can be pre-acquired, and the positions of the individual objects can be marked in the map. The robotic device can detect the distance from the surrounding objects with the echo detection unit, and determine the specific positions of the individual objects in combination with the acquired images and the map of the physical space. Specifically, computer aided design (CAD) and geographic information system (GIS) can be used, and a positioning algorithm can be used to determine the positions. Alternatively and / or additionally, a tracking unit can be deployed at important objects in the physical space, for example, a tracking unit can be added at the remote controller of a household appliance (e.g., a television remote controller, an air conditioner remote controller), so that the robotic device can timely acquire the accurate positions of the important objects, and the like.

[0081] According to some implementations of the present disclosure, the original position of the robotic device itself and the destination position to which the robotic device is expected to go can be determined based on the methods described above. The robotic device can determine a path from the original position to the destination position. For example, the surrounding environment images can be constantly acquired, and the path can be constantly updated while ensuring to avoid obstacles, and the robotic device can be caused to move along the path to the destination position.

[0082] According to some implementations of the present disclosure, after reaching the destination position, the robotic device can perform a specified task. For example, a specified object can be acquired and moved to a corresponding position. The constraint conditions, i.e., the constraint conditions that should be followed during the execution of the task, can be determined with the language model and / or the knowledge base. For example, the image and the corresponding prompt word can be acquired, and the image and the prompt word can be input to the language model, and then the constraint conditions can be received from the language model. For example, the prompt word can be determined as: “please determine the constraint conditions that should be followed during the movement of the XXX object based on the following image”, or “please determine the precautions during the movement of the XXX object”, and the like.

[0083] At this time, it can be determined that the original posture of the object (e.g., bottled water, a plate, a bowl, and the like) should be maintained (e.g., the vertical direction should be maintained, and the object should not be tilted) during the movement of the object. Further, the constraint conditions can be input to the action model, and a series of actions output by the action model will perform the corresponding task while ensuring the constraint conditions. With some implementations of the present disclosure, the safety during the operation of the robotic device can be ensured, so that the accidental damage to an object can be avoided, and the like.

[0084] With the example implementations of the present disclosure, a robotic device can perform a user task in a complex physical space. In this way, the robotic device can be controlled to move a plurality of objects within a first range to corresponding destination locations. The flexibility and accuracy of the robotic device to perform a task in a complex environment can be improved, thereby completing an intended user task.

[0085] Example process

[0086] FIG. 7 illustrates a flowchart of a method 700 for performing a user task, according to some implementations of the present disclosure. At block 710, a user task is received from a user, the user task instructing a robotic device to sort a plurality of objects within a first range in a physical space. At block 720, a plurality of text items respectively associated with the plurality of objects are identified from an image including the plurality of objects. At block 730, for a first object of the plurality of objects, a first destination location of the first object in the physical space is determined based on a first text item of the plurality of text items associated with the first object. At block 740, the robotic device moves the first object to the first destination location.

[0087] According to some implementations of the present disclosure, the image is determined based on: in response to determining that there is an occlusion relationship between the plurality of objects, the robotic device moves at least one object of the plurality of objects; and obtaining the image including the plurality of objects.

[0088] According to some implementations of the present disclosure, the image of the first object is determined based on: adjusting a position of the first object so as to obtain the image of the first object; or adjusting a position of a capturing device used to capture the image so as to obtain the image of the first object.

[0089] According to some implementations of the present disclosure, the method 700 further includes: obtaining an image of the physical space in which the robotic device is located; locating the first range based on the image; and the robotic device moving to the first range.

[0090] According to some implementations of the present disclosure, determining the first destination location includes: determining a first type of the first object based on the first text item; determining a position of a second object of the first type in the physical space; and determining the first destination location based on the position of the second object.

[0091] According to some implementations of the present disclosure, determining the position of the second object includes at least one of: identifying the second object in the image of the physical space to determine the position of the second object.

[0092] According to some implementations of the present disclosure, determining the position of the second object includes: constructing a prompt based on the image of the physical space and the first type of the first object, the prompt indicating to determine the position of the object of the first type in the image of the physical space; and determining the position of the second object based on a response to the prompt by the machine learning model.

[0093] According to some implementations of the present disclosure, moving the first object to the first destination position by the robotic device includes: determining a motion trajectory from the position of the first object to the first destination position based on the image of the physical space; and moving the first object to the first destination position by the robotic device according to the motion trajectory.

[0094] According to some implementations of the present disclosure, moving the first object to the first destination position by the robotic device includes: providing a message associated with the first object to the user; and moving the first object to the first destination position by the robotic device in response to receiving an answer to the message from the user.

[0095] According to some implementations of the present disclosure, moving the first object to the first destination position by the robotic device includes: in response to determining that a number of a group of objects of the first type in the plurality of objects satisfies a threshold condition, obtaining a third object by the robotic device; and moving the group of objects to the first destination position by the robotic device via the third object.

[0096] Example apparatuses and devices

[0097] FIG. 8 illustrates a block diagram of an apparatus 800 for performing a user task according to some implementations of the present disclosure. The apparatus 800 includes: a receiving module 810 configured to receive a user task from a user, the user task indicating a robotic device to sort a plurality of objects within a first range in a physical space; an identifying module 820 configured to identify a plurality of text items respectively associated with the plurality of objects from an image including the plurality of objects; a determining module 830 configured to determine, for a first object in the plurality of objects, a first destination position of the first object in the physical space based on a first text item in the plurality of text items associated with the first object; and a performing module 840 configured to cause the robotic device to move the first object to the first destination position.

[0098] According to some implementations of the present disclosure, the image is determined based on: in response to determining that there is an occlusion relationship between the plurality of objects, causing the robotic device to move at least one object in the plurality of objects; and obtaining the image including the plurality of objects.

[0099] According to some implementations of the present disclosure, the image of the first object is determined based on adjusting a position of the first object so as to obtain the image of the first object, or adjusting a position of a capturing device used to capture the image so as to obtain the image of the first object.

[0100] According to some implementations of the present disclosure, the execution module 840 is further configured to: obtain an image of a physical space where the robotic device is located; locate the first range based on the image; and cause the robotic device to move to the first range.

[0101] According to some implementations of the present disclosure, the determination module 830 is further configured to: determine the first type of the first object based on the first text item; determine a position of a second object of the first type in the physical space; and determine the first destination position based on the position of the second object.

[0102] According to some implementations of the present disclosure, the determination module 830 is further configured to: identify the second object in the image of the physical space to determine the position of the second object.

[0103] According to some implementations of the present disclosure, the determination module 830 is further configured to: construct a prompt based on the image of the physical space and the first type of the first object, the prompt indicating to determine a position of an object of the first type in the image of the physical space; and determine the position of the second object based on a response to the prompt by the machine learning model.

[0104] According to some implementations of the present disclosure, the execution module 840 is further configured to: determine a motion trajectory from the position of the first object to the first destination position based on the image of the physical space; and cause the robotic device to move the first object to the first destination position according to the motion trajectory.

[0105] According to some implementations of the present disclosure, the execution module 840 is further configured to: provide a message associated with the first object to the user; and in response to receiving an answer to the message from the user, cause the robotic device to move the first object to the first destination position.

[0106] According to some implementations of the present disclosure, the execution module 840 is further configured to: in response to determining that a number of a group of objects of the first type in the plurality of objects satisfies a threshold condition, cause the robotic device to obtain a third object; and cause the robotic device to move the group of objects to the first destination position via the third object.

[0107] FIG. 9 illustrates a block diagram of a device 900 that can implement a number of implementations of the present disclosure. It should be understood that the computing device 900 illustrated in FIG. 9 is merely an example and should not be construed to limit the functionality and scope of the implementations described herein. The computing device 900 illustrated in FIG. 9 can be used to implement the methods described above.

[0108] As illustrated in FIG. 9, the computing device 900 is in the form of a general- purpose computing device. Components of the computing device 900 can include, but are not limited to, one or more processors or processing units 910, a memory 920, a storage device 930, one or more communication units 940, one or more input devices 950, and one or more output devices 960. The processing unit 910 can be a real or virtual processor and is capable of executing various processing in accordance with programs stored in the memory 920. In a multi-processing system, multiple processing units execute computer-executable instructions in parallel to improve the processing power of the computing device 900.

[0109] The computing device 900 typically includes a plurality of computer storage media. Such media can be volatile and nonvolatile media and removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules, and other data. The memory 920 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 930 can be a removable or non-removable media and can include machine readable media such as flash drives, disks, or any other media capable of storing information and / or data (e.g., training data for training) and accessible by the computing device 900.

[0110] The computing device 900 can further include additional removable / non-removable, volatile / non-volatile storage media. Although not shown in FIG. 9, a disk drive and a disk drive interface can be provided for reading from or writing to a removable, non-removable, volatile, or non-volatile media such as a floppy disk, a ZIP® disk, a magnetic tape, or an optical disk. In these cases, each drive can be connected to the bus by one or more data media interfaces. The memory 920 can include a computer program product 925 having one or more program modules configured to carry out the various methods or actions of the implementations of the present disclosure.

[0111] The communication units 940 enable communications with other computing devices over a communication medium. Additionally, the functionality of the components of the computing device 900 can be implemented in a single computing cluster or a plurality of computer machines that are capable of communicating with each other through a communication connection. Thus, the computing device 900 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network nodes in the networking environment.

[0112] The input device 950 can be one or more input devices, such as a mouse, a keyboard, a trackball, etc. The output device 960 can be one or more output devices, such as a display, a speaker, a printer, etc. The computing device 900 can also communicate with one or more external devices (not shown) such as a storage device, a display device, etc. through the communication unit 940, as needed, communicate with one or more devices that enable a user to interact with the computing device 900, or any devices (e.g., a network card, a modem, etc.) that enable the computing device 900 to communicate with one or more other computing devices. Such communication can be carried out through an input / output (I / O) interface (not shown).

[0113] According to an example implementation of the present disclosure, a computer readable storage medium is provided, having stored thereon computer executable instructions, wherein the computer executable instructions are executed by a processor to implement the method described above. According to an example implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer readable medium and includes computer executable instructions, wherein the computer executable instructions are executed by a processor to implement the method described above. According to an example implementation of the present disclosure, a computer program product is provided, having stored thereon a computer program, which when executed by a processor implements the method described above.

[0114] Various aspects of the disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, and computer program products according to this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0115] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0116] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0117] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0118] The implementations of the disclosure have been described above with the intent to be illustrative rather than limiting. Although being shown in only a few of the various implementations, the principle of each implementation can be extended to any other implementation. Some of the present implementations have also been described with the intent to be illustrative rather than restrictive. Many modifications and variations of the described implementations are possible in light of the above teachings. It is therefore contemplated that the application can encompass modifications and variations provided they come within the scope of the appended claims. It is also contemplated that the implementing specific electric circuitry such as, for example, application specific integrated circuits (ASICs) can be configured to implement one or more of the processes described herein. The choice of language in the claims is intended to be interpreted as limiting only when the language claims a combination of means plus function unless otherwise explicitly provided.

Claims

1. A method for performing a user task, comprising: receiving a user task from a user, the user task instructing a robotic device to sort a plurality of objects within a first range in a physical space; identifying a plurality of text items respectively associated with the plurality of objects from an image comprising the plurality of objects; determining, for a first object of the plurality of objects, a first destination location of the first object in the physical space based on a first text item of the plurality of text items associated with the first object; and moving, by the robotic device, the first object to the first destination location.

2. The method of claim 1, wherein the image is determined based on: moving, by the robotic device, at least one object of the plurality of objects in response to determining that there is an occlusion relationship between the plurality of objects; and capturing the image comprising the plurality of objects.

3. The method of claim 1, wherein the image of the first object is determined based on: adjusting a position of the first object so as to capture the image of the first object; or adjusting a position of a capturing device used to capture images so as to capture the image of the first object.

4. The method of claim 1, further comprising: capturing an image of a physical space in which the robotic device is located; locating the first range based on the image; and moving, by the robotic device, to the first range.

5. The method of claim 1, wherein determining the first destination location comprises: determining a first type of the first object based on the first text item; determining a location of a second object of the first type in the physical space; and determining the first destination location based on the location of the second object.

6. The method of claim 5, wherein determining the location of the second object comprises at least either of: identifying the second object in an image of the physical space to determine the location of the second object.

7. The method of claim 5, wherein determining the location of the second object comprises: constructing, based on an image of the physical space and the first type of the first object, a prompt word instructing to determine a location of an object of the first type in the image of the physical space; and determining the location of the second object based on a response to the prompt word by a machine learning model.

8. The method of claim 1, wherein moving, by the robotic device, the first object to the first destination location comprises: determining, based on an image of the physical space, a motion trajectory from a location of the first object to the first destination location; and moving, by the robotic device, the first object to the first destination location following the motion trajectory.

9. The method of claim 1, wherein moving, by the robotic device, the first object to the first destination location comprises: providing, to the user, a message associated with the first object; and ​ ​ ​ ​ ​ In response to receiving an answer to the message from the user, the robotic device moves the first object to the first destination location.

10. The method of claim 5, wherein the robotic device moving the first object to the first destination location comprises: In response to determining that a number of a group of objects of the first type among the plurality of objects satisfies a threshold condition, the robotic device obtains a third object; The robotic device moves the group of objects to the first destination location via the third object.

11. An apparatus for performing a user task, comprising: a receiving module configured to receive a user task from a user, the user task instructing a robotic device to sort a plurality of objects within a first range in a physical space; an identifying module configured to identify a plurality of text items respectively associated with the plurality of objects from an image comprising the plurality of objects; a determining module configured to determine, for a first object among the plurality of objects, a first destination location of the first object in the physical space based on a first text item associated with the first object among the plurality of text items; and an executing module configured to cause the robotic device to move the first object to the first destination location.

12. An electronic device, comprising: at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions when executed by the at least one processing unit cause the electronic device to perform the method according to any one of claims 1-10.

13. A computer-readable storage medium having stored thereon a computer program which, when executed by a processor, causes the processor to implement the method according to any one of claims 1-10.

14. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1-10.

Images