Data transmission method, base station, terminal, device, storage medium and program product

Abstract

A robot control method, comprising: acquiring a target task instruction (210); in response to the target task instruction (210), using a first machine learning model to process visual information (125) of a target environment (100) to determine a text description (230) of the target environment (100); and on the basis of the text description (230) of the target environment (100), using a second machine learning model to generate a control instruction (240) for a robot (120), wherein the control instruction (240) is used for controlling the robot (120) to execute an operation corresponding to the target task instruction (210). Further comprised are an apparatus for implementing the robot control method, an electronic device, a storage media and a program product. A user only needs to issue a target task instruction, and a robot system can automatically complete environment perception, and call a language model to automatically execute a task.

Description

Data transmission method, base station, terminal, device, storage medium and program product

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This disclosure claims priority to Chinese patent application number 202410256817.9, filed on March 6, 2024, entitled “Methods, devices, apparatus and storage media for robot control”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] Example embodiments of the present disclosure relate generally to the field of robotics, and more particularly to methods, apparatuses, devices, storage media, and computer program products for robot control. Background Art

[0004] With the rapid development of robotics, it is being widely used not only in industrial production but also in everyday life. Consequently, the demand for intelligent robots is becoming increasingly demanding. Traditional robotics typically have limited functionality, only being able to perform their functions in specific scenarios, resulting in limited versatility. Summary of the Invention

[0005] In a first aspect of the present disclosure, a method for controlling a robot is provided. The method comprises: obtaining a target task instruction; in response to the target task instruction, processing visual information of the target environment using a first machine learning model to determine a textual description of the target environment; and generating control instructions for the robot using a second machine learning model based on the textual description of the target environment, the control instructions being used to control the robot to perform an operation corresponding to the target task instruction.

[0006] In a second aspect of the present disclosure, a device for controlling a robot is provided. The device includes: an acquisition module configured to acquire a target task instruction; a determination module configured to, in response to the target task instruction, determine a textual description of the target environment by processing visual information of the target environment using a first machine learning model; and a generation module configured to generate control instructions for the robot using a second machine learning model based on the textual description of the target environment, the control instructions being used to control the robot to perform an operation corresponding to the target task instruction.

[0007] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the electronic device to perform the method of the first aspect.

[0008] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the medium, and when the computer program is executed by a processor, the method of the first aspect is implemented.

[0009] In a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, which implements the method of the first aspect when executed by a processor.

[0010] It should be understood that the content described in this section is not intended to limit the key features or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0012] FIG1 shows a schematic diagram of an example environment in which embodiments of the present disclosure can be implemented;

[0013] FIG2 illustrates a schematic block diagram of an example architecture for robotic manipulation control according to some embodiments;

[0014] FIG3 shows a flow chart of a process 300 for robot control according to some embodiments of the present disclosure;

[0015] FIG4 shows a schematic structural block diagram of an apparatus for robot control according to some embodiments of the present disclosure; and

[0016] FIG5 illustrates a block diagram of an electronic device suitable for implementing one or more embodiments of the present disclosure. DETAILED DESCRIPTION

[0017] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0018] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, i.e., "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may be included below.

[0019] Herein, unless explicitly stated otherwise, executing a step “in response to A” does not mean executing the step immediately after “A” but may include one or more intermediate steps.

[0020] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition, use, storage or deletion of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0021] It is understandable that before using the technical solutions disclosed in the various embodiments of the present disclosure, the type, scope of use, usage scenarios, etc. of the information involved in the present disclosure should be informed to relevant users and authorization should be obtained from relevant users in an appropriate manner in accordance with relevant laws and regulations. The relevant users may include any type of right holders, such as individuals, enterprises, and groups.

[0022] For example, in response to receiving an active request from a user, a prompt message is sent to the relevant user to clearly prompt the relevant user that the operation requested to be performed will require obtaining and using the information of the relevant user, so that the relevant user can independently choose whether to provide information to the software or hardware such as the electronic device, application, server or storage medium that executes the operation of the technical solution of the present disclosure based on the prompt message.

[0023] As an optional but non-limiting implementation, in response to receiving an active request from a relevant user, a prompt message may be sent to the relevant user in the form of a pop-up window, in which the prompt message may be presented in text form. Furthermore, the pop-up window may also include a selection control for the user to select "agree" or "disagree" to provide information to the electronic device.

[0024] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0025] As used herein, the term "model" can learn the association between corresponding inputs and outputs from training data, so that after training is completed, corresponding outputs can be generated for given inputs. The generation of the model can be based on machine learning technology. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs by using multiple layers of processing units. A neural network model is an example of a model based on deep learning. In this article, "model" may also be referred to as "machine learning model", "learning model", "machine learning network" or "learning network", and these terms are used interchangeably in this article.

[0026] Generally speaking, machine learning can be roughly divided into three stages, namely the training stage, the testing stage and the application stage (also called the inference stage). In the training stage, a given model can be trained using a large amount of training data, and the parameter values ​​are continuously updated iteratively until the model can obtain consistent inferences that meet the expected goals from the training data. Through training, the model can be considered to be able to learn the association between input and output (also called input-to-output mapping) from the training data. The parameter values ​​of the trained model are determined. In the testing stage, the test input is applied to the trained model to test whether the model can provide the correct output, thereby determining the performance of the model. The testing stage can sometimes be integrated into the training stage. In the application stage, the trained model can be used to process the actual model input based on the parameter values ​​obtained through training to determine the corresponding model output.

[0027] 1 shows a schematic diagram of an example target environment 100 in which embodiments of the present disclosure can be implemented. In the example target environment 100 , there are visual information 125 , a robot 120 , and a user 110 . The user 110 can interact with the robot 120 .

[0028] In some embodiments, target environment 100 can be any environment in which robot 120 is to perform a task. For example, target environment 100 can be user 110's work environment, user 110's home environment, a shopping mall, a coffee shop, and so on. Visual information 125 is visual information recognized by robot 120. Visual information 125 can represent any entity in target environment 100.

[0029] In some embodiments, the user 110 interacts with the robot 120 to trigger the robot 120 to complete the task instructed by the user 110 based on the visual information 125 .

[0030] It should be understood that the structures and functions of the various elements in the target environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of the present disclosure.

[0031] Currently, language models (LMs), particularly large language models (LLMs), enable robots to complete a series of tasks based on the current target environment and user instructions. For example, if there is a glass of water in the target environment and the user's target instruction is "I'm thirsty," these two pieces of information are fed to the language model simultaneously. The language model can infer that the robot should now be controlled to fetch the water to the user. For example, if the user directly gives the robot the target instruction "I want to drink water," the robot will then search for the item "water" in the target environment and bring it to the user.

[0032] However, traditional approaches to implementing robot control logic based on language models require knowledge of the target environment, either by adding a pre-defined description of the target environment or by specifying the target directly in the user's task instructions. This results in the system being limited to handling known tasks within a known environment and preventing flexible environmental expansion. Consequently, traditional robotics technology is typically limited in functionality, capable of functioning only in specific scenarios, resulting in limited versatility.

[0033] In an embodiment of the present disclosure, an improved scheme for robot control is proposed. In this scheme, a target task instruction is obtained. In response to the target task instruction, a text description of the target environment is determined by processing the visual information of the target environment using a first machine learning model. And based on the text description of the target environment, a second machine learning model is used to generate control instructions for the robot, and the control instructions are used to control the robot to perform operations corresponding to the target task instructions. In this way, the user only needs to issue the target task instruction, and the robot can automatically complete the environmental perception and call the language model to realize the execution of the task. Furthermore, it can solve the problem that in a specific robot operating environment, the environmental content needs to be set in advance, or the content that can only express the user instruction can be displayed.

[0034] Some example embodiments of the present disclosure will be described in detail below with reference to examples in the accompanying drawings.

[0035] 2 is a schematic block diagram of an example architecture for robot operation control according to some embodiments. Through the architecture 200 , it is possible to control the robot 120 to provide services to the user 110 based on the target task instruction 210 of the user 110 .

[0036] Part or all of architecture 200 may be integrated into robot 120 or remotely deployed independently of robot 120, for example, on another remote device with computing capabilities (e.g., a terminal device or a service device). Accordingly, the determination and generation of control instructions for robot 120 may be performed by a remote device and transmitted to robot 120, or may be performed by a device integrated with robot 120. For ease of discussion, the following description of the control process of robot 120 via architecture 200 will be from the perspective of robot 120.

[0037] In some embodiments, the robot 120 receives a target task instruction 210 from the user 110 .

[0038] In the embodiment of the present disclosure, the target task instruction can be understood as the target task instruction given by the user that needs to be executed by the robot.

[0039] In some embodiments, the target task instruction 210 can be an abstract target task instruction or a specific target task instruction. For example, the target task instruction 210 can be an abstract instruction such as "I'm thirsty" spoken by the user. For another example, the target task instruction 210 can be a more specific instruction such as "Help me get that glass of water on the table" spoken by the user. In some embodiments, the target task instruction 210 can be input and provided to the robot 120 in any form. For example, the target task instruction 210 can be input via voice, text, or any other suitable form of input.

[0040] In some embodiments, the robot 120 responds to the target task instruction 210 and determines a textual description 230 of the target environment 100 by processing the visual information 125 in the target environment 100 using a first machine learning model.

[0041] In some embodiments, when the robot 120 receives the target task instruction 210 from the user 110, the robot 120 perceives the target environment 100 for the target task instruction 210 to be executed by using a first machine learning model, and then determines a text description 230 of the target environment 100.

[0042] For example, when the robot receives the user's voice saying "I'm thirsty", the robot uses the first machine learning model to perceive the target environment where "I'm thirsty" needs to be executed, and perceives the user's intention to need water, and then determines the text description of other entities such as "Cola" in the target environment.

[0043] In some embodiments, the robot 120 generates control instructions 240 for the robot 120 based on the text description of the target environment 100 using the second machine learning model.

[0044] In the embodiment of the present disclosure, the control instruction 240 is used to control the robot 120 to perform an operation corresponding to the target task instruction 210 .

[0045] In some embodiments, for example, when the robot 120 receives the target task instruction 210 “I am thirsty” from the user 110 and senses that the user 110 wants water, the robot 120 generates a control instruction 240 by utilizing the second machine learning model to control the robot 120 to bring water to the user 110 .

[0046] In the disclosed embodiment, the robot obtains the user's target instruction and generates a text description of the entity based on the environment perception of the first machine learning model. Furthermore, based on the text description of the entity, the control instruction is generated through the second machine learning, which can control the robot to perform operations corresponding to the user's target task instruction. In this way, the user only needs to issue abstract instructions, and the robot system can automatically complete the environment perception and call the large language model to execute the task. This avoids the problem of needing to set the environment content in advance or only being able to explicitly express the content of the user instruction in a specific robot operating environment.

[0047] The following embodiment of the present disclosure describes how the robot 120 responds to the target task instruction 210 and performs environmental perception based on the first machine learning model to determine a text description of the target environment 100.

[0048] In some embodiments, the robot 120 determines at least one target problem associated with the target task instruction 210. For example, as shown in FIG2 , at block 214 , the target problem determined by the robot 120 may be target problem 1, target problem 2, and so on.

[0049] In some embodiments, at least one target question is used to ask for information that needs to be known in order to complete the target task instruction 210 .

[0050] In some embodiments, the target question may be a question about some attributes of the visual information 125 of the target environment. For example, what is a certain entity in the visual information 125? What is the color of a certain entity in the visual information 125? Is a certain entity in the visual information 125 drinkable? And so on.

[0051] In some embodiments, the robot 120 determines at least one answer corresponding to at least one target question from the visual information 125 of the target environment.

[0052] In some embodiments, at least one answer determined by the robot may be as shown in FIG. 2 , in box 224 , sub-image 1: answer 1, answer 2, and so on, sub-image 2: answer 1, answer 2, and so on, sub-image 3: answer 1, answer 2, and so on, and so on.

[0053] For example, a certain entity in the visual information 125 of the target environment is a "can of Coke". The determined at least one target question may include:

[0054] What is a certain entity in the visual information 125?

[0055] What is the color of an entity in Visual Information 125?

[0056] Is the entity in visual information 125 drinkable?

[0057] By analyzing the visual information 125 , at least one answer to each target question may be determined, such as: a certain entity in the visual information 125 is a can of cola, is red in color, and is drinkable.

[0058] In some embodiments, the robot 120 generates a text description of the target environment based on at least one target question and at least one answer, as shown in box 230 in Figure 2, a text description of sub-image 1, a text description of sub-image 2, a text description of sub-image 3, and so on.

[0059] For example, the at least one target question may include, but is not limited to, the following: What is a certain entity in the visual information 125? What is the color of a certain entity in the visual information 125? Is a certain entity in the visual information 125 drinkable? Corresponding answers include, for example, "The certain entity in the visual information 125 is a can of cola, its color is red, and it is drinkable." The text description generated for the sub-image of the can of cola may be: "Red can of cola, drinkable."

[0060] In some embodiments, if an entity in the visual information 125 is a metal cup, its color is silver, and it can be drunk, the text description generated for the sub-image corresponding to the metal cup may be: the silver metal cup can be drunk.

[0061] In some embodiments, a certain entity in the visual information 125 is a keyboard, is black in color, and is not drinkable. The text description generated for the sub-image corresponding to the keyboard may be: a black keyboard is not drinkable.

[0062] In some embodiments, if an entity in the visual information 125 is a screwdriver, the color is none, and it is not allowed to drink, the text description generated for the sub-image corresponding to the screwdriver may be: screwdriver cannot be drunk.

[0063] In some embodiments, if an entity in the visual information 125 is a cable, its color is white, and it is not drinkable, the text description generated for the sub-image corresponding to the cable may be: white cable is not drinkable.

[0064] In some embodiments, an entity in the visual information 125 is boxed milk, white in color, and drinkable. The text description generated for the sub-image corresponding to the boxed milk may be: white boxed milk is drinkable.

[0065] In some embodiments, if an entity in the visual information 125 is a mouse, the color is none, and the user cannot drink, the text description generated for the sub-image corresponding to the mouse may be: The user cannot drink.

[0066] In some embodiments, the first machine learning model includes a first language model 212. The first language model can be any language model, such as a chatbot model. The first language model 212 can generate a target question based on a prompt word.

[0067] In some embodiments, the robot 120 provides the target task instruction 210 to the first language model 212 , and the first language model outputs at least one target question associated with the target task instruction 210 .

[0068] In some embodiments, the visual information 125 of the target environment includes an image of the target environment, for example, an image of all entities on the desk of the user 110 .

[0069] In some embodiments, the first machine learning model includes an image segmentation model 218 , which can be any segmentation model, such as a general vision model (Fast Segment Anything Model, abbreviated as FastSAM).

[0070] In some embodiments, image segmentation model 218 is used to segment the image of the target environment. It can be understood that image segmentation model 218 determines the specific location of each entity in the image of the target environment and extracts sub-images of the image of the target environment. A sub-image can be understood as a separate image of each entity contained in the image of the target environment, such as sub-image 1, sub-image 2, sub-image 3, etc., as shown in box 220 in FIG. 2 .

[0071] For example, the target environment image is an image of the user 110's desktop, which includes at least one entity, such as a mouse, a computer, a can of cola, a box cutter, etc. Segmenting the target environment image based on the image segmentation model 218 will generate sub-images of each entity, such as an image of a separate mouse, an image of a separate computer, an image of a separate can of cola, and an image of a separate box cutter.

[0072] In some embodiments, the robot 120 determines at least one answer corresponding to at least one target question based on the respective sub-images of at least one entity.

[0073] In some embodiments, the first machine learning model further includes a question answering model 222. The question answering model 222 can be any question answering model, such as a visual question answering model (VQA model).

[0074] In some embodiments, a question-answering model is used to determine at least one answer corresponding to at least one target question from a respective sub-image of at least one entity.

[0075] For example, if the target question is "What is a certain entity?", "What is the color of a certain entity?", or "Is a certain entity drinkable?", then for sub-image 1, such as an image of a can of cola, at least one answer might be: "It is a can of cola, red, and drinkable." For sub-image 2, such as an image of a box cutter, at least one answer might be: "It is a box cutter, yellow, and drinkable." For sub-image 3, such as an image of a mouse, at least one answer might be: "It is a mouse, white, and drinkable."

[0076] In some embodiments, the first language learning model further includes a second language model 232. The second language model 232 can be any language model.

[0077] In some embodiments, the second language model may be the same language model as the first language model, or the first language model and the second language model may be different language models.

[0078] In some embodiments, the second language model 232 is used for attribute summarization, for example, summarizing at least one answer output by the question-answering model 222 to form a text description.

[0079] In some embodiments, the robot 120 provides at least one answer determined based on the at least one target question, as shown in block 224 in FIG. 2 , to the second language model 232. Furthermore, the second language model 232 outputs a text description of the target environment, as shown in block 230 in FIG. 2 , including a text description of sub-image 1, a text description of sub-image 2, a text description of sub-image 3, and so on.

[0080] For example, for at least one answer, sub-image 1 is an image of a single can of cola, representing a can of cola, red, and drinkable. Sub-image 2 is an image of a single box cutter, representing a box cutter, yellow, and not drinkable. Sub-image 3 is an image of a mouse, representing a mouse, white, and not drinkable. The text description output by the second language model 232 is: a red can of cola is drinkable, a yellow box cutter is not drinkable, and a white mouse is not drinkable.

[0081] In some embodiments, the second machine learning model includes a third language model 238. The third language model 238 can be any language model.

[0082] In some embodiments, the third language model may be the same language model as the first language model 212 and the second language model 232 , or may be a different language model.

[0083] In some embodiments, the third language model 238 is used in the instruction generation process.

[0084] In some embodiments, the robot 120 provides the text description to the third language model 238. In turn, the third language model 238 outputs control instructions 240 for the robot 120.

[0085] In some embodiments, the robot 120 performs operations corresponding to the target task instructions based on the control instructions 240 .

[0086] For example, the robot 120 receives the user 110's call "I'm thirsty." Based on the first and second machine learning models, the robot 120 senses that the user needs water and identifies a "can of Coke" in the target environment. The robot then gives the "can of Coke" to the user 110. If no entity capable of providing the user with a drink is identified in the target environment, the robot 120 will not respond or may inform the user in voice that "your need has not been detected," etc.

[0087] The following embodiment of the present disclosure describes a process in which a correction input link is introduced to manually correct the text description because there may be errors or inappropriateness in the text description generated above.

[0088] In some embodiments, the robot 120 provides the text description to the user 110. For example, the text description may be provided to the user 110 in the form of a display screen of the robot 120, or the robot 120 provides the text description to the user 110 in the form of language playback, etc.

[0089] In some embodiments, the robot 120 determines a corrected text description 226 of the target environment based on the correction input 234 of the user 110 , such as: a text description of sub-image 1 , a text description after the dumplings in sub-image 2 , and a text description of sub-image 3 .

[0090] In one example, the user 110 corrects the text description according to the correction input 234 provided by the robot 120 .

[0091] In some examples, the robot 120 generates control instructions 240 for the robot based on the corrected text description 226 .

[0092] In some embodiments, the robot 120 may utilize the third language model 238 to generate control instructions 240 based on the corrected text description 226 .

[0093] In some embodiments, the second machine learning model includes a fourth language model 228. The fourth language model 228 can be any language model.

[0094] In some embodiments, the fourth language model 228 may be the same language model as the first language model 212 , the second language model 232 , and the third language model 238 , or may be a different language model.

[0095] In some embodiments, the fourth language model 228 may be used for updating the text description.

[0096] In some embodiments, the correction input 234 and the text description 230 are provided to the fourth language model 228 , which then outputs a corrected text description.

[0097] In some embodiments, the user 110 may modify the text description 230 using the fourth language model 228 in a dialog format.

[0098] For example, the text description 230 output by the second language model 232 is: "A red can of Coke is drinkable, a black box cutter is not drinkable, and a white mouse is not drinkable." When the robot 120 displays the text description 230 to the user 110, the user 110 finds that the text description 230 is inaccurate. Therefore, the user 110 can modify it by inputting correction 234. The corrected text description output by the fourth language model is: "A red can of Coke is drinkable, a yellow box cutter is not drinkable, and a white mouse is not drinkable."

[0099] Based on target task instructions, the disclosed embodiment utilizes the reasoning capabilities of language models to construct a system for perceiving target questions, using a segmentation model to extract images of any entity in the target environment. A question-answering model is then used to generate answers, ultimately generating a textual description of the target environment, thereby achieving semantic perception of the specific robot target task environment. Furthermore, this approach can transform environmental perception from passive input or perceived settings to active robot inspiration, significantly expanding the application scenarios of language models integrated into robot control systems, beyond being limited to specific spaces and specific targets.

[0100] FIG3 illustrates a flowchart of a process 300 for controlling a robot according to some embodiments of the present disclosure. Process 300 may be implemented in the robot 120 of FIG1 and FIG2 , or implemented in or included in other devices for controlling the robot 120, such as other terminal devices or service devices. For ease of description, it is assumed that process 300 may be implemented in the robot 120, and process 300 will be described below with reference to FIG1 and FIG2 .

[0101] At block 310 , the robot 120 obtains a target task instruction.

[0102] At block 312 , the robot 120 determines a textual description of the target environment by processing visual information of the target environment using a first machine learning model in response to the target task instruction.

[0103] In box 314, the robot 120 uses the second machine learning model to generate control instructions for the robot based on the text description of the target environment, and the control instructions are used to control the robot to perform operations corresponding to the target task instructions.

[0104] In some embodiments, determining a textual description of the target environment includes: determining at least one target question associated with the target task instructions, the at least one target question being used to ask information that needs to be known in order to complete the target task instructions; determining at least one answer corresponding to the at least one target question from visual information of the target environment; and generating a textual description of the target environment based on the at least one target question and the at least one answer.

[0105] In some embodiments, the first machine learning model includes a first language model, and determining at least one target question associated with the target task instruction includes: providing the target task instruction to the first language model; and determining the at least one target question based on an output of the first language model.

[0106] In some embodiments, the visual information of the target environment includes an image of the target environment, the first machine learning model includes an image segmentation model, and determining at least one answer corresponding to at least one target question includes: performing segmentation on the image using the image segmentation model to obtain a sub-image of at least one entity in the target environment; and determining at least one answer corresponding to at least one target question based on the sub-image of at least one entity.

[0107] In some embodiments, the first machine learning model also includes a question-answering model, and wherein determining at least one answer corresponding to at least one target question based on the respective sub-image of at least one entity includes: using the question-answering model to determine at least one answer to at least one target question from the respective sub-image of at least one entity.

[0108] In some embodiments, the first machine learning model further includes a second language model, and wherein generating a text description of the target environment based on at least one target question and at least one answer comprises: providing the at least one target question and the at least one answer to the second language model; and generating a text description of the target environment based on an output of the second language model.

[0109] In some embodiments, the second machine learning model includes a third language model, and wherein generating control instructions for the robot includes: providing the text description to the third language model; and generating control instructions for the robot based on an output of the third language model.

[0110] In some embodiments, generating control instructions for the robot includes: providing a text description to a user; determining a corrected text description of the target environment based on a correction input of the user; and generating control instructions for the robot based on the corrected text description.

[0111] In some embodiments, the second machine learning model includes a fourth language model.

[0112] In some embodiments, determining the corrected text description based on the user's correction input includes: providing the correction input and the text description to a fourth language model; and determining the corrected text description based on an output of the fourth language model.

[0113] Embodiments of the present disclosure also provide corresponding apparatuses for implementing the above-described methods or processes. Figure 4 shows a schematic structural block diagram of an apparatus 400 for robot control according to some embodiments of the present disclosure. Apparatus 400 may, for example, be implemented or included in robot 120, or implemented or included in other devices for controlling robot 120, such as other terminal devices or service devices. The various modules / components in apparatus 400 may be implemented using hardware, software, firmware, or any combination thereof.

[0114] As shown, apparatus 400 includes an acquisition module 410 configured to acquire a target task instruction. Apparatus 400 also includes a determination module 420 configured to, in response to the target task instruction, determine a textual description of the target environment by processing visual information of the target environment using a first machine learning model. Apparatus 400 also includes a generation module 430 configured to generate control instructions for a robot based on the textual description of the target environment using a second machine learning model. The control instructions are used to control the robot to perform operations corresponding to the target task instruction.

[0115] In some embodiments, the determination module 420 includes: determining at least one target question associated with the target task instructions, the at least one target question being used to ask information that needs to be known in order to complete the target task instructions; determining at least one answer corresponding to the at least one target question from visual information of the target environment; and generating a text description of the target environment based on the at least one target question and the at least one answer.

[0116] In some embodiments, the first machine learning model includes a first language model, and determining at least one target question associated with the target task instruction includes: providing the target task instruction to the first language model; and determining the at least one target question based on an output of the first language model.

[0117] In some embodiments, the visual information of the target environment includes an image of the target environment, the first machine learning model includes an image segmentation model, and determining at least one answer corresponding to at least one target question includes: performing segmentation on the image using the image segmentation model to obtain a sub-image of at least one entity in the target environment; and determining at least one answer corresponding to at least one target question based on the sub-image of at least one entity.

[0118] In some embodiments, the first machine learning model also includes a question-answering model, and wherein determining at least one answer corresponding to at least one target question based on the respective sub-image of at least one entity includes: using the question-answering model to determine at least one answer to at least one target question from the respective sub-image of at least one entity.

[0119] In some embodiments, the first machine learning model further includes a second language model, and wherein generating a text description of the target environment based on at least one target question and at least one answer comprises: providing the at least one target question and the at least one answer to the second language model; and generating a text description of the target environment based on an output of the second language model.

[0120] In some embodiments, the generating module 430 includes providing the text description to a third language model and generating control instructions for the robot based on output of the third language model.

[0121] In some embodiments, the generation module 430 includes: providing the text description to the user; determining a corrected text description of the target environment based on the user's correction input; and generating control instructions for the robot based on the corrected text description.

[0122] In some embodiments, the second machine learning model includes a fourth language model.

[0123] In some embodiments, determining the corrected text description based on the user's correction input includes: providing the correction input and the text description to a fourth language model; and determining the corrected text description based on an output of the fourth language model.

[0124] As shown in FIG5 , electronic device 500 is a general-purpose electronic device. Components of electronic device 500 may include, but are not limited to, one or more processors or processing units 510, memory 520, storage device 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. Processing unit 510 may be a real or virtual processor and is capable of performing various processes according to programs stored in memory 520. In a multi-processor system, multiple processing units execute computer-executable instructions in parallel to enhance the parallel processing capabilities of electronic device 500.

[0125] The electronic device 500 typically includes a plurality of computer storage media. Such media can be any accessible media that can be obtained by the electronic device 500, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 520 can be a volatile memory (e.g., registers, cache, random access memory (RAM)), a non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 530 can be a removable or non-removable medium and can include a machine-readable medium, such as a flash drive, a disk, or any other medium that can be used to store information and / or data and can be accessed within the electronic device 500.

[0126] The electronic device 500 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not shown in FIG. 5 , a disk drive for reading from or writing to a removable, non-volatile disk (e.g., a “floppy disk”) and an optical drive for reading from or writing to a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to a bus (not shown) by one or more data media interfaces. The memory 520 may include a computer program product 525 having one or more program modules configured to perform various methods or actions of various embodiments of the present disclosure.

[0127] The communication unit 540 enables communication with other electronic devices via a communication medium. Additionally, the functions of the components of the electronic device 500 can be implemented in a single computing cluster or multiple computing machines that can communicate via a communication connection. Thus, the electronic device 500 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0128] Input device 550 may be one or more input devices, such as a mouse, keyboard, or trackball. Output device 560 may be one or more output devices, such as a display, a speaker, or a printer. Electronic device 500 may also communicate with one or more external devices (not shown) via communication unit 540 as needed, such as a storage device, a display device, or the like, with one or more devices that allow a user to interact with electronic device 500, or with any device that allows electronic device 500 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0129] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0130] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0131] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, such that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0132] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0133] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple implementations of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part for a module, program segment or instruction, and a part for a module, program segment or instruction comprises one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0134] While various implementations of the present disclosure have been described above, the foregoing description is intended to be illustrative, not exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is selected to best explain the principles of the implementations, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for robot control, comprising: Get target task instructions; In response to the target task instruction, determining a textual description of the target environment by processing visual information of the target environment using a first machine learning model; and Based on the text description of the target environment, a second machine learning model is used to generate control instructions for the robot, where the control instructions are used to control the robot to perform operations corresponding to the target task instructions.

2. The method of claim 1 , wherein determining a textual description of the target environment comprises: determining at least one target question associated with the target task instruction, wherein the at least one target question is used to inquire about information required to complete the target task instruction; determining at least one answer corresponding to the at least one target question from visual information of the target environment; as well as A text description of the target environment is generated based on the at least one target question and the at least one answer.

3. The method of claim 2, wherein the first machine learning model comprises a first language model, and determining at least one target question associated with the target task instruction comprises: Providing the target task instruction to the first language model; as well as The at least one target question is determined based on an output of the first language model.

4. The method of claim 2, wherein the visual information of the target environment comprises an image of the target environment, the first machine learning model comprises an image segmentation model, and wherein determining at least one answer to the at least one target question comprises: Segmenting the image using the image segmentation model to obtain a sub-image of at least one entity in the target environment; as well as At least one answer corresponding to the at least one target question is determined based on the respective sub-images of the at least one entity.

5. The method of claim 4, wherein the first machine learning model further comprises a question-answering model, and wherein determining at least one answer corresponding to the at least one target question based on the respective sub-images of the at least one entity comprises: The question-answering model is used to determine the at least one answer to the at least one target question from the respective sub-images of the at least one entity.

6. The method of claim 2, wherein the first machine learning model further comprises a second language model, and wherein generating a textual description of the target environment based on the at least one target question and the at least one answer comprises: providing the at least one target question and the at least one answer to the second language model; as well as The text description of the target environment is generated based on the output of the second language model.

7. The method of claim 1 , wherein the second machine learning model comprises a third language model, and wherein generating control instructions for the robot comprises: providing the text description to the third language model; as well as A control instruction for the robot is generated based on the output of the third language model.

8. The method according to claim 1, wherein generating a control instruction for the robot comprises: providing the text description to a user; determining a corrected text description of the target environment based on the user's correction input; as well as Control instructions for the robot are generated based on the corrected text description.

9. The method of claim 8, wherein the second machine learning model comprises a fourth language model, wherein determining a corrected text description based on the user's correction input comprises: providing the correction input and the text description to the fourth language model; as well as The corrected text description is determined based on the output of the fourth language model.

10. A device for robot control, comprising: an acquisition module, configured to acquire target task instructions; a determination module configured to determine, in response to the target task instruction, a text description of the target environment by processing visual information of the target environment using a first machine learning model; as well as A generation module is configured to generate control instructions for the robot based on the text description of the target environment using a second machine learning model, wherein the control instructions are used to control the robot to perform operations corresponding to the target task instructions.

11. An electronic device comprising: at least one processing unit; as well as At least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 9 when executed by the at least one processing unit.

12. A computer-readable storage medium having a computer program stored thereon, wherein the computer program can be executed by a processor to implement the method according to any one of claims 1 to 9.

13. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images