Method and apparatus for controlling a robot

Abstract

The application discloses a robot control method and device, wherein, in the case that a user input text to be processed is acquired, the text to be processed is analyzed to obtain at least one instruction text; a simulation environment map corresponding to a real physical environment in a preset area is acquired; based on the instruction text and the simulation environment map, task instructions required for the robot to complete the instruction text on the simulation environment map are determined, and each task instruction corresponding to each instruction text is obtained; and the robot is controlled to sequentially execute each task instruction. The application can reduce the complexity of converting the high-level language text to be processed into a task instruction, thereby improving the accuracy of controlling the robot.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a robot control method and device. Background Technology

[0002] With the increasing application of intelligent robots in indoor home settings, the requirements for their intelligence are becoming more and more demanding, such as intelligent navigation, intelligent companionship, and intelligent services. However, current robots mainly rely on machine task instructions for driving, while user input is mostly in high-level language text. The instructions in high-level language text are quite complex, resulting in low accuracy in robot control.

[0003] In other words, the accuracy of robot control in existing technologies is relatively low. Summary of the Invention

[0004] This application provides a robot control method and apparatus, which can improve the accuracy of robot control.

[0005] Firstly, the robot control method provided in this application includes:

[0006] Upon receiving the user-inputted text to be processed, the text is parsed to obtain at least one instruction text;

[0007] Obtain a simulation environment map corresponding to the real physical environment within a preset area;

[0008] Based on the instruction text and the simulation environment map, determine the task instructions that the robot needs to perform on the simulation environment map to complete the instruction text, and obtain each task instruction corresponding to each instruction text;

[0009] The robot is controlled to execute each of the task instructions in sequence.

[0010] Secondly, the robot control device provided in this application includes:

[0011] The parsing module is used to parse the text to be processed by the user input to obtain at least one instruction text.

[0012] The acquisition module is used to acquire a simulation environment map corresponding to the real physical environment within a preset area;

[0013] The determination module is used to determine, based on the instruction text and the simulation environment map, the task instructions that the robot needs to perform on the simulation environment map to complete the instruction text, and to obtain each task instruction corresponding to each instruction text;

[0014] The control module is used to control the robot to execute each of the task instructions in sequence.

[0015] Thirdly, the electronic device provided in this application includes a memory and a processor. The memory stores a computer program, and the processor is used to run the computer program in the memory to implement the steps in the robot control method provided in this application.

[0016] Fourthly, the computer-readable storage medium provided in this application stores multiple instructions that are adapted for loading by a processor to implement the steps in the robot control method provided in this application.

[0017] Fifthly, the computer program product provided in this application includes a computer program or instructions that, when executed by a processor, implement the steps in the robot control method provided in this application.

[0018] In this application, compared to related technologies, upon obtaining user-inputted text to be processed, the text is parsed to obtain at least one instruction text; a simulation environment map corresponding to the real physical environment within a preset area is obtained; based on the instruction text and the simulation environment map, the task instructions required for the robot to complete the instruction text on the simulation environment map are determined, resulting in each task instruction corresponding to each instruction text; and the robot is controlled to execute each task instruction sequentially. This application first decomposes the text to be processed into at least one instruction text, and then combines it with the simulation environment map to map the instruction text into task instructions that the robot can execute on the simulation environment map. This allows for the decomposition of user-inputted high-level language text into instruction text, and then mapping it into task instructions that the robot can directly execute. This reduces the complexity of converting high-level language text into task instructions, thereby improving the accuracy of robot control. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of a scenario for the robot control system provided in an embodiment of this application;

[0021] Figure 2 This is a flowchart illustrating one embodiment of the robot control method provided in this application.

[0022] Figure 3This is a schematic diagram of multiple simulation environments in one embodiment of the robot control method provided in this application;

[0023] Figure 4 This is a schematic diagram illustrating the changes in the simulation environment before and after the robot executes task instructions in one embodiment of the robot control method provided in this application.

[0024] Figure 5 This is a schematic diagram illustrating the generation of an execution video in one embodiment of the robot control method provided in this application.

[0025] Figure 6 This is a schematic diagram of a preset training set in one embodiment of the robot control method provided in this application;

[0026] Figure 7 This is a schematic diagram of adding a bypass module and a fusion module to a preset large-scale language model in one embodiment of the robot control method provided in this application.

[0027] Figure 8 This is a schematic diagram illustrating the training of a preset text parsing model in one embodiment of the robot control method provided in this application.

[0028] Figure 9 This is a flowchart illustrating another embodiment of the robot control method provided in this application.

[0029] Figure 10 This is a schematic diagram of a prompt template in another embodiment of the robot control method provided in this application;

[0030] Figure 11 This is a schematic diagram of the structure of one embodiment of the robot control device provided in this application;

[0031] Figure 12 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0032] It should be noted that the principles of this application are illustrated by example in a suitable computing environment. The following description is based on the specific embodiments of this application that are illustrated, and should not be regarded as limiting other specific embodiments not detailed herein.

[0033] In the following description of this application, "some embodiments" are referred to, which describe a subset of all possible embodiments. However, it is understood that "some embodiments" may be the same subset or different subset of all possible embodiments, and may be combined with each other without conflict.

[0034] In the following description of this application, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0035] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0036] To improve the accuracy of robot control, embodiments of this application provide a robot control method, a robot control device, an electronic device, a computer-readable storage medium, and a computer program product. The robot control method can be executed by the robot control device or by an electronic device integrating the robot control device.

[0037] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0038] Please refer to Figure 1 This application also provides a control system for a robot, such as Figure 1 As shown, the robot's control system includes an electronic device 100 and a robot 300 connected to the electronic device 100. The electronic device 100 integrates the robot control device provided in this application.

[0039] Among them, electronic device 100 can be any device equipped with a processor and having processing capabilities, such as mobile electronic devices with processors such as smartphones, tablets, PDAs, laptops, and smart speakers, or fixed electronic devices with processors such as desktop computers, televisions, servers, and industrial equipment.

[0040] In addition, the robot's control system may also include a memory 200 for storing raw data, intermediate data, and result data.

[0041] In this embodiment of the application, the storage device can be a cloud storage device. Cloud storage is a new concept that is extended and developed from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as a storage system) refers to a storage system that uses cluster applications, grid technology and distributed storage file system functions to bring together a large number of storage devices of various types in the network (storage devices are also called storage nodes) through application software or application interfaces to work together to provide data storage and business access functions to the outside world.

[0042] Currently, the storage method of storage systems is as follows: Logical volumes are created. During the creation of a logical volume, physical storage space is allocated to each logical volume. This physical storage space may consist of a single storage device or the disks of several storage devices. Clients store data on a logical volume, which means storing the data on the file system. The file system divides the data into many parts, each part being an object. Each object contains not only the data but also additional information such as a data identifier (ID, ID entity). The file system writes each object to the physical storage space of that logical volume and records the storage location information of each object. Therefore, when a client requests access to data, the file system can allow the client to access the data based on the storage location information of each object.

[0043] The process by which a storage system allocates physical storage space to a logical volume is as follows: the physical storage space is pre-divided into strips according to the capacity estimate of the objects stored in the logical volume (this estimate often has a large margin relative to the actual capacity of the objects to be stored) and the grouping of Redundant Array of Independent Disks (RAID). A logical volume can be understood as a strip, thus allocating physical storage space to the logical volume.

[0044] It should be noted that, Figure 1 The schematic diagram of the robot control system shown is merely an example. The robot control system and scenario described in this application are intended to more clearly illustrate the technical solutions of this application and do not constitute a limitation on the technical solutions provided in this application. As those skilled in the art will know, with the evolution of robot control systems and the emergence of new business scenarios, the technical solutions provided in this application are also applicable to similar technical problems.

[0045] The following sections provide detailed descriptions of each example. It should be noted that the sequence numbers of the following embodiments are not intended to limit the preferred order of the embodiments.

[0046] Please refer to Figure 2 , Figure 2This is a flowchart illustrating one embodiment of the robot control method provided in this application, as shown below. Figure 2 As shown, the flow of the robot control method provided in this application is as follows:

[0047] 201. Given the user-inputted text to be processed, parse the text to be processed to obtain at least one instruction text.

[0048] The text to be processed is the user-input text in natural language. Natural language is a naturally formed language used for human communication. It consists of a series of phonemes, words, syntax, and semantics, and can be understood and used by humans or other organisms. There are many types of natural languages, such as Chinese, English, and French. The advantages of natural language are its rich expression and flexibility; its disadvantages are its complex rules and inconsistencies.

[0049] The text to be processed can be in various languages ​​such as Chinese, English, and French. For example, the text to be processed might be "Take a bottle of milk from the refrigerator." The user inputs the text hoping the robot can understand "Take a bottle of milk from the refrigerator" and execute the instruction.

[0050] The instruction text consists of low-level language commands that the robot can understand and execute. These instructions can be directly input into the robot for execution. At least one instruction text represents a series of commands that the robot needs to execute to fulfill the intent expressed in the text to be processed. Optionally, the text to be processed can be decomposed into at least two instruction texts. Parsing the text yields more instruction texts, resulting in more accurate robot control. For example, if the text to be processed is "Get a bottle of milk from the refrigerator," after parsing, six instruction texts are obtained: "{find}". <milk>”、"{walk} <milk>”、"find} <fridge>”、"{open} <fridge>”、"{grab} <milk>”、"{open} <fridge>".

[0051] In one specific embodiment, keywords are extracted from the text to be processed, and matching instruction texts are searched in a preset instruction text library based on the extracted keywords to obtain at least one instruction text.

[0052] In another specific embodiment, a preset text parsing model is pre-trained based on a preset training set. This preset text parsing model is then used to parse the text to be processed, yielding at least one instruction text. The preset text parsing model can be any type of neural network model, and is not limited here. By performing semantic analysis on the text to be processed using the preset text parsing model, compared to keyword matching, the user intent in the text to be processed can be extracted more accurately, resulting in more accurate instruction text and improving the accuracy of robot control.

[0053] 202. Obtain the simulation environment map corresponding to the real physical environment within the preset area.

[0054] The simulation environment map is a 3D electronic map of a preset area. It is based on a 3D electronic map database and is a 3D, abstract description of one or more aspects of the real world or a part of it, according to a certain scale. The network 3D electronic map not only provides users with map search and navigation functions through intuitive geographic scene simulation, but also integrates a series of services such as life information, e-government, e-commerce, virtual communities, and travel navigation.

[0055] The system allows the robot to avoid obstacles in a pre-defined real-world environment using LiDAR and to scan the environment 360 degrees using its built-in camera. Simultaneous Localization and Mapping (SLAM) algorithms are then used to create a simulated map of the real-world environment within the pre-defined area. SLAM can be described as follows: the robot starts moving from an unknown location in an unknown environment, performs self-localization based on its position and the map, and simultaneously builds an incremental map based on its self-localization, thus achieving autonomous localization and navigation.

[0056] The simulation environment map identifies objects within a predefined area and the relationships between them. These objects and relationships can be represented using a knowledge graph. Each point in the knowledge graph represents information about an individual object, and each edge represents the relationships between objects. The object attribute information corresponding to each node includes: object index, category, current state, bounding box, and 3D information. The current state of an object primarily includes the following states: open, closed, on, and in.

[0057] In this embodiment of the application, there can be multiple simulation environment maps corresponding to the real physical environment within the preset area. These multiple simulation environment maps are used to allow the robot to execute command texts in different scenarios. For example... Figure 3 As shown, this application simulates six different simulation environment maps. Figure 3 The six different simulation environment maps cover the following main scenario categories: navigation, home management, companionship, entertainment, and daily user activities.

[0058] 203. Based on the instruction text and the simulation environment map, determine the task instructions that the robot needs to perform to complete the instruction text on the simulation environment map, and obtain the task instructions corresponding to each instruction text.

[0059] In this embodiment, the instruction text refers to the target action and the target object on which the target action acts. For example, one instruction text is: {walk} <milk>The target action is {walk}, and the target object is milk, meaning the action moves to the position of the target object milk.

[0060] Since the model's command output and simulation input may not be completely consistent, command mapping is required, aligning the command text with the simulation environment map. In a specific embodiment, based on the command text and the simulation environment map, the task instructions the robot needs to perform to complete the command text on the simulation environment map are determined, resulting in the various task instructions corresponding to each command text, including:

[0061] (1) Obtain robot attribute information on the simulation environment map and object attribute information of the target object on the simulation environment map.

[0062] In one specific embodiment, the robot attribute information includes the robot's position on the simulation environment map, the robot's state, etc.; the target object's object attribute information on the simulation environment map includes the target object's position on the simulation environment map, its current state, etc.

[0063] For example, robot attribute information is the robot's position, and target object attribute information on the simulation environment map is the target object's position on the simulation environment map.

[0064] (2) Based on the robot attribute information on the simulation environment map, the object attribute information of the target object on the simulation environment map, and the task instruction corresponding to the target action determination instruction text.

[0065] In one specific embodiment, the robot attribute information is the robot's robot position, the target object's attribute information on the simulation environment map is the target object's position on the simulation environment map, and the target action is "walk". Based on the target action, the navigation task to be executed by the instruction text is determined. Therefore, the task instruction corresponding to the instruction text is: Navigate the robot from its robot position to the target object position on the simulation environment map.

[0066] For example, the instruction text is: {walk} <milk>Mapping the instruction text to task instructions yields the following task instructions:<char()> {walk} <milk>The task instructions are:<char()> {walk} <milk>This indicates that the robot is navigating from its current location to the location of milk on the simulation environment map.

[0067] In one specific embodiment, the text to be processed is "Get a bottle of milk from the refrigerator". After parsing, six instruction texts are obtained, which are: "{find} <milk>”、"{walk} <milk>”、"{find} <fridge>”、"{open} <fridge>”、"{grab} <milk>”、"{open} <fridge>"Based on the instruction text and the simulation environment map, the task instructions that the robot needs to perform to complete the instruction text on the simulation environment map are determined, resulting in the corresponding task instructions for each instruction text. The six corresponding task instructions are as follows:"<char()> {find} <milk>”、"<char()>{walk} <milk>”、"<char()>{find} <fridge>”、"<char()>{open} <fridge>”、"<char()>{grab} <milk>”、"<char()>{open} <fridge>".

[0068] Since the parsed instruction text may be unexecutable by the robot and become disordered, in order to avoid this situation, in a specific embodiment, when each target action in each instruction text belongs to a preset action set, the task instructions that the robot needs to perform to complete the instruction text in the simulation environment map are determined based on the instruction text and the simulation environment knowledge graph, thus obtaining each task instruction corresponding to each instruction text.

[0069] The preset action set can be pre-defined; for example, it can include {walk}, {grab}, {lie}, and {read}. When each target action in each instruction text belongs to the preset action set, it indicates that each instruction text contains actions that the robot can execute. In this case, based on the instruction text and the simulation environment knowledge graph, the robot can then determine the task instructions it needs to perform to complete the instruction text on the simulation environment map. This avoids the robot being unable to execute the parsed instruction text, resulting in disorder.

[0070] Furthermore, when the positions of all target objects in each instruction text are located within a preset area, it indicates that the target object is obtainable by the robot within the preset area. At this point, based on the instruction text and the simulation environment knowledge graph, the task instructions that the robot needs to perform to complete the instruction text in the simulation environment map can be determined, which can avoid the robot being unable to execute the parsed instruction text and causing disorder.

[0071] This application simulates six different simulation environment maps. Figure 3 The six different simulation environment maps cover the following main scenario categories: navigation, home management, companionship, entertainment, and daily user activities.

[0072] In a specific embodiment, the scene category of each instruction text is obtained. If the scene categories of each instruction text belong to the same scene category, the simulation environment map corresponding to the scene category of each instruction text is obtained. Based on the instruction text and the simulation environment map, the task instructions that the robot needs to perform to complete the instruction text on the simulation environment map are determined, and the task instructions corresponding to each instruction text are obtained.

[0073] In another specific embodiment, the simulation environment map corresponding to the scene category set by the user is obtained. Based on the instruction text and the simulation environment map corresponding to the scene category set by the user, the task instructions that the robot needs to perform in the simulation environment map to complete the instruction text are determined, and the task instructions corresponding to each instruction text are obtained.

[0074] 204. Control the robot to execute each task instruction in sequence.

[0075] Furthermore, after obtaining the various task instructions, the robot needs to call various applications or services to execute the task instructions on the simulation environment map.

[0076] Specifically, the instruction text includes the target action and the target object on which the target action acts. If the instruction text is "move to the target object," and after mapping the instruction text to task instructions, the robot's target action is "move," then the tasks to be performed are navigation and movement. The robot is controlled by calling various navigation applications to plan a movement path to the target object on the simulation environment map to complete the navigation task. The robot is then controlled by calling the manipulation module to move to the target object according to the movement path on the simulation environment map to complete the movement task, thus fulfilling the task instructions.

[0077] For example, one of the task instructions is:<char()> {walk} <milk>If the target action is {walk} and the target object is milk, then the robot needs to first obtain its own position and the position of the object milk on the simulation environment map, then perform a navigation task to obtain the movement path to the object milk, and finally perform a movement task to move to the object milk according to the movement path.

[0078] like Figure 4 As shown, Figure 4 The upper half of the image shows the current state of the simulation environment map within the preset area. The text to be processed is "turn on TV and turn light off". Parsing the text yields at least one instruction text. The robot is then controlled to execute these instructions sequentially to complete at least one instruction text. After the robot executes the instructions sequentially, the state of the simulation environment map within the preset area is as follows: Figure 4 The lower half is shown.

[0079] In this embodiment of the application, the robot control method further includes: controlling the robot to capture execution videos during the process of controlling the robot to execute each task instruction in sequence; and outputting the execution videos after controlling the robot to execute each task instruction in sequence.

[0080] See Figure 5 The user inputs the text "Find where the bananas are stored and put the bananas in the fridge". The robot parses the text to obtain at least one instruction text, and then controls the robot to execute each instruction in sequence to complete at least one instruction text. During the execution of each instruction, the robot records video footage; after the robot has executed each instruction, the video is output.

[0081] In one specific embodiment, parsing the text to be processed to obtain at least one instruction text includes parsing the text to be processed based on a preset text parsing model to obtain at least one instruction text.

[0082] In this embodiment, the preset text parsing model includes a preset large-scale language model, a bypass module, and a fusion module. The preset large-scale language model is Chatglm-6b. The GLM model is a framework developed by Tsinghua University. Using Chatglm-6b is a modification of the Transformer model, employing a bidirectional approach of autoencoding and autoregression to transform the problem into a cloze test. It has 6.2 billion parameters and uses a 1TB bilingual corpus as its training dataset to optimize Chinese question-and-answer and dialogue. Currently, this model can generate answers that conform to human preferences. Of course, in other embodiments, the preset large-scale language model can also be a ChatGPT series or LLama series model, depending on the specific circumstances.

[0083] In one specific embodiment, the text to be processed is parsed based on a preset text parsing model to obtain at least one instruction text. Before this, the process includes: adding a bypass module and a fusion module to a preset large-scale language model to obtain an initial language model; and training the initial language model based on a preset training set to obtain a preset text parsing model. Specifically, the parameters of the preset large-scale language model in the initial language model are not updated during training, while the parameters of the bypass module in the initial language model are updated during training. Specifically, the bypass module includes a dimensionality reduction matrix and a dimensionality increase matrix. The dimensionality reduction matrix is ​​used to process the text to be processed to obtain dimensionality-reduced data; the dimensionality increase matrix is ​​used to process the dimensionality-incremented data to obtain bypass output data.

[0084] The preset training set consists of corpus data of robots performing tasks in indoor home scenarios. Since there are virtually no publicly available datasets for corpus data of robots performing tasks in indoor home scenarios, it is necessary to design and construct the corpus dataset ourselves to obtain the preset training set. This application constructs 6 indoor home scenario environments and 12 major categories of indoor home corpus types, including: leisure, work, room cleaning, room arrangement, food preparation, hygiene, social interaction, diet, activities, sleep, and preparation work, thus constructing a massive amount of corpus data.

[0085] The preset training set includes training text and a corresponding instruction text set, wherein the instruction text set includes at least one instruction text. Optionally, the instruction text set includes at least two instruction texts. See also Figure 6 Each training text, such as Figure 6 The corpus task list shows that the training texts can be "Read book", "Watch TV", and "Listen to music". The training texts and corresponding instruction text sets are as follows: Figure 6 The corpus format is shown in the figure.

[0086] For example, if the training text is "Read book", the corresponding instruction text set includes multiple instruction texts, such as "1.{walk}".<living_room> "、"2.{walk} <book>”、"3.{grab} <book>”、"4.{walk} <sofa>”、"5.{lie} <sofa>”、"6.{read} <book>".

[0087] like Figure 7 As shown, Figure 7 The left side of the middle section represents the weights of the pre-defined large-scale language model, with both input and output dimensions being d; x represents the processed text to be processed, with dimension d; W0 represents the weights of the pre-defined large-scale language model, which can be decomposed using low-rank decomposition, as shown in formula (1). A and B are decomposition matrices, where A is the dimension reduction matrix and B is the dimension increase matrix. The dimension reduction matrix A and the dimension increase matrix B form a bypass module. During training, the weights W0 are frozen and cannot be updated with gradients, but the dimension reduction matrix A and the dimension increase matrix B contain trainable parameters, and their forward propagation formula is shown in formula (2).

[0088] W0+ΔW=W0+BA (1)

[0089] h=W0x+ΔWx=W0x+BAx (2)

[0090] In this model, the reduced-dimensional matrix A is initialized using a random Gaussian distribution, while the increased-dimensional matrix B is initialized with zero values. Therefore, in the early stages of model training, ΔW = BA is zero. r is the rank of the matrix, which scales ΔWx by α / r, where α is a constant in r, and h is the output of the fusion module after data fusion.

[0091] In one specific embodiment, a preset training set is first obtained, which is in JSON format. Then, the preset training set is preprocessed to convert the JSON data into JSONL data format. At the same time, each training text in the preset training set is segmented and the data is saved. Then, the chatGLM-6b large model is fine-tuned by setting training parameters, saving the training model, and obtaining a fine-tuned large-scale language model.

[0092] The GLM model employs a combination of autoencoding and autoregression, similar to the concept of a cloze test. Autoencoding refers to the random deletion of consecutive tokens, while autoregression allows access to predicted tokens, thus enabling the model to effectively understand the context.

[0093] like Figure 8 As shown in the embodiments of this application, the steps for training an initial language model based on a preset training set to obtain a preset text parsing model are as follows:

[0094] Suppose the input is [x1, x2, x3, x4, x5, x6]. We sample with different spans following a Poisson distribution, namely [x3] and [x5, x6]. The input is divided into two parts: Part A is x1, x2, [M], x4, x5, [M]], and Part B is [x3] and [x5, x6]. Part A is the corrupted text x... corrupt Part B is the covered segment. Words in Part A can see each other, but cannot see any words in Part B. Words in Part B can see the prepositions in Part A and Part B, but cannot see the subsequent words in Part B.

[0095] GLM autoregression generates Part B. We concatenate Part A and Part B, where each span in Part B takes [S] as input and adds [E] as output. Two positional codes are used here: 2D positional code 1 represents the index, and for each span in Part B, its corresponding mask [M] marks the position; 2D positional code 2 represents the relative position within the span, so both are 0 in Part A.

[0096] Given an input text x = [x1, x2, x3, x4, x5, ..., xn], sample multiple text fragments [s1, s2, s3, s4, ..., sn]. m ], where each fragment s i This corresponds to a series of consecutive words in x. Each segment is replaced with a single [MASK] symbol, forming a corrupted text x. corrupt The model predicts missing words from damaged text in an autoregressive manner, meaning that when predicting missing words in a segment, the model can access both the damaged text and previously predicted segments. To fully capture the interdependencies between different segments, we randomly shuffle the order of the segments, similar to a permutation language model. All permutations of the Poisson sampled set, partB, are then performed, resulting in Zm, the set of all possible permutations of an index sequence of length m [1, 2, 3, ..., m]. The objective function of the initial language model is shown below.

[0097]

[0098]

[0099] In this process, the word in each blank is generated in order from left to right, thus generating fragment s. i The probability pθ can be decomposed into the following format:

[0100]

[0101] The initial language model is trained iteratively based on the objective function, and a preset text parsing model is obtained when the preset stopping condition is met.

[0102] In this embodiment of the application, the text to be processed is parsed based on a preset text parsing model to obtain at least one instruction text, including:

[0103] (1) Use the bypass module to process the text to be processed and obtain the bypass output data.

[0104] In this embodiment, the bypass module includes a dimensionality reduction matrix and an dimensionality increase matrix. The bypass module is used to process the text to be processed to obtain bypass output data, including: processing the text to be processed using the dimensionality reduction matrix to obtain dimensionality reduction data; and processing the dimensionality increase matrix to obtain bypass output data.

[0105] (2) Use a pre-set large-scale language model to process the text to be processed and obtain the model output data.

[0106] (3) Use the fusion module to fuse the bypass output data and the model output data to obtain at least one instruction text.

[0107] Please refer to Figure 9 , Figure 9 This is a flowchart illustrating another embodiment of the robot control method provided in this application, as shown below. Figure 9 As shown, the flow of the robot control method provided in this application is as follows:

[0108] 401. Preset text parsing model.

[0109] Specifically, a bypass module and a fusion module are added to a pre-defined large-scale language model to obtain an initial language model. The initial language model is then trained based on a pre-defined training set to obtain a pre-defined text parsing model.

[0110] The preset training set includes multiple training texts and corresponding instruction text sets. The instruction text set contains multiple instruction texts. The training texts are high-level human language texts. The instruction texts are low-level language instruction texts that can be used to control the robot.

[0111] 402. When obtaining user input data, determine whether the user input data is of text type.

[0112] In this embodiment, the user input data may be voice data, image data, or text data. Users can input data through a chat window. The system determines whether the user input data is text. If the user input data is voice data, speech recognition is performed to obtain the text to be processed; if the user input data is image data, image recognition is performed to obtain the text to be processed; if the user input data is text data, it is determined as the text to be processed.

[0113] 403. Identify the user input data as the text to be processed and obtain the pre-constructed prompt template.

[0114] In this embodiment of the application, if the user input data is text data, then the user input data is determined as the text to be processed.

[0115] The prompt template is used to instruct the processing text to be split into at least one instruction text. The prompt template is called a Prompt. In machine learning and natural language processing, a Prompt typically refers to an input text fragment used for generating text or training a language model. A Prompt can be a question, a topic, a passage of text, or a set of keywords, used to guide the model in generating relevant text. For example, in text generation tasks, a Prompt can be used to specify the topic or style of the generated text, in order to generate more suitable text. In language model training, a Prompt can be part of the input sequence, used to specify the language rules and contextual information that the model needs to learn.

[0116] Among them, the pre-constructed prompt templates are as follows: Figure 10 As shown, where, Figure 10 The text to be processed is "take a bottle of milk from the refrigerator", and LLM is the preset text parsing model.

[0117] 404. Input the prompt template and the text to be processed into the preset text parsing model to obtain at least one instruction text corresponding to the text to be processed.

[0118] For example, the text to be processed is "Get a bottle of milk from the refrigerator". The prompt template and the text to be processed are input into a preset text parsing model to obtain at least one instruction text corresponding to the text to be processed. After parsing, six instruction texts are obtained, namely: "1.{walk}<living_room> "、"2.{walk} <book>”、"3.{grab} <book>”、"4.{walk} <sofa>”、"5.{lie} <sofa>”、"6.{read} <book>".

[0119] 405. Determine whether each target action in each instruction text belongs to the preset action set.

[0120] In this embodiment, it is determined whether each target action in each instruction text belongs to a preset action set. If each target action in each instruction text belongs to the preset action set, then step 406 is executed: determine whether the position of each target object in each instruction text is located within a preset area. If each target action in each instruction text does not belong to the preset action set, then a prompt message is issued.

[0121] For example, the six instruction texts are: "<char()> {find} <milk>”、"<char()>{walk} <milk>”、"<char()>{find} <fridge>”、"<char()>{open} <fridge>”、"<char()>{grab} <milk>”、"<char()>{open} <fridge>The target actions in the six instruction texts are: "{find}", "{walk}", "{find}", "{open}", "{grab}", and "{open}".

[0122] 406. Determine whether the positions of each target object in each instruction text are all within the preset area.

[0123] In this embodiment, it is determined whether the positions of all target objects in each instruction text are all located within a preset area. If the positions of all target objects in each instruction text are all located within the preset area, it indicates that each instruction text is executable. Then, step 407 is executed: based on the instruction text and the simulation environment knowledge graph, the task instructions that the robot needs to perform to complete the instruction text on the simulation environment map are determined, and the task instructions corresponding to each instruction text are obtained. If the positions of all target objects in each instruction text are not all located within the preset area, a prompt message is issued.

[0124] For example, the six instruction texts are: "<char()> {find} <milk>”、"<char()>{walk} <milk>”、"<char()>{find} <fridge>”、"<char()>{open} <fridge>”、"<char()>{grab} <milk>”、"<char()>{open} <fridge>The target objects in the six instruction texts are: "{milk}", "{milk}", "{fridge}", "{fridge}", "{milk}", and "{fridge}".

[0125] 407. Based on the instruction text and the simulation environment knowledge graph, determine the task instructions that the robot needs to perform to complete the instruction text in the simulation environment map, and obtain the task instructions corresponding to each instruction text.

[0126] In one specific embodiment, the text to be processed is "Get a bottle of milk from the refrigerator". After parsing, six instruction texts are obtained, which are: "{find} <milk>”、"{walk} <milk>”、"{find} <fridge>”、"{open} <fridge>”、"{grab} <milk>”、"{open} <fridge>"Based on the instruction text and the simulation environment map, the task instructions that the robot needs to perform to complete the instruction text on the simulation environment map are determined, resulting in the corresponding task instructions for each instruction text. The six corresponding task instructions are as follows:"<char()> {find} <milk>”、"<char()>{walk} <milk>”、"<char()>{find} <fridge>”、"<char()>{open} <fridge>”、"<char()>{grab} <milk>”、"<char()>{open} <fridge>".

[0127] 408. Control the robot to execute each task instruction in sequence.

[0128] 409. During the process of controlling the robot to execute various task instructions in sequence, the robot is controlled to capture execution videos.

[0129] In this embodiment, 409 and 408 are executed simultaneously, and the execution video can show the entire process of the robot executing each task instruction in sequence.

[0130] 410. After the robot executes each task instruction in sequence, output the execution video.

[0131] In this embodiment, after the robot executes each task instruction sequentially, an execution video is output. The user can see the entire process of the robot executing each task instruction in sequence. Furthermore, based on the quality of the execution video, the constructed prompt template can be adjusted, thereby further improving the accuracy of robot control.

[0132] This application enables embodied intelligence in robots, giving their large language models the ability to interact with the physical world and providing a one-stop service from text input to video output. First, a massive dataset of indoor home scene language is constructed. Second, through fine-tuning techniques on a GLM pre-trained large language model, the model becomes an "expert" in a specific domain. To further improve model accuracy, prompt templates are constructed. Then, a robot simulation environment map is built to enable command execution. Finally, the large model and the simulation environment are chained together to map commands, realizing a one-stop service chain from text input to video output of the robot executing commands. This application also provides more intelligent possibilities for smart home robots, allowing for closer interaction between intelligent robots and humans. It enables natural language models to go beyond simply generating chat-like dialogues, allowing systems that can directly drive robots through text to achieve embodied intelligence.

[0133] To facilitate better implementation of the robot control method provided in the embodiments of this application, the embodiments of this application also provide a robot control device based on the above-described robot control method. The meanings of the terms used are the same as in the robot control method described above, and for specific implementation details, please refer to the descriptions in the above method embodiments.

[0134] Please refer to Figure 11 , Figure 11 This is a schematic diagram of the structure of the robot control device provided in an embodiment of this application. The robot control device may include a parsing module 601, an acquisition module 602, a determination module 603, and a control module 604, wherein...

[0135] The parsing module 601 is used to parse the text to be processed when the user input is obtained, and obtain at least one instruction text.

[0136] The acquisition module 602 is used to acquire a simulation environment map corresponding to the real physical environment within a preset area;

[0137] The determination module 603 is used to determine the task instructions that the robot needs to perform on the simulation environment map to complete the instruction text based on the instruction text and the simulation environment map, and to obtain the task instructions corresponding to each instruction text.

[0138] The control module 604 is used to control the robot to execute various task instructions in sequence.

[0139] For details on the implementation of each of the above modules, please refer to the previous examples, which will not be repeated here.

[0140] This application also provides an electronic device, including a memory and a processor, wherein the processor executes steps in the robot control method provided in this embodiment by calling a computer program stored in the memory.

[0141] Please refer to Figure 12 , Figure 12 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0142] The electronic device may include components such as a processor 101 with one or more processing cores, a memory 102 with one or more computer-readable storage media, a power supply 103, and an input unit 104. Those skilled in the art will understand that the electronic device structure shown in the figures does not constitute a limitation on the electronic device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0143] The processor 101 is the control center of the electronic device, connecting various parts of the device via various interfaces and lines. It executes software programs and / or modules stored in the memory 102, and calls data stored in the memory 102, to perform various functions and process data. Optionally, the processor 101 may include one or more processing cores; alternatively, the processor 101 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, while the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 101.

[0144] The memory 102 can be used to store software programs and modules. The processor 101 executes various functional applications and data processing by running the software programs and modules stored in the memory 102. The memory 102 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 102 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 102 may also include a memory controller to provide the processor 101 with access to the memory 102.

[0145] The electronic device also includes a power supply 103 that supplies power to the various components. Optionally, the power supply 103 can be logically connected to the processor 101 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 103 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0146] The electronic device may also include an input unit 104, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0147] Although not shown, the electronic device may also include a display unit, an image acquisition component, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 101 in the electronic device loads one or more executable codes corresponding to computer programs into the memory 102 according to the following instructions, and the processor 101 executes the steps in the robot control method provided in this application, such as:

[0148] Upon receiving the user-inputted text to be processed, the system parses the text to obtain at least one instruction text; it then acquires a simulation environment map corresponding to the real physical environment within a preset area; based on the instruction text and the simulation environment map, it determines the task instructions that the robot needs to perform on the simulation environment map to complete the instruction text, thus obtaining each task instruction corresponding to each instruction text; and finally, it controls the robot to execute each task instruction in sequence.

[0149] It should be noted that the electronic device provided in this application embodiment and the robot control method in the above embodiment belong to the same concept. The specific implementation process can be found in the above related embodiments, and will not be repeated here.

[0150] This application also provides a computer-readable storage medium storing a computer program thereon. When the computer program stored thereon is executed on the processor of the electronic device provided in the embodiments of this application, the processor of the electronic device performs the steps in the robot control method provided in this application. The storage medium may be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0151] This application also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform various optional implementations of the robot control method described above.

[0152] The above provides a detailed description of a robot control method and apparatus provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

[0153] It should be noted that when the above embodiments of this application are applied to specific products or technologies, and user-related data is involved, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.< / fridge> < / milk> < / fridge> < / fridge> < / milk> < / milk> < / fridge> < / milk> < / fridge> < / fridge> < / milk> < / milk> < / fridge> < / milk> < / fridge> < / fridge> < / milk> < / milk> < / fridge> < / milk> < / fridge> < / fridge> < / milk> < / milk> < / book> < / sofa> < / sofa> < / book> < / book> < / book> < / sofa> < / sofa> < / book> < / book> < / milk> < / fridge> < / milk> < / fridge> < / fridge> < / milk> < / milk> < / fridge> < / milk> < / fridge> < / fridge> < / milk> < / milk> < / milk> < / milk> < / milk> < / milk> < / fridge> < / milk> < / fridge> < / fridge> < / milk> < / milk>

Claims

1. A method for controlling a robot, characterized in that, include: By adding a bypass module and a fusion module to a pre-defined large-scale language model, an initial language model is obtained. An initial language model is trained based on a preset training set to obtain a preset text parsing model. The parameters of the preset large-scale language model in the initial language model are not updated during training, while the parameters of the bypass module in the initial language model are updated during training. The bypass module includes a dimensionality reduction matrix and an dimensionality increase matrix, which are obtained by performing low-rank decomposition on the weights of the preset large-scale language model. When user-input text is received, it is parsed to obtain at least one instruction text. A pre-constructed prompt template is obtained, which indicates that the text to be processed should be split into at least one instruction text. The prompt template and the text to be processed are input into the preset text parsing model to obtain at least one instruction text corresponding to the text to be processed. The instruction text includes a target action and the target object acting on the target action. The at least one instruction text is a series of instructions that the robot needs to execute to complete the intent expressed by the text to be processed. Obtain a simulation environment map corresponding to the real physical environment within a preset area; Based on the instruction text and the simulation environment map, determine the task instructions that the robot needs to perform on the simulation environment map to complete the instruction text, and obtain each task instruction corresponding to each instruction text; The robot is controlled to execute each of the task instructions in sequence. During the execution of each task instruction, the robot is controlled to capture execution videos. After the robot executes each task instruction in sequence, the execution videos are output, and the constructed prompt template is adjusted based on the execution videos.

2. The robot control method according to claim 1, characterized in that, The process of determining the task instructions that the robot needs to perform on the simulation environment map to complete the task instructions based on the instruction text and the simulation environment map, and obtaining the task instructions corresponding to each instruction text, includes: When each of the target actions in each of the instruction texts belongs to a preset action set, the task instructions that the robot needs to perform on the simulation environment map to complete the instruction texts are determined based on the instruction texts and the simulation environment map, thus obtaining each of the task instructions corresponding to each instruction text.

3. The robot control method according to claim 1, characterized in that, The process of determining the task instructions required for the robot to complete the instruction text based on the instruction text and the simulation environment knowledge graph, and obtaining the respective task instructions corresponding to each instruction text, includes: When the positions of all target objects in each of the instruction texts are located within the preset area, the task instructions that the robot needs to perform on the simulation environment map to complete the instruction texts are determined based on the instruction texts and the simulation environment map, thus obtaining each of the task instructions corresponding to each instruction text.

4. The robot control method according to claim 3, characterized in that, The step of inputting the prompt template and the text to be processed into a preset text parsing model to obtain at least one instruction text corresponding to the text to be processed includes: The bypass module is used to process the text to be processed to obtain bypass output data. The text to be processed is processed using the preset large-scale language model to obtain model output data; The bypass output data and the model output data are fused using the fusion module to obtain at least one instruction text.

5. The robot control method according to claim 4, characterized in that, The bypass module includes a dimension reduction matrix and a dimension increase matrix. The process of using the bypass module to process the text to obtain bypass output data includes: The text to be processed is processed using the dimensionality reduction matrix to obtain dimensionality-reduced data; The dimensionality-reduced data is processed using the dimensionality-up matrix to obtain the bypass output data.

6. The robot control method according to claim 1, characterized in that, The step of determining the task instructions that the robot needs to perform on the simulation environment map to complete the task based on the instruction text and the simulation environment map includes: Obtain robot attribute information of the robot on the simulation environment map and object attribute information of the target object on the simulation environment map; Based on the robot attribute information of the robot on the simulation environment map, the object attribute information of the target object on the simulation environment map, and the target action, the task instruction corresponding to the instruction text is determined.

7. A control device for a robot, characterized in that, include: The parsing module is used to add a bypass module and a fusion module to a pre-defined large-scale language model to obtain an initial language model. An initial language model is trained based on a preset training set to obtain a preset text parsing model. The parameters of the preset large-scale language model in the initial language model are not updated during training, while the parameters of the bypass module in the initial language model are updated during training. The bypass module includes a dimensionality reduction matrix and an dimensionality increase matrix, which are obtained by performing low-rank decomposition on the weights of the preset large-scale language model. When user-input text is received, it is parsed to obtain at least one instruction text. A pre-constructed prompt template is obtained, which indicates that the text to be processed should be split into at least one instruction text. The prompt template and the text to be processed are input into the preset text parsing model to obtain at least one instruction text corresponding to the text to be processed. The instruction text includes a target action and the target object acting on the target action. The at least one instruction text is a series of instructions that the robot needs to execute to complete the intent expressed by the text to be processed. The acquisition module is used to acquire a simulation environment map corresponding to the real physical environment within a preset area; The determination module is used to determine, based on the instruction text and the simulation environment map, the task instructions that the robot needs to perform on the simulation environment map to complete the instruction text, and to obtain each task instruction corresponding to each instruction text; The control module is used to control the robot to execute each of the task instructions in sequence, and to control the robot to capture execution videos during the execution of each of the task instructions in sequence; after controlling the robot to execute each of the task instructions in sequence, the execution video is output, and the constructed prompt template is adjusted according to the execution video.

8. An electronic device, characterized in that, The device includes a memory and a processor, the memory storing a computer program and the processor running the computer program in the memory to perform the steps in the robot control method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform the steps of the robot control method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by the processor, they implement the steps in the robot control method according to any one of claims 1 to 6.

Images