Heterogeneous robot control method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a heterogeneous robot control method and device, electronic equipment and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: generating a task cue word according to acquired total task data and environment perception data; and in the dialogue of the current round, responding to the obtained historical dialogue text and the task prompt word, generating a new dialogue text, updating the new dialogue text into the historical dialogue text, and completing the dialogue of the current round until all the heterogeneous robots complete speaking, when the target heterogeneous robot is the first speaking in the plurality of heterogeneous robots, the historical dialogue text is null; and after at least one round of dialogue is ended, based on a historical dialogue text updated by the last speaking heterogeneous robot, extracting sub-task data corresponding to the target heterogeneous robot, and executing a corresponding sub-target task. According to the method, the efficiency of cooperatively completing the total target task by multiple heterogeneous robots can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a control method, device, electronic device and storage medium for a heterogeneous robot. Background Art

[0002] Heterogeneous robots refer to robots with different characteristics (such as different degrees of freedom, different end effector types) in a collaborative system. In actual application, complex overall target tasks are completed through the collaboration between multiple heterogeneous robots.

[0003] In the related art, subtasks to be completed are manually set in advance for each heterogeneous robot to achieve the purpose of completing complex overall target tasks. However, since the working environment of the heterogeneous robots is usually changing all the time, the pre-set subtasks cannot flexibly respond to dynamically changing environmental factors, which leads to low efficiency of the heterogeneous robots in completing the established overall target tasks. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a control method, device, electronic device and storage medium for a heterogeneous robot, aiming to improve the efficiency of multiple heterogeneous robots collaborating to complete an overall target task.

[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a control method for a heterogeneous robot, which is applied to any target heterogeneous robot among a plurality of heterogeneous robots. The method comprises:

[0006] Obtaining total task data and environmental perception data around the target heterogeneous robot, and generating task prompt words based on the total task data and environmental perception data;

[0007] In the current round of dialogue, the historical dialogue texts between multiple heterogeneous robots are obtained, and responses are made based on the historical dialogue texts and task prompts to generate new dialogue texts, and the new dialogue texts are updated to the historical dialogue texts until all heterogeneous robots have finished speaking, completing the current round of dialogue. Among them, when the target heterogeneous robot is the first to speak among multiple heterogeneous robots, the historical dialogue text is empty;

[0008] After at least one round of dialogue is finished, subtask data corresponding to the target heterogeneous robot is extracted based on the historical dialogue text updated by the last speaking heterogeneous robot, and the corresponding sub-target task is executed based on the subtask data.

[0009] In some embodiments, after extracting the subtask data corresponding to the target heterogeneous robot, the method further includes:

[0010] Based on the subtask data, obtain the corresponding task text information, task constraint information and task waypoint information;

[0011] If the task text information includes the preset keywords, it is determined that the task text information verification has passed;

[0012] If the task constraint information is consistent with the preset constraint action, it is determined that the task constraint information verification is passed;

[0013] If the mission waypoint information is consistent with the preset inverse kinematics data, it is determined that the mission waypoint information verification has passed;

[0014] When the task text information, task constraint information and task waypoint information are all verified, the subtask data verification is passed;

[0015] If the subtask data is verified, the corresponding sub-target task is executed based on the subtask data.

[0016] In some embodiments, responding based on the historical dialogue text and the task prompt words to generate new dialogue text includes:

[0017] The task prompt words are updated according to the historical dialogue text to obtain updated task prompt words;

[0018] Generate new dialogue text based on the updated task prompt words.

[0019] In some embodiments, after obtaining corresponding task text information, task constraint information and task waypoint information based on the subtask data, the method further includes:

[0020] If the subtask data verification fails, a failure assessment message is generated;

[0021] The task prompt words are updated according to the failure evaluation information to obtain updated task prompt words, and the next round of dialogue is conducted based on the updated task prompt words;

[0022] After the next round of dialogue ends, the subtask data corresponding to the target heterogeneous robot is extracted based on the historical dialogue text updated by the last speaking heterogeneous robot, and the corresponding sub-target task is executed based on the subtask data.

[0023] In some embodiments, after executing the corresponding sub-target task based on the sub-task data, the method further includes:

[0024] If the sub-target task fails to be executed, the environmental perception data around the target heterogeneous robot is re-acquired to obtain updated environmental perception data;

[0025] According to the total task data and the updated environment perception data, the task prompt words are updated to obtain updated task prompt words, and the next dialogue round is carried out based on the updated task prompt words;

[0026] After the next round of dialogue ends, the subtask data corresponding to the target heterogeneous robot is extracted based on the historical dialogue text updated by the last speaking heterogeneous robot, and the corresponding sub-target task is executed based on the subtask data.

[0027] In some embodiments, when the heterogeneous robots include a leader robot, after updating the new dialogue text to the historical dialogue text, the method further includes:

[0028] After at least one round of dialogue is over, the leader robot becomes the last heterogeneous robot to speak and updates the historical dialogue text;

[0029] Receive subtask data corresponding to the target heterogeneous robot determined by the leader robot, and execute the corresponding sub-target task based on the subtask data.

[0030] In some embodiments, after the current round of dialogue is completed, the process further includes:

[0031] After the preset free discussion time is over, the historical dialogue text updated by the last speaking heterogeneous robot is obtained, wherein during the free discussion time, the target heterogeneous robot speaks zero times or at least once;

[0032] Based on the historical dialogue text, the subtask data corresponding to the target heterogeneous robot is extracted, and the corresponding sub-target task is executed based on the subtask data.

[0033] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a control device for a heterogeneous robot, the device comprising:

[0034] An acquisition module is used to acquire total task data and environmental perception data around the target heterogeneous robot, and generate task prompt words according to the total task data and environmental perception data;

[0035] The current round dialogue module is used to obtain the historical dialogue texts between multiple heterogeneous robots in the current round of dialogue, and respond based on the historical dialogue texts and task prompts to generate new dialogue texts, and update the new dialogue texts to the historical dialogue texts until all heterogeneous robots have finished speaking, completing the current round of dialogue. When the target heterogeneous robot is the first to speak among multiple heterogeneous robots, the historical dialogue text is empty;

[0036] The target execution module is used to extract the subtask data corresponding to the target heterogeneous robot based on the historical dialogue text updated by the last speaking heterogeneous robot after at least one round of dialogue ends, and execute the corresponding sub-target task based on the subtask data.

[0037] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the control method of the heterogeneous robot of the above-mentioned first aspect is implemented.

[0038] To achieve the above objectives, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the control method of the heterogeneous robot of the first aspect is implemented.

[0039] The control method, device, electronic device and storage medium of the heterogeneous robot proposed in the present application obtain total task data and environmental perception data around the target heterogeneous robot, and generate task prompt words according to the total task data and environmental perception data; in the current round of dialogue, obtain the historical dialogue text between multiple heterogeneous robots, and respond based on the historical dialogue text and task prompt words to generate new dialogue text, and update the new dialogue text to the historical dialogue text until all heterogeneous robots have finished speaking, completing the current round of dialogue, wherein when the target heterogeneous robot is the first to speak among multiple heterogeneous robots, the historical dialogue text is empty; after at least one round of dialogue is completed, based on the historical dialogue text updated by the last speaking heterogeneous robot, extract the subtask data corresponding to the target heterogeneous robot, and execute the corresponding sub-target task based on the subtask data. The multiple heterogeneous robots in the embodiment of the present application discuss the feasibility of the overall target task corresponding to the overall task data by speaking one by one. This approach allows each heterogeneous robot to fully consider the previous discussion and existing decisions when speaking. In this way, it is possible to reduce information islands, efficiently promote the continuous evolution of information flow, and thereby improve the efficiency of multiple heterogeneous robots collaborating and autonomously completing target tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a schematic diagram of an optional application scenario of the control device of the heterogeneous robot provided in an embodiment of the present application;

[0041] Figure 2 is an optional flow chart of the control method of the heterogeneous robot provided in the embodiment of the present application;

[0042] Figure 3 It is an optional schematic diagram for discussing multiple heterogeneous robots in the control method of heterogeneous robots provided in the embodiment of the present application;

[0043] Figure 4 yes Figure 2 A flowchart of an implementation of step 102 in FIG.

[0044] Figure 5 yes Figure 2 An implementation flow chart after step 103 in FIG. 1 ;

[0045] Figure 6 yes Figure 5 An implementation flow chart after step 301 in FIG. 1 ;

[0046] Figure 7 yes Figure 6 An implementation flow chart after step 403 in FIG. 1 ;

[0047] Figure 8 yes Figure 2 An implementation flow chart after step 102 in FIG. 1 ;

[0048] Fig. 9 yes Figure 2 An implementation flow chart after step 101 in FIG. 1 ;

[0049] Fig.10 is an optional flow chart of a control device for a heterogeneous robot provided in an embodiment of the present application;

[0050] Fig.11 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0052] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0054] First, some nouns involved in this application are analyzed:

[0055] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0056] Natural language processing (NLP): NLP uses computers to process, understand and apply human languages ​​(such as Chinese, English, etc.). NLP is a branch of artificial intelligence and an interdisciplinary subject between computer science and linguistics. It is often referred to as computational linguistics. Natural language processing includes grammatical analysis, semantic analysis, and text understanding. Natural language processing is often used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information intent recognition, information extraction and filtering, text classification and clustering, public opinion analysis and opinion mining. It involves data mining related to language processing, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computing.

[0057] Information Extraction: A text processing technology that extracts specified types of entity, relationship, event and other factual information from natural language text and forms structured data output. Information extraction is a technology that extracts specific information from text data. Text data is composed of some specific units, such as sentences, paragraphs, and chapters. Text information is composed of some small specific units, such as characters, words, phrases, sentences, paragraphs, or a combination of these specific units. Extracting noun phrases, names, place names, etc. from text data are all text information extraction. Of course, the information extracted by text information extraction technology can be various types of information.

[0058] Heterogeneous robots refer to robots with different characteristics (such as different degrees of freedom, different end effector types) in a collaborative system. In practical applications, multiple heterogeneous robots are controlled to collaborate to complete complex target tasks.

[0059] In the related art, subtasks to be completed are manually set in advance for each heterogeneous robot to achieve the purpose of completing complex target tasks. However, since the working environment of the heterogeneous robot is usually changing all the time, the pre-set subtasks cannot flexibly respond to dynamically changing environmental factors, which leads to low efficiency of the heterogeneous robot in completing the set target tasks.

[0060] Based on this, the embodiments of the present application provide a control method, device, electronic device and storage medium for a heterogeneous robot, aiming to improve the efficiency of multiple heterogeneous robots collaborating to complete a target task.

[0061] It should be noted that in the embodiments of the present application, when it comes to information related to user characteristics such as user basic information or user identity, the user's permission or consent will be obtained first, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. In addition, when the embodiments of the present application need to obtain sensitive personal information of the user, the user's separate permission or separate consent will be obtained first. After the separate permission or separate consent of the user is clearly obtained, when multiple heterogeneous robots are used to move the target items in the target area managed by a certain user, it is necessary to obtain the environmental perception data of the target area to generate task prompts. In this case, the embodiments of the present application will first obtain the consent of the relevant users, otherwise the environmental perception data used in the embodiments of the present application cannot be obtained. The other relevant data used in the embodiments of the present application are also obtained legally and with the consent of the relevant users, and will not be repeated here.

[0062] The control method, device, electronic device and storage medium of a heterogeneous robot provided in the embodiments of the present application are specifically described by the following embodiments. First, the application scenario of the control device of the heterogeneous robot proposed in the embodiments of the present application (for the convenience of description, it may also be referred to as "control device" below) is described, such as Figure 1 As shown, Figure 1 is a schematic diagram of an optional application scenario of the control device of the heterogeneous robot provided in the embodiment of the present application. Figure 1In the embodiment, multiple heterogeneous robots (including heterogeneous robot a, heterogeneous robot b and heterogeneous robot c) work together in a certain target area. When it is necessary to jointly complete a certain target task with heterogeneous robot a, heterogeneous robot b and heterogeneous robot c, taking heterogeneous robot b as an example (heterogeneous robot b is called the target heterogeneous robot at this time): the total task data and the environmental perception data around the target heterogeneous robot are obtained, and the task prompt words are generated according to the total task data and the environmental perception data; in the current round of dialogue, the historical dialogue texts between the multiple heterogeneous robots are obtained (for example, before this, the heterogeneous robot a and the heterogeneous robot c had a dialogue, then the historical dialogue text is updated according to the dialogue text generated by the heterogeneous robot a and the heterogeneous robot c), and respond based on the historical dialogue text and the task prompt words, generate a new dialogue text, and update the new dialogue text to the historical dialogue text; then, the heterogeneous robot b extracts the subtask data from the updated historical dialogue text, and determines its corresponding sub-target task according to the subtask data. The same is true for heterogeneous robots a and c, which will not be elaborated here. In other words, multiple heterogeneous robots can determine their corresponding sub-target tasks through dialogue, and multiple heterogeneous robots can achieve the purpose of completing the overall target task by completing the sub-target tasks, thereby improving the efficiency of multiple heterogeneous robots collaborating to complete the target task.

[0063] After understanding the application scenarios of the control device proposed in the present application, the control method of the heterogeneous robot proposed in the embodiment of the present application (for the sake of convenience, it may also be referred to as the "control method" below) is described.

[0064] In the embodiments of the present application, the description will be made from the perspective of a control device, which can be integrated into a computer device, such as a server. Figure 2 As shown, Figure 2 is an optional flow chart of the control method of the heterogeneous robot provided in the embodiment of the present application. Figure 2 The method may include but is not limited to the following steps 101 to 103. When the control device executes the control method, the specific process is as follows. It should be noted that the present embodiment Figure 2 The order of step 101 to step 103 is not specifically limited, and the order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0065] Step 101, obtaining total task data and environmental perception data around the target heterogeneous robot, and generating task prompt words according to the total task data and environmental perception data.

[0066] The following is a detailed description of step 101.

[0067] Among them, heterogeneous robots mainly refer to robots that have differences in function, performance, design or application scenarios. These differences may come from the robot's structure, material, control system, perception ability, task execution ability and other aspects. For example, Figure 1 As shown in the figure, heterogeneous robots a, b and c are all heterogeneous robots, but their task execution capabilities are different. For example, heterogeneous robot a is used to detect the target object, heterogeneous robot b is used to grasp the target object, and heterogeneous robot c is used to carry the target object. Multiple heterogeneous robots participating in the execution of the overall target task form a collaborative system.

[0068] Furthermore, each heterogeneous robot is equipped with a large language model (LLM), and the corresponding LLM performance is different according to the different capabilities of each heterogeneous robot. For example, the LLM carried by a heterogeneous robot with simple grasping capabilities is simpler than the LLM of other heterogeneous robots. In this way, the simple LLM can quickly respond to and process simple instructions, while the complex LLM can process more complex language inputs and generate more accurate responses, thereby saving total computing resources and improving overall work efficiency.

[0069] Among them, LLM refers to a type of artificial intelligence model that uses a large amount of data for training and aims to understand input content and generate natural language text through natural language processing (NLP) and information extraction. For example, LLM can be:

[0070] (1) Generative Pre-trained Transformer (GPT): The GPT series of models has tens of billions of parameters and can achieve a wide range of natural language understanding and generation tasks;

[0071] (2) Bidirectional Encoder Representations from Transformers (BERT): BERT significantly improves the accuracy of natural language understanding by taking contextual information into account at the same time. Among them, BERT variants include the Robert pre-training model (Bidirectional Encoder Representations from Transformers, RoBERTa) and the Albert pre-training model (A Lite version of BERT, ALBERT).

[0072] (3) Neural network architecture model based on self-attention mechanism (Transformer): It is a model that uses the attention mechanism to process sequence data. Transformer greatly improves the training efficiency of the model by processing all elements in the sequence in parallel.

[0073] It should be noted that, in addition to the several optional LLM types illustrated in the above examples, the LLM in the embodiments of the present application can also be specifically selected according to actual conditions, and the embodiments of the present application do not limit this.

[0074] The total task data is used to characterize the relevant data of the total target task that multiple heterogeneous robots need to complete together. For example, in a certain target work area of ​​industrial manufacturing, multiple heterogeneous robots include handling robots, assembly robots, and inspection robots. Multiple heterogeneous robots need to collaborate to complete the total target task of finished product inspection at a fixed position. The total task data corresponding to the total target task includes but is not limited to the map of the target work area, the fixed position, the time limit for collaborative completion, etc.

[0075] Furthermore, each heterogeneous robot will obtain the same total task data, which is used to indicate the conversation direction of each heterogeneous robot in the subsequent conversation process, and to guide the overall completion goal of each heterogeneous robot. That is, the sub-target tasks determined by each heterogeneous robot should be aimed at completing the overall target task.

[0076] Among them, the environmental perception data may include but is not limited to the location information of the target heterogeneous robot itself, the relevant data of the surrounding environment in the target working area that the target heterogeneous robot can perceive, etc. For example, when the target heterogeneous robot includes a camera, the perception data can be image data that characterizes the size and shape of each object in the target working area; when the target heterogeneous robot includes a temperature sensor, the perception data includes a temperature value used to characterize the temperature of the target working area; when the target heterogeneous robot includes a laser radar, the perception data includes coordinate data used to characterize the precise position of each object in the target working area. Alternatively, the environmental perception data can be specifically set according to the actual situation and the different capabilities of each heterogeneous robot, and the embodiments of the present application do not limit this.

[0077] Furthermore, each heterogeneous robot can obtain different environmental perception data according to its own capabilities. Based on this, each heterogeneous robot can generate corresponding task prompt words according to the overall task data and personalized environmental perception data, so that each heterogeneous robot can then generate dialogue text for discussion according to the task prompt words.

[0078] Among them, the task prompt words include the first key information in the total task data, the second key information in the environmental perception data, and the dialogue guidance instructions. Furthermore, the first key information is obtained by the LLM model understanding the total task data obtained and extracting key elements from it; the second key information is obtained by the LLM model understanding the environmental perception data obtained by the heterogeneous robot and extracting key elements from it. Exemplarily, the first key information is generated through the following steps: ① Obtaining the total task data: The total task data obtained by a heterogeneous robot is "Please move all goods marked as 'urgent' from area A to area B, and take inventory during the transportation process"; ② Understanding the total task data: The LLM model carried by the heterogeneous robot performs natural language processing on the total task data to identify the key elements of the task, such as action (transportation, inventory), object (goods marked as 'urgent'), starting position (area A), target position (area B), etc.; ③ Keyword extraction: Based on the understanding of the task data, the LLM model will further extract keywords. In this example, the keywords extracted by LLM include "transportation", "inventory", "urgent", "goods", "area A", "area B", etc. Then, the first key information is generated based on the extracted keywords.

[0079] It should be noted that in addition to the first key information and the second key information, other key information may be added according to actual conditions in the task prompt word. The key information may also include pre-set constraint information, such as the execution time of the total target task, etc. The embodiment of the present application does not limit the specific content of the key information.

[0080] Furthermore, the dialogue guidance instructions are used to guide the target heterogeneous robot on how to respond to other heterogeneous robots and correctly format the output results. The dialogue guidance instructions are usually added after the key information to clarify the specific details of the dialogue text expected to be generated or the additional operation content, etc. For example, the dialogue guidance instruction can be: "Excuse me, are there any other urgent goods that need to be handled first?".

[0081] It can be understood that the task prompt words can concisely convey to the target heterogeneous robot the content including but not limited to the task background, target work area environmental data, etc., and then can guide and promote the dialogue interaction and decision-making between multiple heterogeneous robots, thereby improving the accuracy of the generated dialogue text.

[0082] Step 102, in the current round of dialogue, obtain the historical dialogue texts between multiple heterogeneous robots, respond based on the historical dialogue texts and task prompts, generate new dialogue texts, and update the new dialogue texts to the historical dialogue texts until all heterogeneous robots have finished speaking, completing the current round of dialogue. Among them, when the target heterogeneous robot is the first to speak among multiple heterogeneous robots, the historical dialogue text is empty.

[0083] The following is a detailed description of step 102.

[0084] In some embodiments, after generating the corresponding task prompt words, each heterogeneous robot will conduct multiple rounds of dialogue based on the task prompt words. When it is the turn of the current heterogeneous robot to speak, the heterogeneous robot will first obtain the historical dialogue texts generated between the multiple heterogeneous robots, wherein the historical dialogue texts are generated during the current round of dialogue, or can be all the historical dialogue texts generated during all the rounds of dialogue before the current round of dialogue, so as to generate new dialogue texts based on the historical dialogue texts and the task prompt words, thereby helping to improve the accuracy of determining the corresponding sub-target task of the current heterogeneous robot.

[0085] Each heterogeneous robot can store the dialogue text generated by itself and the dialogue text received from other heterogeneous robots to form historical dialogue text. The historical dialogue text can help the heterogeneous robot better understand the background of the overall target task, execution progress and other information when speaking next time, and then make a more accurate response to the imagined operations of other heterogeneous robots.

[0086] like Figure 3 As shown, Figure 3 This is an optional multiple heterogeneous robot discussion diagram of the control method of the heterogeneous robot provided in the embodiment of the present application. In each round of dialogue, all heterogeneous robots participating in the execution of the overall target task need to speak one by one. When speaking, the heterogeneous robot first determines a new dialogue text based on the acquired historical dialogue text and the task prompt words generated by itself; then, the speaking heterogeneous robot stores the dialogue text and sends the dialogue text to other heterogeneous robots so that other heterogeneous robots can store the dialogue text to update the historical dialogue text. In this way, the heterogeneous robot that has not spoken can make a more accurate speech based on the updated historical dialogue text.

[0087] Furthermore, the speaking order of heterogeneous robots can be divided into the following cases:

[0088] (1) First speech: If the target heterogeneous robot is the first to speak among multiple heterogeneous robots, the historical dialogue text is empty. The target heterogeneous robot generates a dialogue text based on the task prompt and sends it to other heterogeneous robots until each heterogeneous robot has spoken once, completing the current round of dialogue.

[0089] (2) Non-first speech: If the target heterogeneous robot is not the first speaker among multiple heterogeneous robots, then the target heterogeneous robot will first obtain the historical dialogue text updated by other heterogeneous robots in the current round of dialogue; then, the target heterogeneous robot generates a new dialogue text based on the task prompt word and the obtained historical dialogue text; and, while updating the historical dialogue text based on the new dialogue text, it sends the dialogue text to other heterogeneous robots.

[0090] In some embodiments, Figure 4 As shown, Figure 4 yes Figure 2 A flowchart for implementing step 102 in the embodiment of the present invention, responding based on the historical dialogue text and the task prompt word to generate a new dialogue text, including the following steps 201 to 202:

[0091] Step 201, updating the task prompt words according to the historical dialogue text to obtain updated task prompt words.

[0092] Step 202: Generate new dialogue text based on the updated task prompt words.

[0093] Steps 201 to 202 are described in detail below.

[0094] In some embodiments, after obtaining the historical dialogue text, the target heterogeneous robot first updates the task prompt words according to the historical dialogue text. The task prompt words may also include historical dialogue key information. Specifically, the historical dialogue key information is generated by the LLM after understanding the historical dialogue text. In this way, the updated task prompt words may include: first key information, second key information, historical dialogue key information and dialogue guiding instructions.

[0095] Furthermore, the target heterogeneous robot can respond to the historical dialogue text according to the updated task prompt words to generate a new dialogue text, wherein the new dialogue text may include a summary of the dialogue text generated by the heterogeneous robot that spoke previously, as well as the response content generated by itself.

[0096] For example, the historical dialogue text is: "Robot 1: 'I have the ability to grab objects under 1,000 kilograms. I am currently in target area A and can reach target area B within one minute.'", then the new dialogue text generated by the target heterogeneous robot (robot 2) can be: "I am in target area B, I am in target area B, I have the ability to carry objects under 1,000 kilograms to the designated location, and I can handle multiple goods at the same time. According to the instructions of Robot 1, Robot 2 recommends that Robot 1 first reach target area B, grab and place all the goods that can be carried, and then Robot 2 can assist in transporting these goods to the final destination. At the same time, I will also check whether there are other goods or obstacles in target area B that require special handling."

[0097] Among them, the speaking order of each heterogeneous robot can be pre-set, so that after each heterogeneous robot completes the current round of speech in the established order, the current round of dialogue ends; or, any heterogeneous robot can be designated to speak first, and then, after receiving the dialogue text of the first heterogeneous robot, other heterogeneous robots understand the dialogue text through LLM and determine whether they can establish a task collaboration processing relationship with the first heterogeneous robot. If so, they speak; if multiple heterogeneous robots can establish a task collaboration processing relationship with the first heterogeneous robot that speaks, they can speak in turn according to the distance between them and the first heterogeneous robot that speaks. Of course, the speaking order of each heterogeneous robot can also be set according to the actual situation, and the embodiment of the present application does not limit this.

[0098] It is understandable that each heterogeneous robot can construct corresponding response content based on the dialogue text of the heterogeneous robot that spoke previously, thereby making the entire discussion process more natural and coherent, avoiding information isolation and repetition. In this way, by simulating the alternating speech method in a real dialogue, the understanding ability of multiple heterogeneous robots to deal with the overall target task is improved, thereby improving the efficiency of multiple heterogeneous robots collaborating to complete the target task.

[0099] Step 103, after at least one round of dialogue ends, based on the historical dialogue text updated by the last speaking heterogeneous robot, subtask data corresponding to the target heterogeneous robot is extracted, and the corresponding sub-target task is executed based on the subtask data.

[0100] The following is a detailed description of step 103.

[0101] In some embodiments, each heterogeneous robot participates in at least one round of dialogue, that is, each heterogeneous robot speaks at least once. Since the next heterogeneous robot to speak will summarize based on the historical dialogue text, as the number of heterogeneous robots that speak continues to increase, the content of the dialogue will continue to deepen, and the sub-target tasks that each heterogeneous robot needs to complete will also become clear.

[0102] Furthermore, if the performance of the LLM model carried by the heterogeneous robots is strong enough, the last speaking heterogeneous robot will determine the sub-target tasks that each heterogeneous robot needs to perform; or, the last speaking heterogeneous robot will not perform task summary, and each heterogeneous robot will extract its corresponding sub-task data from the last updated historical dialogue text, and determine the sub-target tasks to be performed based on the sub-task data.

[0103] Among them, subtask data refers to the specific task information for the target heterogeneous robot extracted from the historical dialogue text. These data usually include key elements such as the specific requirements of the task, execution conditions, required resources, and expected results. They are the basis for guiding the target heterogeneous robot to perform sub-target tasks.

[0104] Among them, the sub-target task refers to the task content determined by the target heterogeneous robot based on the sub-task data. For example, "driving to the target area B and grabbing all items on the ground in the target area B" is a sub-target task; each heterogeneous robot corresponds to one or more sub-target tasks, and achieves the overall target task after completing the execution of all sub-target tasks.

[0105] Furthermore, after the sub-target task is determined, the target heterogeneous robot can pass the relevant data to its own execution components, such as a multi-arm motion planner based on rapidly-exploring random trees (RRT), to execute the sub-target task.

[0106] It can be understood that the multiple heterogeneous robots in the embodiments of the present application discuss the feasibility of the overall target task by speaking one by one. This approach allows each heterogeneous robot to fully consider the previous discussion and existing decisions when speaking. In this way, information islands can be reduced, and the continuous evolution of information flow can be efficiently promoted, thereby improving the efficiency of multiple heterogeneous robots collaborating and autonomously completing target tasks.

[0107] In some embodiments, Figure 5 As shown, Figure 5 yes Figure 2 An implementation flow chart after step 103 in the embodiment, after extracting the subtask data corresponding to the target heterogeneous robot, further includes the following steps 301 to 306:

[0108] Step 301, based on the subtask data, obtain the corresponding task text information, task constraint information and task waypoint information.

[0109] Step 302: If the task text information includes a preset keyword, it is determined that the task text information has passed verification.

[0110] Step 303: If the task constraint information is consistent with the preset constraint action, it is determined that the task constraint information verification is passed.

[0111] Step 304: If the mission waypoint information is consistent with the preset inverse kinematics data, it is determined that the mission waypoint information verification has passed.

[0112] Step 305: When the task text information, task constraint information and task waypoint information are all verified, the subtask data is verified.

[0113] Step 306: If the subtask data is verified, the corresponding sub-target task is executed based on the subtask data.

[0114] Steps 301 to 306 are described in detail below.

[0115] In some embodiments, the target heterogeneous robot will verify the subtask data before executing the sub-target task. The pre-verification mechanism of the embodiment of the present application ensures the rationality and feasibility of the target heterogeneous robot plan, thereby improving the practicality and robustness of the collaborative system.

[0116] Further, based on the extracted subtask data, the corresponding task text information is determined. The task text information is obtained by parsing the subtask data, and the task text information includes format information and plan information. Accordingly, the preset keywords include format keywords and plan keywords. The format keywords define how the subtask data should be formatted and presented to ensure that the target heterogeneous robot can accurately parse and execute the corresponding sub-target tasks. Specifically, the format keywords may include the task execution order, task execution priority, specific separators, etc. The plan keywords define the task framework that the target heterogeneous robot should execute. For example, for a grabbing heterogeneous robot, its corresponding sub-target task should be roughly grabbing x items at position x (where x represents unknown). Then, if the subtask data obtained by the target heterogeneous robot does not contain keywords such as "target area" and "target item", it is considered that the task text information verification fails; otherwise, the verification passes.

[0117] Further, based on the extracted subtask data, the corresponding task constraint information is determined. Among them, the task constraint information includes but is not limited to time constraints (such as the sub-target task must be completed within a certain time period), space constraints (such as the sub-target task can only be executed in a specified target area), resource constraints (such as the maximum amount of computing resources required to execute the sub-target task), etc.; for example, the subtask data corresponding to the target heterogeneous robot a is "drive the target heterogeneous robot to the target area B to grab the target object on the ground", and then the target heterogeneous robot obtains the corresponding preset constraint action of "target area B", such as the constraint action can be "target area B does not allow target heterogeneous robots a and target heterogeneous robots b to enter", so in this example, the task constraint information verification fails.

[0118] Furthermore, based on the extracted subtask data, the corresponding task waypoint information is determined. Among them, the task waypoint information is used to characterize the target position points that the target heterogeneous robot needs to pass through in the process of executing the sub-target task, and the target action that needs to be performed at the target position point. Inverse kinematics refers to the process of setting the parameters of the joint movable object to achieve the required posture, and the inverse kinematics data is the data related to inverse kinematics. Inverse kinematics parameters may include but are not limited to motion joint angles, motion speeds, etc. The target heterogeneous robot verifies whether the target heterogeneous robot will encounter abnormal situations such as collisions and overweight when executing the corresponding sub-target task by comparing the task waypoint information determined from the subtask data with the preset inverse kinematics parameters, and then determines whether the verification of the task waypoint information has passed.

[0119] It should be noted that keywords, constraint actions and inverse kinematics data can be specifically set according to actual conditions, and the embodiments of the present application do not limit this.

[0120] Furthermore, when the task text information, task constraint information and task waypoint information are all verified, the subtask data is verified; then, the target heterogeneous robot can determine and execute the corresponding sub-target task based on the subtask data.

[0121] In some embodiments, if the current round of dialogue has not ended, but the target heterogeneous robot has been able to determine its own subtask data from the historical dialogue text, the target heterogeneous robot can verify the extracted subtask data during the discussion of other heterogeneous robots, thereby speeding up the efficiency of the collaborative system in processing the overall target task.

[0122] Alternatively, after at least one round of dialogue is completed and each heterogeneous robot determines its corresponding subtask data, each heterogeneous robot shares the corresponding subtask data with other heterogeneous robots. While verifying its own corresponding subtask data, the target heterogeneous robot takes the corresponding subtask data of other heterogeneous robots as one of the considerations to ensure that each heterogeneous robot does not encounter abnormal situations such as collision when executing the corresponding sub-target tasks.

[0123] In some embodiments, Figure 6 As shown, Figure 6 yes Figure 5 An implementation flow chart after step 301 in the embodiment, after obtaining the corresponding task text information, task constraint information and task waypoint information based on the subtask data, further includes the following steps 401 to 403:

[0124] Step 401: If the subtask data verification fails, a failure evaluation message is generated.

[0125] Step 402: update the task prompt word according to the failure evaluation information to obtain an updated task prompt word, and conduct the next round of dialogue based on the updated task prompt word.

[0126] Step 403, after the next round of dialogue ends, based on the historical dialogue text updated by the last speaking heterogeneous robot, the subtask data corresponding to the target heterogeneous robot is extracted, and the corresponding sub-target task is executed based on the subtask data.

[0127] Steps 401 to 403 are described in detail below.

[0128] The failure evaluation information is used to indicate the reason why the subtask data verification fails. When the subtask data verification fails, the target heterogeneous robot can generate failure evaluation information and update the task prompt word based on the failure evaluation information. In addition, the failure evaluation information can include multiple reasons such as format error, logic error, insufficient authority, etc.

[0129] Furthermore, since the overall target task can only be successfully executed when all subtask data are verified, when the subtask data corresponding to any heterogeneous robot fails to be verified, multiple heterogeneous robots will conduct the next round of dialogue to re-determine the subtask data of each heterogeneous robot. In the next round of dialogue, the target heterogeneous robot will generate a new dialogue text based on the updated task prompt words to determine the corresponding sub-target task.

[0130] Furthermore, each heterogeneous robot can obtain the historical dialogue text generated in the previous dialogue round during the next dialogue round, so as to generate a new dialogue text based on the historical dialogue text and the dialogue text updated based on the failure evaluation information. It is understandable that the historical dialogue texts generated in multiple rounds can help enrich the heterogeneous robot's understanding of the overall target task, thereby improving the efficiency of the target heterogeneous robot in determining the sub-target tasks that can be correctly executed. Among them, the specific implementation of step 403 is similar to step 103, which will not be repeated here.

[0131] In some embodiments, Figure 7 As shown, Figure 7 yes Figure 6 An implementation flow chart after step 403 in the embodiment, after executing the corresponding sub-target task based on the sub-task data, further includes the following steps 501 to 503:

[0132] Step 501: If the sub-target task fails to be executed, the environmental perception data around the target heterogeneous robot is reacquired to obtain updated environmental perception data.

[0133] Step 502, updating the task prompt words according to the total task data and the updated environment perception data to obtain updated task prompt words, and conducting the next dialogue round based on the updated task prompt words.

[0134] Step 503, after the next round of dialogue ends, based on the historical dialogue text updated by the last speaking heterogeneous robot, the subtask data corresponding to the target heterogeneous robot is extracted, and the corresponding sub-target task is executed based on the subtask data.

[0135] Steps 501 to 503 are described in detail below.

[0136] In some embodiments, even if the subtask data corresponding to each heterogeneous robot has been verified, the sub-target task may still fail to execute. The reasons for this phenomenon include but are not limited to hardware failure of the heterogeneous robot, environmental changes (such as light changes, temperature fluctuations, electromagnetic interference, etc.), and cooperation failure between heterogeneous robots (for example, the target object grasped by heterogeneous robot b is not accurately placed on the heterogeneous robot c used for transportation).

[0137] Furthermore, the failure of the sub-target task indicates that the target heterogeneous robot has moved, or the target object in the target area has moved, that is, the environment of the target area has changed. In this case, the target heterogeneous robot needs to reacquire the environmental perception data around it, and then update the task prompt word; then, the target heterogeneous robot will conduct the next round of local dialogue based on the updated task prompt word to re-determine the sub-target task to be performed. Among them, the specific implementation of step 503 is similar to step 103, which will not be repeated here.

[0138] Furthermore, after the sub-target task fails to execute, the target heterogeneous robot can autonomously analyze the reasons for the task failure and update the task prompt words based on this; and, in the next round of dialogue, the target heterogeneous robot can also obtain the historical dialogue text generated in the previous round or previous rounds to further help the target heterogeneous robot understand the reasons for the failure of the sub-target task, and then more accurately re-determine the corresponding sub-target task.

[0139] It is understandable that after the target heterogeneous robot fails to perform the sub-target task, it can re-determine the corresponding sub-target task by autonomously updating the environmental perception data. Since the working environment of each heterogeneous robot is not static during the collaborative work, for example, the heterogeneous robot b equipped with a camera needs to go to the target area B to grab the target object b in turn. When the heterogeneous robot b arrives at the target area B, it finds that the target object b has been taken away by humans. At this time, the heterogeneous robot b needs to re-acquire the environmental perception data to re-determine the corresponding sub-target task; or, in the case of sufficient computing resources, the heterogeneous robot b can obtain the surrounding environmental perception data in real time. When the heterogeneous robot b finds that the target object b is not in the specified location, it will no longer continue to perform the sub-target task, but immediately stop the execution of the current sub-target task and re-determine the corresponding sub-target task.

[0140] In the related art, each heterogeneous robot usually performs a predetermined sub-target task. Since a single heterogeneous robot does not have the ability to coordinate other heterogeneous robots, when a sub-target task fails to be executed, relevant personnel need to manually reset the sub-target task. In contrast, after the target heterogeneous robot in the embodiment of the present application fails to execute a sub-target task, it can generate a new dialogue text and discuss with other heterogeneous robots to redetermine the sub-target task that can be executed, rather than stagnating or only reporting an error, thereby improving the efficiency of multiple heterogeneous robots in collaboratively processing the overall target task.

[0141] In some embodiments, Figure 8 As shown, Figure 8 yes Figure 2An implementation flow chart after step 102 in the embodiment, when the heterogeneous robots include a leading robot, after the new dialogue text is updated to the historical dialogue text, the following steps 601 to 602 are also included:

[0142] Step 601: After at least one round of dialogue ends, the leading robot becomes the last heterogeneous robot to speak and updates the historical dialogue text.

[0143] Step 602, receiving subtask data corresponding to the target heterogeneous robot determined by the leader robot, and executing the corresponding sub-target task based on the subtask data.

[0144] Steps 601 to 602 are described in detail below.

[0145] Among them, the LLM carried by the leader robot has better performance than the LLM carried by other heterogeneous robots, so the leader robot can have leadership or coordination capabilities in the collaborative system. When there is a leader robot among multiple heterogeneous robots, the leader robot will update the historical dialogue text as the last heterogeneous robot to speak in each round of dialogue, so that the leader robot can have a holistic understanding of the dialogue texts generated by other heterogeneous robots, and then better disassemble the overall target task and determine the sub-target tasks corresponding to each heterogeneous robot.

[0146] Furthermore, the leading robot can directly determine the subtask data corresponding to each heterogeneous robot from the perspective of the last speaker, and send multiple subtask data to each heterogeneous robot; after receiving the corresponding subtask data, the target heterogeneous robot will execute the corresponding sub-target task based on the subtask data.

[0147] Furthermore, the leading robot may be one that participates in executing the sub-target task; or, the leading robot may not participate in the execution of the sub-target task, and the leading robot only receives and analyzes the dialogue texts generated by the heterogeneous robots. The association relationship between the leading robot and other heterogeneous robots can be specifically set according to actual conditions, and the embodiments of the present application do not limit this.

[0148] In some embodiments, Fig. 9 As shown, Fig. 9 yes Figure 2 An implementation flow chart after step 101 in the embodiment, after completing the current round of dialogue, further includes the following steps 701 to 702:

[0149] Step 701, after the preset free discussion time ends, obtain the historical dialogue text updated by the last speaking heterogeneous robot, wherein during the free discussion time, the target heterogeneous robot speaks zero times or at least once.

[0150] Step 702: extract subtask data corresponding to the target heterogeneous robot based on the historical dialogue text, and execute the corresponding sub-target task based on the subtask data.

[0151] Steps 701 to 702 are described in detail below.

[0152] Among them, the free discussion time is a time period set to promote information exchange, exchange of opinions and problem solving among heterogeneous robots. During the free discussion time, heterogeneous robots can speak freely to further share information and discuss through the dialogue text in order to better determine the sub-target tasks of the target heterogeneous robot.

[0153] Furthermore, during the free discussion time, the target heterogeneous robot can remain silent, speak once, or speak multiple times, so that when silent, it can better improve the extracted subtask data by understanding the dialogue texts sent by other heterogeneous robots; and when speaking, it can use the dialogue texts as a carrier to further contribute its own views, information, and questions to other heterogeneous robots, so as to further promote the in-depth discussion and the determination of sub-target tasks.

[0154] It should be noted that the specific duration of the free discussion time can be set according to the actual situation. In addition, the sub-target tasks corresponding to each heterogeneous robot can be determined by the historical dialogue text updated by the last heterogeneous robot that spoke after the free discussion time ends; or, after the free discussion time ends, another round of dialogue is conducted, and each heterogeneous robot is allowed to speak again to supplement or organize important information or details, so that the target heterogeneous robot can extract more accurate sub-task data, and then better determine the sub-target task it needs to perform based on the sub-task data.

[0155] like Fig.10 As shown, Fig.10 This is an optional flow chart of a control device for a heterogeneous robot provided in an embodiment of the present application. The control device for a heterogeneous robot includes the following steps 801 to 803:

[0156] The acquisition module 801 is used to acquire the total task data and the environmental perception data around the target heterogeneous robot, and generate task prompt words according to the total task data and the environmental perception data.

[0157] The current round dialogue module 802 is used to obtain the historical dialogue texts between multiple heterogeneous robots in the current round of dialogue, and respond based on the historical dialogue texts and task prompts to generate new dialogue texts, and update the new dialogue texts to the historical dialogue texts until all heterogeneous robots have finished speaking, completing the current round of dialogue. Among them, when the target heterogeneous robot is the first to speak among multiple heterogeneous robots, the historical dialogue text is empty.

[0158] The target execution module 803 is used to extract the subtask data corresponding to the target heterogeneous robot based on the historical dialogue text updated by the last speaking heterogeneous robot after at least one round of dialogue ends, and execute the corresponding sub-target task based on the subtask data.

[0159] The control method, device, electronic device and storage medium of the heterogeneous robot proposed in the present application obtain total task data and environmental perception data around the target heterogeneous robot, and generate task prompt words according to the total task data and environmental perception data; in the current round of dialogue, obtain the historical dialogue text between multiple heterogeneous robots, and respond based on the historical dialogue text and task prompt words to generate new dialogue text, and update the new dialogue text to the historical dialogue text until all heterogeneous robots have finished speaking, completing the current round of dialogue, wherein when the target heterogeneous robot is the first to speak among multiple heterogeneous robots, the historical dialogue text is empty; after at least one round of dialogue is completed, based on the historical dialogue text updated by the last speaking heterogeneous robot, extract the subtask data corresponding to the target heterogeneous robot, and execute the corresponding sub-target task based on the subtask data. The multiple heterogeneous robots in the embodiment of the present application discuss the feasibility of the overall target task corresponding to the overall task data by speaking one by one. This approach allows each heterogeneous robot to fully consider the previous discussion and existing decisions when speaking. In this way, it is possible to reduce information islands, efficiently promote the continuous evolution of information flow, and thereby improve the efficiency of multiple heterogeneous robots collaborating and autonomously completing target tasks.

[0160] The specific implementation of the control device is basically the same as the specific implementation of the above-mentioned control method, and will not be repeated here.

[0161] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above control method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0162] like Fig.11 As shown, Fig.11 : is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application, the electronic device includes:

[0163] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0164] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the control method of the embodiment of the present application;

[0165] Input / output interface 903, used to implement information input and output;

[0166] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0167] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0168] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0169] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program, and the computer program implements the above-mentioned control method when executed by a processor.

[0170] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0171] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0172] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0173] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0174] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0175] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0176] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0177] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0178] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0179] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0180] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0181] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A control method for a heterogeneous robot, characterized in that: Applied to any one target heterogeneous robot among a plurality of heterogeneous robots, the method comprises: Acquire total task data and environmental perception data around the target heterogeneous robot, and generate task prompt words according to the total task data and the environmental perception data; In the current round of dialogue, historical dialogue texts between the plurality of heterogeneous robots are obtained, and responses are made based on the historical dialogue texts and the task prompt words to generate new dialogue texts, and the new dialogue texts are updated to the historical dialogue texts until all the heterogeneous robots have finished speaking, thus completing the current round of dialogue, wherein when the target heterogeneous robot is the first to speak among the plurality of heterogeneous robots, the historical dialogue text is empty; After at least one round of dialogue ends, subtask data corresponding to the target heterogeneous robot is extracted based on the historical dialogue text updated by the heterogeneous robot that spoke last, and the corresponding sub-target task is executed based on the subtask data.

2. The method according to claim 1, characterized in that After extracting and obtaining the subtask data corresponding to the target heterogeneous robot, the method further includes: Based on the subtask data, obtaining corresponding task text information, task constraint information and task waypoint information; If the task text information includes a preset keyword, determining that the task text information has passed verification; If the task constraint information is consistent with the preset constraint action, determining that the task constraint information is verified; If the mission waypoint information is consistent with the preset inverse kinematics data, it is determined that the mission waypoint information is verified; When the task text information, the task constraint information and the task waypoint information are all verified, the subtask data is verified; If the subtask data passes the verification, the corresponding sub-target task is executed based on the subtask data.

3. The method according to claim 1, characterized in that The step of responding based on the historical dialogue text and the task prompt word to generate a new dialogue text includes: The task prompt word is updated according to the historical conversation text to obtain the updated task prompt word; Generate new dialogue text based on the updated task prompt words.

4. The method according to claim 2, characterized in that: After acquiring the corresponding task text information, task constraint information and task waypoint information based on the subtask data, the method further includes: If the subtask data verification fails, a failure assessment message is generated; updating the task prompt word according to the failure evaluation information to obtain the updated task prompt word, and conducting the next round of dialogue based on the updated task prompt word; After the next round of dialogue ends, based on the historical dialogue text updated by the last speaking heterogeneous robot, subtask data corresponding to the target heterogeneous robot is extracted, and the corresponding sub-target task is executed based on the subtask data.

5. The method according to claim 1, characterized in that: After executing the corresponding sub-target task based on the sub-task data, the method further includes: If the sub-target task fails to be executed, reacquire the environmental perception data around the target heterogeneous robot to obtain updated environmental perception data; The task prompt word is updated according to the total task data and the updated environment perception data to obtain the updated task prompt word, and the next dialogue round is conducted based on the updated task prompt word; After the next round of dialogue ends, based on the historical dialogue text updated by the last speaking heterogeneous robot, subtask data corresponding to the target heterogeneous robot is extracted, and the corresponding sub-target task is executed based on the subtask data.

6. The method according to claim 1, characterized in that When the heterogeneous robots include a leading robot, after updating the new dialogue text to the historical dialogue text, the method further includes: After at least one round of dialogue ends, the leader robot serves as the last heterogeneous robot to speak and updates the historical dialogue text; The subtask data corresponding to the target heterogeneous robot determined by the leader robot is received, and the corresponding sub-target task is executed based on the subtask data.

7. The method according to claim 1, characterized in that After the current round of dialogue is completed, the following steps are also included: After the preset free discussion time is over, obtaining the historical dialogue text updated by the last speaking heterogeneous robot, wherein during the free discussion time, the target heterogeneous robot speaks zero times or at least once; Based on the historical dialogue text, subtask data corresponding to the target heterogeneous robot is extracted, and the corresponding sub-target task is executed based on the subtask data.

8. A control device for a heterogeneous robot, characterized in that: The device comprises: An acquisition module, used to acquire total task data and environmental perception data around the target heterogeneous robot, and generate task prompt words according to the total task data and the environmental perception data; The current round dialogue module is used to obtain the historical dialogue texts between the plurality of heterogeneous robots in the current round of dialogue, and respond based on the historical dialogue texts and the task prompt words to generate new dialogue texts, and update the new dialogue texts to the historical dialogue texts until all the heterogeneous robots have finished speaking, thus completing the current round of dialogue, wherein when the target heterogeneous robot is the first to speak among the plurality of heterogeneous robots, the historical dialogue text is empty; The target execution module is used to extract the subtask data corresponding to the target heterogeneous robot based on the historical dialogue text updated by the heterogeneous robot that spoke last after at least one round of dialogue is completed, and execute the corresponding sub-target task based on the subtask data.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the control method of the heterogeneous robot according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the control method of the heterogeneous robot according to any one of claims 1 to 7 is implemented.

Images