Heterogeneous robot control method and apparatus, electronic device, and storage medium

Abstract

Embodiments of the present application provide a control method and device of a heterogeneous robot, electronic equipment and storage medium, belonging to the technical field of artificial intelligence. The method comprises: generating a task prompt word according to obtained total task data and environment perception data; in the current round of dialogue, responding to the obtained historical dialogue text and the task prompt word to generate new dialogue text, updating the new dialogue text to the historical dialogue text, and completing the dialogue in the current round when all the heterogeneous robots complete speaking, wherein the historical dialogue text is empty when the target heterogeneous robot is the first to speak among the multiple heterogeneous robots; after the dialogue in at least one round ends, based on the historical dialogue text updated by the last speaking heterogeneous robot, sub-task data corresponding to the target heterogeneous robot is extracted, and the corresponding sub-target task is executed. The present application can improve the efficiency of multiple heterogeneous robots in completing the total target task.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a control method and device of heterogeneous robots, an electronic device and a storage medium. BACKGROUND

[0002] Heterogeneous robots refer to robots with different characteristics (such as different degrees of freedom and different end effector types) in a cooperative system. In practical applications, a complex overall target task is completed through cooperation between multiple heterogeneous robots.

[0003] In related technologies, subtasks to be completed are manually set for each heterogeneous robot in advance to achieve the purpose of completing a complex overall target task. However, since the working environment of the heterogeneous robots usually changes all the time, the pre-set subtasks cannot flexibly cope with dynamically changing environmental factors, thereby resulting in low efficiency of the heterogeneous robots in completing the intended overall target task. SUMMARY

[0004] The main purpose of the embodiments of the present application is to propose a control method and device of heterogeneous robots, an electronic device and a storage medium, which aims to improve the efficiency of multiple heterogeneous robots in completing an overall target task.

[0005] To achieve the above purpose, a first aspect of the embodiments of the present application proposes a control method of heterogeneous robots, applied to any target heterogeneous robot in multiple heterogeneous robots, and the method comprises:

[0006] Obtaining overall task data and environmental perception data around the target heterogeneous robot, and generating a task prompt word according to the overall task data and the environmental perception data;

[0007] In the current round of dialogue, obtaining historical dialogue text between the multiple heterogeneous robots, and responding based on the historical dialogue text and the task prompt word to generate new dialogue text, updating the new dialogue text to the historical dialogue text, and until all the heterogeneous robots finish speaking, completing the current round of dialogue, wherein when the target heterogeneous robot is the first to speak among the multiple heterogeneous robots, the historical dialogue text is empty;

[0008] After the dialogue of at least one round ends, based on the historical dialogue text updated by the last speaking heterogeneous robot, extracting subtask data corresponding to the target heterogeneous robot, and executing a corresponding sub-target task based on the subtask data.

[0009] In some embodiments, after the subtask data corresponding to the target heterogeneous robot is extracted, the method further comprises:

[0010] Based on the sub-task data, corresponding task text information, task constraint information and task waypoint information are obtained;

[0011] If the task text information includes a preset keyword, it is determined that the task text information is verified;

[0012] If the task constraint information is consistent with the preset constraint action, it is determined that the task constraint information is verified;

[0013] If the task waypoint information is consistent with the preset inverse kinematics data, it is determined that the task waypoint information is verified;

[0014] When the task text information, the task constraint information and the task waypoint information are all verified, the sub-task data is verified;

[0015] If the sub-task data is verified, the corresponding sub-target task is executed based on the sub-task data.

[0016] In some embodiments, the new dialogue text is generated based on the historical dialogue text and the task prompt word, including:

[0017] The task prompt word is updated according to the historical dialogue text to obtain an updated task prompt word;

[0018] The new dialogue text is generated based on the updated task prompt word.

[0019] In some embodiments, after obtaining the corresponding task text information, task constraint information and task waypoint information based on the sub-task data, the method further includes:

[0020] If the sub-task data is not verified, failure evaluation information is generated;

[0021] The task prompt word is updated according to the failure evaluation information to obtain an updated task prompt word, and the next round of dialogue is performed based on the updated task prompt word;

[0022] After the next round of dialogue ends, the sub-task data corresponding to the target heterogeneous robot is extracted based on the historical dialogue text updated by the last speaker heterogeneous robot, and the corresponding sub-target task is executed based on the sub-task data.

[0023] In some embodiments, after executing the corresponding sub-target task based on the sub-task data, the method further includes:

[0024] If the sub-target task fails to execute, the environment perception data around the target heterogeneous robot is re-obtained to obtain updated environment perception data;

[0025] According to the total task data and the updated environment perception data, the task prompt word is updated to obtain an updated task prompt word, and the next dialogue round is performed based on the updated task prompt word;

[0026] After the next round of dialogue ends, the target heterogeneous robot corresponding sub-task data is extracted based on the historical dialogue text updated by the last speaking heterogeneous robot, and the corresponding sub-target task is executed based on the sub-task data.

[0027] In some embodiments, when the leader robot is included in the heterogeneous robots, after the new dialogue text is updated into the historical dialogue text, it further includes:

[0028] After at least one round of dialogue ends, the leader robot is the last speaking heterogeneous robot and updates the historical dialogue text;

[0029] The target heterogeneous robot corresponding sub-task data determined by the leader robot is received, and the corresponding sub-target task is executed based on the sub-task data.

[0030] In some embodiments, after completing the current round of dialogue, it further includes:

[0031] After the end of the preset free discussion time, the historical dialogue text updated by the last speaking heterogeneous robot is obtained, wherein the target heterogeneous robot speaks zero times or at least once during the free discussion time;

[0032] Based on the historical dialogue text, the target heterogeneous robot corresponding sub-task data is extracted, and the corresponding sub-target task is executed based on the sub-task data.

[0033] To achieve the above-mentioned purpose, a second aspect of the embodiment of the present application proposes a control device of a heterogeneous robot, the device includes:

[0034] The acquisition module is configured to acquire total task data and environment perception data around the target heterogeneous robot, and generate a task prompt word according to the total task data and the environment perception data;

[0035] The current round dialogue module is configured to, in the current round of dialogue, acquire historical dialogue text between a plurality of heterogeneous robots, and respond based on the historical dialogue text and the task prompt word to generate new dialogue text, update the new dialogue text into the historical dialogue text, and complete the current round of dialogue when all the heterogeneous robots complete speaking, wherein the historical dialogue text is empty when the target heterogeneous robot is the first speaking one among the plurality of heterogeneous robots;

[0036] a target execution module, configured to, after the end of the dialogue of at least one round, extract sub-task data corresponding to the target heterogeneous robot based on the historical dialogue text updated by the last speaking heterogeneous robot, and execute a corresponding sub-target task based on the sub-task data.

[0037] To achieve the above object, a third aspect of the embodiments of the present application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the control method of the heterogeneous robot of the first aspect when executing the computer program.

[0038] To achieve the above object, a fourth aspect of the embodiments of the present application provides a computer readable storage medium, which stores a computer program, and the computer program implements the control method of the heterogeneous robot of the first aspect when executed by a processor.

[0039] The control method, device, electronic device and storage medium of the heterogeneous robot provided by the present application, by obtaining total task data and environmental perception data around the target heterogeneous robot, and generating a task prompt word according to the total task data and the environmental perception data; in the current round of dialogue, obtaining historical dialogue text among a plurality of heterogeneous robots, and responding based on the historical dialogue text and the task prompt word to generate new dialogue text, updating the new dialogue text into the historical dialogue text, until all the heterogeneous robots complete the speaking, the current round of dialogue is completed, wherein when the target heterogeneous robot is the first speaking one among the plurality of heterogeneous robots, the historical dialogue text is empty; after the end of the dialogue of at least one round, based on the historical dialogue text updated by the last speaking heterogeneous robot, sub-task data corresponding to the target heterogeneous robot is extracted, and a corresponding sub-target task is executed based on the sub-task data. The plurality of heterogeneous robots in the embodiments of the present application discuss the executability of the total target task corresponding to the total task data through speaking one by one. Such a way makes each heterogeneous robot fully consider the previous discussion and existing decisions when speaking, so that the information island is reduced, the continuous evolution of the information flow is efficiently promoted, and the efficiency of the plurality of heterogeneous robots in cooperatively and autonomously completing the target task is improved. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 is an optional application scenario schematic diagram of the control device of the heterogeneous robot provided by the embodiments of the present application;

[0041] Figure 2 is an optional flowchart of the control method of the heterogeneous robot provided by the embodiments of the present application;

[0042] Figure 3 is an optional discussion schematic diagram of a plurality of heterogeneous robots of the control method of the heterogeneous robot provided by the embodiments of the present application;

[0043] Figure 4 is Figure 2 an implementation flowchart of step 102 in

[0044] Figure 5 is Figure 2 an implementation flowchart after step 103 in

[0045] Figure 6 is Figure 5 an implementation flowchart after step 301 in

[0046] Figure 7 is Figure 6 an implementation flowchart after step 403 in

[0047] Figure 8 is Figure 2 an implementation flowchart after step 102 in

[0048] Figure 9 is Figure 2 an implementation flowchart after step 101 in

[0049] Figure 10 is an optional flowchart of the control device of the heterogeneous robot provided by the embodiments of the present application;

[0050] Figure 11 is a hardware structure schematic diagram of the electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0051] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0052] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a manner different from the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

[0054] First, the meanings of several terms involved in the present application are explained:

[0055] Artificial intelligence (AI): is a new technical science of studying, developing theories, methods, technologies and application systems for simulating, extending and expanding human intelligence; artificial intelligence is a branch of computer science, and artificial intelligence attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. The research in this field includes robots, language recognition, image recognition, natural language processing and expert systems, etc. Artificial intelligence can simulate the information process of human consciousness and thinking. Artificial intelligence is also the theory, method, technology and application system of using digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, to perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0056] Natural language processing (NLP): NLP uses computers to process, understand and use human language (such as Chinese, English, etc.). NLP is a branch of artificial intelligence and is an interdisciplinary subject of computer science and linguistics, and is also commonly known as computational linguistics. Natural language processing includes syntax analysis, semantic analysis, discourse understanding, etc. Natural language processing is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information intent recognition, information extraction and filtering, text classification and clustering, public opinion analysis and opinion mining, etc. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research related to language processing, and language computing related linguistic research.

[0057] Information extraction: a text processing technology that extracts specified types of entities, relationships, events, etc. from natural language text and forms structured data output. Information extraction is a technology that extracts specific information from text data. Text data is composed of specific units, such as sentences, paragraphs, and chapters. Text information is composed of specific units such as words, phrases, sentences, paragraphs, or combinations of these specific units. Extracting noun phrases, names, and places from text data is text information extraction. Of course, the information extracted by text information extraction technology can be various types of information.

[0058] Heterogeneous robots refer to robots with different characteristics (such as different degrees of freedom, different end effector types) in a collaborative system. In practical applications, multiple heterogeneous robots are controlled to collaborate to complete complex target tasks.

[0059] In the related art, sub-tasks to be completed are manually set for each heterogeneous robot in advance to achieve the purpose of completing a complex target task. However, since the working environment of the heterogeneous robot is usually changing at any time, the pre-set sub-tasks cannot flexibly cope with the dynamically changing environmental factors, thereby leading to low efficiency of the heterogeneous robot in completing the intended target task.

[0060] Based on this, the embodiment of the present application provides a control method and device of a heterogeneous robot, an electronic device and a storage medium, aiming to improve the efficiency of multiple heterogeneous robots in completing a target task.

[0061] It should be noted that in the embodiments of the present application, when it is necessary to obtain user's basic information or information related to user characteristics such as user identity, the user's permission or consent will be obtained first, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. In addition, when the embodiments of the present application need to obtain sensitive personal information of the user, the separate permission or separate consent of the user will be obtained first, and after obtaining the separate permission or separate consent of the user, when multiple heterogeneous robots are used to move target objects in a target area managed by a certain user, the environmental perception data of the target area needs to be obtained to generate task prompt words, in this case, the consent of the relevant user will be obtained first, otherwise the environmental perception data used in the embodiments of the present application cannot be obtained. Other related data used in the embodiments of the present application are also obtained legally and with the consent of the relevant user, which will not be repeated here.

[0062] The control method and device of a heterogeneous robot, an electronic device and a storage medium provided by the embodiments of the present application are specifically described as follows. First, the application scenario of the control device of the heterogeneous robot proposed in the embodiments of the present application is described, which can also be referred to as "control device" for convenience of description, as shown in Figure 1 Figure 1 is an optional application scenario diagram of the control device of the heterogeneous robot provided by the embodiments of the present application, in which Figure 1 ​In the specific implementation, the plurality of heterogeneous robots (including the heterogeneous robot a, the heterogeneous robot b, and the heterogeneous robot c) work together in a target area, and when the heterogeneous robot a, the heterogeneous robot b, and the heterogeneous robot c need to be combined to complete a target task, the heterogeneous robot b is taken as an example (at this time, the heterogeneous robot b is referred to as a target heterogeneous robot): total task data and environmental perception data around the target heterogeneous robot are acquired, and a task prompt word is generated according to the total task data and the environmental perception data; in the current round of dialogue, historical dialogue text between the plurality of heterogeneous robots is acquired (for example, before this, the heterogeneous robot a and the heterogeneous robot c have a dialogue, and the historical dialogue text is updated according to dialogue text generated by the heterogeneous robot a and the heterogeneous robot c), and a response is made based on the historical dialogue text and the task prompt word, new dialogue text is generated, and the new dialogue text is updated to the historical dialogue text; then, the heterogeneous robot b extracts subtask data from the updated historical dialogue text, and determines a corresponding subtarget task according to the subtask data. The same is true for the heterogeneous robot a and the heterogeneous robot c, which will not be described here again. That is, the plurality of heterogeneous robots can determine corresponding subtarget tasks through dialogue, and the plurality of heterogeneous robots can complete the total target task by completing the subtarget tasks, thereby improving the efficiency of the plurality of heterogeneous robots in completing the target task.

[0063] After understanding the application scenario of the control device proposed in the application, the control method of the heterogeneous robot proposed in the embodiment of the application (for the convenience of description, which can also be referred to as a “control method” hereinafter) is described.

[0064] In the embodiment of the application, the control device is described from the dimension, which can be integrated in a computer device, such as a server. As shown in Figure 2 Figure 2 is an optional flowchart of the control method of the heterogeneous robot provided in the embodiment of the application, Figure 2 The method in the embodiment of the application can include but is not limited to the following steps 101 to 103. When the control device executes the control method, the specific process is as follows. It needs to be first pointed out that the order of the steps 101 to 103 in the embodiment is not specifically limited, and the order of the steps can be adjusted or some steps can be reduced or added according to actual needs. Figure 2

[0065] Step 101: Acquire total task data and environmental perception data around a target heterogeneous robot, and generate a task prompt word according to the total task data and the environmental perception data.

[0066] The step 101 is described in detail below.

[0067] ​​Among them, the heterogeneous robots mainly refer to robots that differ in function, performance, design or application scenarios, which may be caused by the structure, material, control system, perception ability, task execution ability and other aspects of the robots. For example, as shown in Figure 1 FIG. 1, the heterogeneous robot a, the heterogeneous robot b and the heterogeneous robot c are all heterogeneous robots, but their task execution abilities are different, for example, the heterogeneous robot a is used for detecting target objects, the heterogeneous robot b is used for grabbing target objects, and the heterogeneous robot c is used for carrying target objects. A plurality of heterogeneous robots participating in the execution of the total target task form a collaborative system.

[0068] Further, each heterogeneous robot is equipped with a large language model (LLM), and the performance of the corresponding LLM is different according to the different abilities of each heterogeneous robot. For example, the LLM carried by the heterogeneous robot with simple grabbing ability is simpler than the LLM of other heterogeneous robots, so that the simple LLM can quickly respond and process simple instructions, while the complex LLM can process more complex language inputs and generate more accurate responses, saving total computing resources while improving overall work efficiency.

[0069] Among them, the LLM refers to a kind of artificial intelligence model that is trained with a large amount of data and aims to understand input content and generate natural language text through natural language processing (NLP) and information extraction. For example, the LLM can be:

[0070] (1) Generative Pre-trained Transformer (GPT): GPT series models have hundreds of billions of parameters, which can realize a wide range of natural language understanding and generation tasks;

[0071] (2) Bidirectional Encoder Representations from Transformers (BERT): BERT significantly improves the accuracy of natural language understanding by considering context information simultaneously; among them, the variants of BERT such as Bidirectional Encoder Representations from Transformers (RoBERTa), A Lite version of BERT (ALBERT) and the like;

[0072] (3) Neural network architecture model based on self-attention mechanism (Transformer): a model using attention mechanism to process sequence data, and the Transformer greatly improves the training efficiency of the model by processing all elements in the sequence in parallel;

[0073] It should be noted that in addition to the above several optional LLM types, the LLM in the embodiments of the present application can also be specifically selected according to actual conditions, and the embodiments of the present application do not limit this.

[0074] Among them, the total task data is used to represent the related data of the total target task that needs to be completed by the plurality of heterogeneous robots. For example, in a certain target work area of industrial manufacturing, the plurality of heterogeneous robots include a carrying robot, an assembly robot, and a detection robot, and the plurality of heterogeneous robots need to cooperate to complete the finished product detection at the fixed point position as the total target task, wherein the total task data corresponding to the total target task includes but is not limited to the map of the target work area, the fixed point position, the time limit for cooperation completion, etc.

[0075] Further, each heterogeneous robot will obtain the same total task data, and the total task data is used to indicate the dialogue direction of each heterogeneous robot in the subsequent dialogue process, and to guide the total completion goal of each heterogeneous robot, that is, the sub-target task determined by each heterogeneous robot should be for the purpose of completing the total target task.

[0076] Among them, the environment perception data can include but is not limited to the position information of the target heterogeneous robot itself, the related data of the surrounding environment in the target work area that can be perceived by the target heterogeneous robot, etc. For example, when the target heterogeneous robot includes a camera, the perception data can be image data representing the size and shape of each object in the target work area; when the target heterogeneous robot includes a temperature sensor, the perception data includes a temperature value for representing the temperature of the target work area; when the target heterogeneous robot includes a laser radar, the perception data includes coordinate data for representing the accurate position of each object in the target work area. Alternatively, the environment perception data can be specifically set according to actual conditions and the different capabilities of each heterogeneous robot, and the embodiments of the present application do not limit this.

[0077] Further, each heterogeneous robot can obtain different environment perception data according to the different capabilities of each heterogeneous robot, based on which each heterogeneous robot can generate corresponding task prompt words according to the total task data and the individualized environment perception data, so that each heterogeneous robot can generate dialogue text for discussion according to the task prompt words.

[0078] The task prompt word includes first key information in the total task data, second key information in the environment perception data, and a dialogue guiding instruction. Further, the first key information is obtained by the LLM model understanding the obtained total task data and extracting key elements therefrom; and the second key information is obtained by the LLM model understanding the environment perception data obtained by the heterogeneous robot and extracting key elements therefrom. For example, the first key information is generated by the following steps: ① obtaining total task data: the total task data obtained by a certain heterogeneous robot is "please move all the goods marked as 'urgent' from area A to area B, and count during the moving process"; ② total task data understanding: the LLM model carried by the heterogeneous robot processes the total task data by natural language processing, and identifies the key elements of the task, such as action (moving, counting), object (goods marked as 'urgent'), starting position (area A), target position (area B), etc.; ③ key word extraction: based on the understanding of the task data, the LLM model further extracts key words, in this example, the key words extracted by the LLM include "moving", "counting", "urgent", "goods", "area A", "area B", etc., and then the first key information is generated based on the extracted key words.

[0079] It should be noted that in addition to the first key information and the second key information, the key information in the task prompt word can also add other key information according to actual conditions, and the key information can also include pre-set constraint condition information such as execution time of the total target task, and the specific content of the key information is not limited by the embodiments of the present application.

[0080] Further, the dialogue guiding instruction is used to guide the target heterogeneous robot how to respond to other heterogeneous robots and correctly format the output results. The dialogue guiding instruction is usually added after the key information, and is intended to clearly specify the specific details or additional operation contents of the generated dialogue text. For example, the dialogue guiding instruction can be "do you have any other urgent goods that need to be handled first?".

[0081] It can be understood that the task prompt word can accurately convey the content including but not limited to task background, target work area environment data, etc. to the target heterogeneous robot, thereby guiding and promoting the dialogue interaction and decision making among multiple heterogeneous robots, thereby improving the accuracy of the generated dialogue text.

[0082] Step 102: In the current round of dialogue, obtain the historical dialogue text between multiple heterogeneous robots, and respond based on the historical dialogue text and task prompts to generate new dialogue text. Update the historical dialogue text with the new dialogue text until all heterogeneous robots have finished speaking, thus completing the current round of dialogue. When the target heterogeneous robot is the first to speak among multiple heterogeneous robots, the historical dialogue text is empty.

[0083] Step 102 is described in detail below.

[0084] In some embodiments, after generating corresponding task prompts, each heterogeneous robot will conduct multiple rounds of dialogue based on the task prompts. When it is the turn of the current heterogeneous robot to speak, it will first acquire the historical dialogue text generated between multiple heterogeneous robots. The historical dialogue text can be generated during the current round of dialogue, or it can be all historical dialogue text generated during all previous rounds of dialogue. This allows it to generate new dialogue text based on the historical dialogue text and the task prompts, thereby helping to improve the accuracy of determining the sub-target task corresponding to the current heterogeneous robot.

[0085] Each heterogeneous robot can store its own generated dialogue text and the dialogue text received from other heterogeneous robots, forming a historical dialogue text. This historical dialogue text helps the heterogeneous robot better understand the background and progress of the overall objective task when it speaks next, thus enabling it to respond more accurately in accordance with the planned actions of other heterogeneous robots.

[0086] like Figure 3 As shown, Figure 3 This is an optional schematic diagram of multiple heterogeneous robots discussing a heterogeneous robot control method provided in this application embodiment. In each round of dialogue, all heterogeneous robots participating in the overall objective task need to speak one by one. When a heterogeneous robot speaks, it first determines a new dialogue text based on the acquired historical dialogue text and its own generated task prompts. Then, while storing the dialogue text, the speaking heterogeneous robot sends the dialogue text to other heterogeneous robots so that the other heterogeneous robots can store the dialogue text to update the historical dialogue text. In this way, heterogeneous robots that have not yet spoken can make more accurate statements based on the updated historical dialogue text.

[0087] Furthermore, the speaking order of heterogeneous robots can be categorized into the following cases:

[0088] (1) First speaking: if the target heterogeneous robot is the first speaking in the multiple heterogeneous robots, the historical dialogue text is empty, the target heterogeneous robot generates dialogue text based on the task prompt word and sends it to other heterogeneous robots, until each heterogeneous robot has spoken once, and the dialogue of the current round is completed;

[0089] (2) Non-first speaking: if the target heterogeneous robot is not the first speaking in the multiple heterogeneous robots, the target heterogeneous robot will first obtain the historical dialogue text updated by other heterogeneous robots in the current round of dialogue; then, the target heterogeneous robot generates new dialogue text based on the task prompt word and the obtained historical dialogue text; and at the same time of updating the historical dialogue text based on the new dialogue text, the dialogue text is sent to other heterogeneous robots.

[0090] In some embodiments, as shown in Figure 4 , Figure 4 is Figure 2 an implementation flowchart of step 102 in

[0091] Step 201, updating the task prompt word according to the historical dialogue text to obtain the updated task prompt word.

[0092] Step 202, generating new dialogue text based on the updated task prompt word.

[0093] The steps 201 to 202 are described in detail below.

[0094] In some embodiments, after obtaining the historical dialogue text, the target heterogeneous robot first updates the task prompt word according to the historical dialogue text. The task prompt word can also include historical dialogue key information, specifically, the historical dialogue key information is generated by the LLM after understanding the historical dialogue text. In this way, the updated task prompt word can include the first key information, the second key information, the historical dialogue key information and the dialogue guiding instruction.

[0095] Further, the target heterogeneous robot can respond to the historical dialogue text according to the updated task prompt word to generate new dialogue text. The new dialogue text can include a summary of the dialogue text generated by the previously speaking heterogeneous robot and the response content generated by itself.

[0096] Exemplarily, the historical dialogue text is: "Robot 1: 'I have the ability to grasp objects below 1000 kg, I am currently in target area A and can reach target area B within one minute'", then the new dialogue text generated by the target heterogeneous robot (robot 2) can be: "I am in target area B, I have the ability to carry objects below 1000 kg to the designated location, and can handle multiple goods at the same time. According to the instructions of robot 1, robot 2 suggests that robot 1 first arrive at target area B, grasp and place all the goods that can be carried, and then robot 2 can assist in transporting these goods to the final destination, and at the same time, I will also check target area B for other goods or obstacles that need special handling".

[0097] The speaking order of each heterogeneous robot can be pre-set, so that the dialogue of the current round ends after each heterogeneous robot completes the speaking of the current round according to the predetermined order; or any heterogeneous robot can be specified to speak first, then other heterogeneous robots receive the dialogue text of the first heterogeneous robot, understand the dialogue text through the LLM, and determine whether they can have a task cooperative processing relationship with the first heterogeneous robot, if so, speak; if multiple heterogeneous robots can have a task cooperative processing relationship with the first speaking heterogeneous robot, they can speak in turn according to the distance from the first speaking heterogeneous robot. Of course, the speaking order of each heterogeneous robot can also be set according to actual conditions, and the embodiments of the present application do not limit this.

[0098] It can be understood that each heterogeneous robot can construct corresponding response content based on the dialogue text of the previously speaking heterogeneous robot, so that the entire discussion process is more natural and coherent, avoiding information isolation and repetition, so that the understanding ability of multiple heterogeneous robots for the total target task to be processed is improved by simulating the alternating speaking manner in real dialogue, and the efficiency of multiple heterogeneous robots in cooperating to complete the target task is improved.

[0099] In step 103, after the dialogue of at least one round ends, the sub-task data corresponding to the target heterogeneous robot is extracted based on the historical dialogue text updated by the last speaking heterogeneous robot, and the corresponding sub-target task is executed based on the sub-task data.

[0100] The step 103 is described in detail below.

[0101] In some embodiments, each heterogeneous robot participates in at least one round of dialogue, that is, each heterogeneous robot speaks at least once. Since the next speaking heterogeneous robot will summarize based on the historical dialogue text, as the number of speaking heterogeneous robots increases, the content of the dialogue will become more in-depth, and the sub-target tasks that each heterogeneous robot needs to complete will become clearer.

[0102] Furthermore, if the LLM model carried by the heterogeneous robot is powerful enough, the last heterogeneous robot to speak determines the sub-target tasks that each heterogeneous robot needs to perform; or, the last heterogeneous robot to speak does not perform task summary, and each heterogeneous robot extracts its corresponding sub-task data from the last updated historical dialogue text, and determines its own sub-target tasks to perform based on the sub-task data.

[0103] Subtask data refers to specific task information extracted from historical dialogue text for the target heterogeneous robot. This data typically includes key elements such as specific task requirements, execution conditions, required resources, and expected results, and serves as the basis for guiding the target heterogeneous robot to execute sub-target tasks.

[0104] Among them, the sub-target task refers to the task content determined by the target heterogeneous robot based on the sub-target data. For example, "driving to target area B and grabbing all items on the ground in target area B" is a sub-target task. Each heterogeneous robot corresponds to one or more sub-target tasks, and the overall target task is achieved after the execution of all sub-target tasks.

[0105] Furthermore, after the sub-target task is determined, the target heterogeneous robot can transmit relevant data to its own execution components, such as a multi-arm motion planner based on Rapidly-exploring Random Trees (RRT), to execute the sub-target task.

[0106] It is understood that the multiple heterogeneous robots in this application embodiment discuss the feasibility of the overall target task by speaking one by one. This approach allows each heterogeneous robot to fully consider the previous discussions and existing decisions when speaking, thereby reducing information silos, efficiently promoting the continuous evolution of information flow, and improving the efficiency of multiple heterogeneous robots to collaboratively and autonomously complete the target task.

[0107] In some embodiments, such as Figure 5 As shown, Figure 5 yes Figure 2 A flowchart following step 103 in the diagram includes the following steps 301 to 306 after extracting the subtask data corresponding to the target heterogeneous robot:

[0108] In step 301, based on the sub-task data, corresponding task text information, task constraint information and task waypoint information are obtained.

[0109] In step 302, if the task text information includes a preset keyword, it is determined that the task text information is verified.

[0110] In step 303, if the task constraint information is consistent with a preset constraint action, it is determined that the task constraint information is verified.

[0111] In step 304, if the task waypoint information is consistent with a preset inverse kinematics data, it is determined that the task waypoint information is verified.

[0112] In step 305, when the task text information, the task constraint information and the task waypoint information are all verified, the sub-task data is verified.

[0113] In step 306, if the sub-task data is verified, a corresponding sub-target task is executed based on the sub-task data.

[0114] The steps 301 to 306 are described in detail below.

[0115] In some embodiments, the target heterogeneous robot verifies the sub-task data before executing the sub-target task. The pre-verification mechanism of the embodiments of the present application ensures the rationality and feasibility of the target heterogeneous robot plan, thereby improving the practicality and robustness of the collaboration system.

[0116] Further, based on the extracted sub-task data, corresponding task text information is determined. The task text information is obtained by analyzing the sub-task data. The task text information includes format information and plan information. Correspondingly, the preset keyword includes a format keyword and a plan keyword. The format keyword defines how the sub-task data should be formatted and presented to ensure that the target heterogeneous robot can accurately analyze and execute the corresponding sub-target task. Specifically, the format keyword can include task execution order, task execution priority, specific delimiters, etc. The plan keyword defines the task framework that the target heterogeneous robot should execute. For example, for a heterogeneous robot for grabbing, its corresponding sub-target task should be roughly to grab x items at x position (where x represents unknown). If the target heterogeneous robot obtains sub-task data without keywords such as "target area" and "target item", it is considered that the task text information is not verified. Otherwise, it is verified.

[0117] Further, based on the extracted sub-task data, corresponding task constraint information is determined. The task constraint information includes but is not limited to time limit (such as the sub-target task must be completed within a certain time period), space limit (such as the sub-target task can only be executed in the specified target area), resource limit (such as the maximum amount of computing resources required to execute the sub-target task), etc. For example, the sub-task data corresponding to the target heterogeneous robot a is "drive the target heterogeneous robot to the target area B to grab the target object on the ground", then the target heterogeneous robot obtains the corresponding preset constraint action of "target area B", such as the constraint action can be "target area B does not allow target heterogeneous robot a and target heterogeneous robot b to enter", therefore in this example, the task constraint information verification fails.

[0118] Further, based on the extracted sub-task data, corresponding task waypoint information is determined. The task waypoint information is used to represent the target position point that the target heterogeneous robot needs to pass through in the process of executing the sub-target task, and the target action that needs to be executed at the target position point. The inverse kinematics refers to the process of setting the parameters of the movable joint object to achieve the required posture, and the inverse kinematics data is the data related to the inverse kinematics. The inverse kinematics parameters can include but are not limited to joint angle, movement speed, etc. The target heterogeneous robot compares the task waypoint information determined from the sub-task data with the preset inverse kinematics parameters to verify whether the target heterogeneous robot will appear abnormal situations such as collision, overload, etc. when executing the corresponding sub-target task, and further judges whether the verification of the task waypoint information passes.

[0119] It should be noted that the keywords, constraint actions and inverse kinematics data can be specifically set according to actual conditions, and the embodiments of the present application do not limit this.

[0120] Further, when the task text information, the task constraint information and the task waypoint information all pass the verification, the sub-task data passes the verification. Then, the target heterogeneous robot can determine and execute the corresponding sub-target task based on the sub-task data.

[0121] In some embodiments, if the current round of conversation has not ended, but the target heterogeneous robot has been able to determine its sub-task data from the historical conversation text, the target heterogeneous robot can verify the extracted sub-task data in the process of discussion by other heterogeneous robots, so as to speed up the efficiency of the collaboration system in processing the total target task.

[0122] Or, after at least one round of dialogue ends and each heterogeneous robot determines the respective corresponding sub-task data, each heterogeneous robot shares the corresponding sub-task data to other heterogeneous robots, and the target heterogeneous robot verifies the corresponding sub-task data of itself while taking the corresponding sub-task data of other heterogeneous robots as one of the considerations to ensure that each heterogeneous robot will not have abnormal situations such as collision when performing the corresponding sub-target task.

[0123] In some embodiments, as shown in Figure 6 Figure 6 is Figure 5 an implementation flowchart after step 301 in

[0124] Step 401, if the sub-task data fails to pass the verification, generate failure evaluation information.

[0125] Step 402, update the task prompt word according to the failure evaluation information to obtain the updated task prompt word, and based on the updated task prompt word, perform the next round of dialogue.

[0126] Step 403, after the next round of dialogue ends, extract the target heterogeneous robot's corresponding sub-task data based on the historical dialogue text updated by the last speaking heterogeneous robot, and execute the corresponding sub-target task based on the sub-task data.

[0127] The steps 401 to 403 are described in detail as follows.

[0128] The failure evaluation information is used to indicate the reason why the sub-task data fails to pass the verification. When the sub-task data fails to pass the verification, the target heterogeneous robot can generate the failure evaluation information and update the task prompt word based on the failure evaluation information. Moreover, the failure evaluation information can include reasons such as format error, logic error, insufficient authority, etc.

[0129] Further, since the total target task can be successfully executed only when all sub-task data passes the verification, when the sub-task data corresponding to any heterogeneous robot fails to pass the verification, the plurality of heterogeneous robots will perform the next round of dialogue to re-determine the sub-task data of each heterogeneous robot. During the next round of dialogue, the target heterogeneous robot will generate new dialogue text based on the updated task prompt word to determine the corresponding sub-target task.

[0130] ​Further, each heterogeneous robot can obtain the historical dialogue text generated in the previous round of dialogue in the process of the next round of dialogue, to generate new dialogue text according to the historical dialogue text and the dialogue text updated based on the failure evaluation information. It can be understood that the historical dialogue text generated in multiple rounds can help to enrich the understanding of the total target task by the heterogeneous robots, and thus improve the efficiency of the target heterogeneous robot in determining the sub-target task that can be correctly executed. The specific implementation of step 403 is similar to that of step 103, and will not be described here.

[0131] In some embodiments, as shown in Figure 7 Figure 7 is Figure 6 an implementation flowchart after step 403 in, after executing the corresponding sub-target task based on the sub-task data, the following steps 501 to 503 are further included:

[0132] Step 501: If the sub-target task fails to be executed, the environmental perception data around the target heterogeneous robot is re-acquired to obtain updated environmental perception data.

[0133] Step 502: The task prompt word is updated according to the total task data and the updated environmental perception data to obtain an updated task prompt word, and the next dialogue round is performed based on the updated task prompt word.

[0134] Step 503: After the end of the next round of dialogue, the sub-task data corresponding to the target heterogeneous robot is extracted based on the historical dialogue text updated by the last speaking heterogeneous robot, and the corresponding sub-target task is executed based on the sub-task data.

[0135] The steps 501 to 503 are described in detail as follows.

[0136] In some embodiments, even if the sub-task data corresponding to each heterogeneous robot is verified to be correct, the phenomenon of failure to execute the sub-target task may still occur. The reasons for this phenomenon include but are not limited to heterogeneous robot hardware failure, environmental change (such as light change, temperature fluctuation, electromagnetic interference, etc.), failure of cooperation between heterogeneous robots (for example, the target object grabbed by the heterogeneous robot b is not accurately placed on the heterogeneous robot c for carrying).

[0137] Further, the sub-target task failure indicates that the target heterogeneous robot has moved or the target object in the target area has moved, i.e., the environment of the target area has changed. In this case, the target heterogeneous robot needs to reacquire the environmental perception data around it and then update the task prompt word; then, the target heterogeneous robot will perform the next round of on-site dialogue based on the updated task prompt word to re-determine the sub-target task to be executed. The specific implementation of step 503 is similar to that of step 103, and thus is not described here.

[0138] Further, after the sub-target task execution fails, the target heterogeneous robot can autonomously analyze the cause of the task failure and update the task prompt word based thereon; and in the next round of dialogue, the target heterogeneous robot can also acquire the historical dialogue text generated in the previous round or rounds to further help the target heterogeneous robot understand the cause of the sub-target task failure and then more accurately re-determine the corresponding sub-target task.

[0139] It can be understood that after the target heterogeneous robot fails to execute the sub-target task, it can re-determine the corresponding sub-target task by autonomously updating the environmental perception data. Since the working environment of each heterogeneous robot is not constant during collaborative work, for example, the heterogeneous robot b equipped with a camera needs to go to the target area B to sequentially grasp the target object b. When the heterogeneous robot b arrives at the target area B, it finds that the target object b has been taken away by a person. At this time, the heterogeneous robot b needs to reacquire the environmental perception data to re-determine the corresponding sub-target task. Alternatively, in the case of sufficient computing resources, the heterogeneous robot b can acquire the environmental perception data around it in real time. When the heterogeneous robot b finds that the target object b is not in the specified position, it no longer continues to execute the sub-target task but stops the execution of the current sub-target task and re-determines the corresponding sub-target task.

[0140] In the related art, each heterogeneous robot usually executes a predetermined sub-target task. Since a single heterogeneous robot does not have the ability to coordinate other heterogeneous robots, when the sub-target task execution fails, relevant personnel need to manually reconfigure the sub-target task. In comparison, in the embodiments of the present application, after the target heterogeneous robot fails to execute the sub-target task, it can generate new dialogue text and discuss with other heterogeneous robots to re-determine the executable sub-target task, instead of stopping or only performing error reporting, thereby improving the efficiency of the multiple heterogeneous robots in processing the total target task.

[0141] In some embodiments, as shown in FIG. 5, Figure 8 Figure 8 is Figure 2 ​A flowchart following step 102 in the previous section, when the heterogeneous robot includes a leader robot, further includes steps 601 to 602 after updating the new dialogue text to the historical dialogue text:

[0142] Step 601: After at least one round of dialogue, the leader robot acts as the last heterogeneous robot to speak and updates the historical dialogue text.

[0143] Step 602: Receive the sub-task data corresponding to the target heterogeneous robot determined by the leader robot, and execute the corresponding sub-target task based on the sub-task data.

[0144] Steps 601 to 602 are described in detail below.

[0145] Among them, the LLM (Limited Learning Model) equipped in the leader robot outperforms those equipped in other heterogeneous robots, thus enabling the leader robot to possess leadership or coordination capabilities within the collaborative system. When a leader robot exists among multiple heterogeneous robots, it updates the historical dialogue text as the last heterogeneous robot to speak in each round of dialogue. This allows the leader robot to gain a holistic understanding of the dialogue texts generated by other heterogeneous robots, thereby better breaking down the overall objective task and determining the sub-objective tasks corresponding to each heterogeneous robot.

[0146] Furthermore, the leader robot can directly determine the sub-task data corresponding to each heterogeneous robot from the perspective of the last speaker, and send multiple sub-task data to each heterogeneous robot; after receiving the corresponding sub-task data, the target heterogeneous robot will execute the corresponding sub-target task based on the sub-task data.

[0147] Furthermore, the leader robot may be one that participates in the execution of sub-target tasks; alternatively, the leader robot may not participate in the execution of sub-target tasks, but may only receive and analyze the dialogue text generated by each heterogeneous robot. The relationship between the leader robot and other heterogeneous robots can be specifically set according to the actual situation, and the embodiments of this application do not impose any limitations on this.

[0148] In some embodiments, such as Figure 9 As shown, Figure 9 yes Figure 2 The flowchart following step 101 includes the following steps 701 to 702 after the current round of dialogue is completed:

[0149] Step 701: After the preset free discussion time ends, obtain the historical dialogue text updated by the last heterogeneous robot to speak, wherein the target heterogeneous robot spoke zero times or at least once during the free discussion time.

[0150] Step 702: Based on the historical dialogue text, extract the sub-task data corresponding to the target heterogeneous robot, and execute the corresponding sub-target task based on the sub-task data.

[0151] Steps 701 to 702 are described in detail below.

[0152] The free discussion time is a designated period designed to facilitate information exchange, opinion sharing, and problem-solving among heterogeneous robots. During this time, heterogeneous robots can speak freely to share and discuss information through dialogue text, thereby better defining the sub-target tasks of the target heterogeneous robot.

[0153] Furthermore, during the free discussion period, the target heterogeneous robot may remain silent, speak once, or speak multiple times. When silent, it can better refine the extracted sub-task data by understanding the dialogue text sent by other heterogeneous robots. When speaking, it can further contribute its own views, information, and questions to other heterogeneous robots through dialogue text, thereby promoting the in-depth discussion and the determination of sub-target tasks.

[0154] It should be noted that the specific duration of the free discussion time can be set according to the actual situation. Furthermore, the sub-target tasks corresponding to each heterogeneous robot can be determined by the updated historical dialogue text of the last heterogeneous robot to speak after the free discussion time ends; alternatively, after the free discussion time ends, another round of dialogue can be conducted, allowing each heterogeneous robot to speak again to supplement or organize important information or details, enabling the target heterogeneous robot to extract more accurate sub-task data, and thus better determine the sub-target tasks it needs to execute based on this sub-task data.

[0155] like Figure 10 As shown, Figure 10 This is an optional flowchart of a control device for a heterogeneous robot provided in an embodiment of this application. The control device for the heterogeneous robot includes the following steps 801 to 803:

[0156] The acquisition module 801 is used to acquire total task data and environmental perception data around the target heterogeneous robot, and generate task prompt words based on the total task data and environmental perception data.

[0157] The current round dialogue module 802 is used to obtain the historical dialogue text between multiple heterogeneous robots in the current round of dialogue, and respond based on the historical dialogue text and task prompts to generate new dialogue text. The new dialogue text is then updated to the historical dialogue text until all heterogeneous robots have finished speaking, thus completing the current round of dialogue. When the target heterogeneous robot is the first to speak among multiple heterogeneous robots, the historical dialogue text is empty.

[0158] The target execution module 803 is configured to extract sub-task data corresponding to the target heterogeneous robot based on the historical dialogue text updated by the last speaking heterogeneous robot after the dialogue of at least one round ends, and execute a corresponding sub-target task based on the sub-task data.

[0159] The control method, device, electronic device, and storage medium provided in the present application obtain total task data and environmental perception data around the target heterogeneous robot, generate a task prompt word based on the total task data and the environmental perception data, obtain historical dialogue text among a plurality of heterogeneous robots in a current round of dialogue, and generate new dialogue text by responding based on the historical dialogue text and the task prompt word. The new dialogue text is updated to the historical dialogue text, and the dialogue of the current round is completed when all the heterogeneous robots finish speaking. When the target heterogeneous robot is the first to speak among the plurality of heterogeneous robots, the historical dialogue text is empty. After the dialogue of at least one round ends, sub-task data corresponding to the target heterogeneous robot is extracted based on the historical dialogue text updated by the last speaking heterogeneous robot, and a corresponding sub-target task is executed based on the sub-task data. In the present application, the plurality of heterogeneous robots discuss the executability of the total target task corresponding to the total task data by speaking one by one, which enables each heterogeneous robot to fully consider the previous discussion and existing decisions when speaking. In this way, information silos can be reduced, the continuous evolution of the information flow can be efficiently promoted, and the efficiency of the plurality of heterogeneous robots in cooperatively and autonomously completing the target task can be improved.

[0160] The specific implementation of the control device is basically the same as that of the above-mentioned specific embodiments of the control method, and thus will not be described herein again.

[0161] The present application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor implements the above-mentioned control method when executing the computer program. The electronic device can be any intelligent terminal, such as a tablet computer or a vehicle-mounted computer.

[0162] As shown in Figure 11 , the hardware structure of the electronic device provided in the present application is shown in FIG. 9. Figure 11 The electronic device includes:

[0163] The processor 901 can be implemented in the form of a general-purpose CPU (Central Processing Unit, central processor), a microprocessor, an ASIC (Application Specific Integrated Circuit, application-specific integrated circuit), or one or more integrated circuits, and is used to execute related programs to implement the technical solutions provided in the present application.

[0164] The memory 902 can be implemented in the form of a Read Only Memory (ROM), a static storage device, a dynamic storage device, or a Random Access Memory (RAM), etc. The memory 902 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 902 and are called and executed by the processor 901 to implement the control method of the embodiments of the present application.

[0165] The input / output interface 903 is configured to realize information input and output.

[0166] The communication interface 904 is configured to realize the communication interaction between the device and other devices. The communication can be realized by a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0167] The bus 905 is configured to transmit information between various components (for example, the processor 901, the memory 902, the input / output interface 903, and the communication interface 904) of the device.

[0168] The processor 901, the memory 902, the input / output interface 903, and the communication interface 904 are connected to each other through the bus 905 to realize the communication connection between the device.

[0169] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the above control method.

[0170] The memory is a non-transitory computer readable storage medium, which can be used to store a non-transitory software program and a non-transitory computer executable program. In addition, the memory can include a high-speed random access memory and can also include a non-transitory memory, for example, at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and the remote memory can be connected to the processor through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0171] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0172] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation to the embodiments of the present application, and can include more or fewer steps than the figures, or combine certain steps, or different steps.

[0173] The apparatus embodiments described above are merely illustrative, and units described as separate components can or can not be physically separate, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments.

[0174] Those skilled in the art can understand that all or some of the steps in the above disclosed method, the function modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0175] The terms "first", "second", "third", "fourth" and the like in the specification of the present application and the above-described drawings, if any, are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in other than the order illustrated or described herein. In addition, the terms "comprise" and "have" and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0176] It should be understood that in the present application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the association between the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can mean a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0177] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented by other manners. For example, the apparatus embodiments described above are merely illustrative, for example, the division of the above units is merely a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or components shown or discussed can be indirect coupling or communication connection through some interfaces, apparatuses or units, and can be electrical, mechanical or other forms.

[0178] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they can be located in one place or distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0179] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0180] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part of the prior art that makes a contribution or the whole or part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method of each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0181] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, but this does not limit the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A control method of a heterogeneous robot, characterized by, The method is applied to any target heterogeneous robot in a plurality of heterogeneous robots, and the method comprises: obtaining total task data and environment perception data around the target heterogeneous robot, and generating a task prompt word according to the total task data and the environment perception data; in the current round of dialogue, obtaining historical dialogue text between a plurality of the heterogeneous robots, and generating new dialogue text based on the historical dialogue text and the task prompt word, updating the new dialogue text into the historical dialogue text, until all the heterogeneous robots complete the speech, completing the dialogue of the current round, wherein when the target heterogeneous robot is the first to speak among the plurality of the heterogeneous robots, the historical dialogue text is empty; after the dialogue of at least one round ends, based on the historical dialogue text updated by the last speaking heterogeneous robot, the subtask data corresponding to the target heterogeneous robot is extracted, and the corresponding subtarget task is executed based on the subtask data; after the subtask data corresponding to the target heterogeneous robot is extracted, the method further comprises: based on the subtask data, obtaining corresponding task text information, task constraint information and task waypoint information; if the task text information includes a preset keyword, it is determined that the task text information is verified; if the task constraint information is consistent with the preset constraint action, it is determined that the task constraint information is verified; if the task waypoint information is consistent with the preset inverse kinematics data, it is determined that the task waypoint information is verified; when the task text information, the task constraint information and the task waypoint information are all verified, the subtask data is verified; if the subtask data is verified, the corresponding subtarget task is executed based on the subtask data.

2. The method of claim 1, wherein, the response based on the historical dialogue text and the task prompt word to generate new dialogue text comprises: updating the task prompt word according to the historical dialogue text to obtain updated task prompt word; generating new dialogue text based on the updated task prompt word.

3. The method of claim 1, wherein, after the subtask data is obtained based on the subtask data, the corresponding task text information, task constraint information and task waypoint information are further included: if the subtask data is not verified, failure evaluation information is generated; updating the task prompt word according to the failure evaluation information to obtain updated task prompt word, and based on the updated task prompt word, the dialogue of the next round is carried out; after the dialogue of the next round ends, based on the historical dialogue text updated by the last speaking heterogeneous robot, the subtask data corresponding to the target heterogeneous robot is extracted, and the corresponding subtarget task is executed based on the subtask data.

4. The method of claim 1, wherein, after the corresponding subtarget task is executed based on the subtask data, the method further comprises: if the subtarget task fails, the environment perception data around the target heterogeneous robot is reacquired to obtain updated environment perception data; updating the task prompt word according to the total task data and the updated environment perception data, obtaining an updated task prompt word, and performing a next dialogue turn based on the updated task prompt word; after the end of the next dialogue turn, extracting sub-task data corresponding to the target heterogeneous robot based on the historical dialogue text updated by the last speaking heterogeneous robot, and performing a corresponding sub-target task based on the sub-task data.

5. The method of claim 1, wherein, When the heterogeneous robots include a leader robot, after the new dialogue text is updated into the historical dialogue text, the method further includes: after the end of at least one dialogue turn, the leader robot is the last speaking heterogeneous robot and updates the historical dialogue text; receiving the sub-task data corresponding to the target heterogeneous robot determined by the leader robot, and performing a corresponding sub-target task based on the sub-task data.

6. The method of claim 1, wherein, After the completion of the current dialogue turn, the method further includes: after the end of the preset free discussion time, obtaining the historical dialogue text updated by the last speaking heterogeneous robot, wherein the target heterogeneous robot speaks zero times or at least once during the free discussion time; based on the historical dialogue text, extracting sub-task data corresponding to the target heterogeneous robot, and performing a corresponding sub-target task based on the sub-task data.

7. A control device of a heterogeneous robot, characterized by comprising: The device includes: an acquisition module configured to acquire total task data and environment perception data around a target heterogeneous robot, and generate a task prompt word based on the total task data and the environment perception data; a current turn dialogue module configured to, in a current turn dialogue, acquire historical dialogue text between a plurality of heterogeneous robots, and respond based on the historical dialogue text and the task prompt word to generate a new dialogue text, update the new dialogue text into the historical dialogue text, and complete the current turn dialogue when all the heterogeneous robots complete speaking, wherein the historical dialogue text is empty when the target heterogeneous robot is the first speaking one among the plurality of heterogeneous robots; a target execution module configured to, after the end of at least one dialogue turn, extract sub-task data corresponding to the target heterogeneous robot based on the historical dialogue text updated by the last speaking heterogeneous robot, and perform a corresponding sub-target task based on the sub-task data; after the extraction of the sub-task data corresponding to the target heterogeneous robot, the device further includes: based on the sub-task data, acquiring corresponding task text information, task constraint information, and task waypoint information; if the task text information includes a preset keyword, determining that the task text information is verified; if the task constraint information is consistent with a preset constraint action, determining that the task constraint information is verified; if the task waypoint information is consistent with preset inverse kinematics data, determining that the task waypoint information is verified; and When the task text information, the task constraint information and the task waypoint information are all verified, the sub-task data is verified; If the sub-task data is verified, a corresponding sub-target task is executed based on the sub-task data.

8. An electronic device, comprising: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor executes the computer program to implement the control method of the heterogeneous robot in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 8. The computer program is executed by the processor to implement the control method of the heterogeneous robot in any one of claims 1 to 6. The computer program is executed by the processor to implement the control method of the heterogeneous robot in any one of claims 1 to 6.

Images