Binary problem decision-making method of heterogeneous robot system based on multi-modal large language model, terminal and readable storage medium

Abstract

The invention relates to the technical field of robots, and discloses a binary problem decision-making method of a heterogeneous robot system based on a multi-modal large language model, a terminal and a readable storage medium, and the method comprises the steps: obtaining environment feedback, carrying out the decision-making judgment of a binary problem decision-making scene based on the multi-modal large language model according to the environment feedback, and obtaining a binary problem decision-making result; obtaining a judgment result; performing opinion exchange with other robots based on a multi-modal large language model, and updating the judgment result to obtain a final judgment result; according to the method, judgment is carried out through the multi-mode large language model, different task scenes and current robot forms can be adapted according to task description and role description, opinions of different forms of robots in a heterogeneous multi-machine system are exchanged by adopting an exchange mechanism, and the system is more flexible to use. And under the condition that the universality is ensured, effective utilization of robots with different capabilities in a heterogeneous multi-machine system can still be realized.

Description

Technical Field

[0001] The present invention relates to the field of robotics technology, and in particular to a binary problem decision-making method, a terminal, and a readable storage medium for a heterogeneous robotic system based on a multimodal large language model. Background Art

[0002] Heterogeneous multi-robot systems, comprising robots with diverse capabilities, sensors, and physical properties, offer unique advantages in handling multifaceted tasks requiring diverse skills. However, due to inherent differences in robot design and functionality, the variability of robot hardware can significantly bias decision-making processes. For example, visual observations collected from different camera perspectives can lead to differential perceptions of the environment. This hardware variability, combined with environmental changes, poses significant challenges to traditional decision-making mechanisms, which traditionally rely on handcrafted scenario-specific rules. For example, aerial robots determine obstacle locations, while ground robots determine surface type, which in turn determines the ground robot's trajectory.

[0003] Since this type of method requires formulating corresponding scenario rules for specific scenarios and specific robot combinations, if the specific application scenario or the robot combination changes, for example, in the above scenario, there is no aerial robot, but there are two ground robots in different positions with different functions, then the original rules will not be effective. Therefore, this type of method has low versatility. Summary of the Invention

[0004] The purpose of the present invention is to provide a binary problem decision-making method, terminal and readable storage medium for a heterogeneous robot system based on a multimodal large language model, aiming to solve the problem that existing robot decision-making methods are not applicable to different heterogeneous systems and different scenarios.

[0005] The technical solutions adopted by the present invention to solve the technical problems are as follows:

[0006] The present invention provides a binary problem decision-making method for a heterogeneous robot system based on a multimodal large language model, the binary problem decision-making method for a heterogeneous robot system based on a multimodal large language model comprising:

[0007] Obtaining environmental feedback, and making a decision judgment on a binary problem decision scenario based on the environmental feedback based on a multimodal large language model to obtain a judgment result;

[0008] Exchanging opinions with other robots based on the multimodal large language model, updating the judgment result, and obtaining a final judgment result;

[0009] A group judgment is obtained according to the final judgment result, and the group judgment is executed.

[0010] Furthermore, the obtaining of environmental feedback and making judgments based on the environmental feedback based on the multimodal large language model specifically include:

[0011] Obtaining a task role and environmental feedback, and generating a judgment prompt word according to the task role and the environmental feedback;

[0012] The judgment prompt word is input into a multimodal large language model, and a result output by the multimodal large language model is obtained as the judgment result.

[0013] Furthermore, the exchange of opinions is performed in multiple rounds, and the updated judgment after the last round of exchange of opinions is used as the final judgment result.

[0014] Furthermore, each round of exchange of opinions includes:

[0015] Obtaining reference judgments of each other robot and evaluating each reference judgment based on the multimodal large language model;

[0016] Sending the evaluation to the corresponding other robots, and obtaining the reference evaluation transmitted by each of the other robots;

[0017] The judgment result is updated according to each of the reference evaluations based on a multimodal large language model.

[0018] Furthermore, the number of the multimodal large language models is multiple;

[0019] Obtaining the reference judgment of each other robot and evaluating each reference judgment based on the multimodal large language model specifically includes:

[0020] Acquire each of the reference judgments from the message pool, and generate an evaluation prompt word for each of the reference judgments according to the reference judgments, to obtain a plurality of evaluation prompt words;

[0021] Each of the evaluation prompt words is input into an independent multimodal large language model to obtain an evaluation of the reference judgment corresponding to the evaluation prompt word.

[0022] Furthermore, obtaining a group judgment based on the final judgment result specifically includes:

[0023] Obtain the final judgment results of all robots and the weights of all robots;

[0024] The group judgment is obtained by performing weighted processing based on each of the final judgment results and the weights.

[0025] Furthermore, all robots have the same weight.

[0026] In addition, to achieve the above objectives, the present invention further provides a binary problem decision-making system for a heterogeneous robot system based on a multimodal large language model. The decision-making system for a success-or-failure scenario of the heterogeneous robot system includes:

[0027] A judgment module obtains environmental feedback and makes a decision judgment on a binary problem decision scenario based on the environmental feedback based on a multimodal large language model to obtain a judgment result;

[0028] A communication module exchanges opinions with other robots based on a multimodal large language model, updates the judgment result, and obtains a final judgment result;

[0029] A decision module acquires a group judgment based on the final judgment result and executes the group judgment. Furthermore, to achieve the above-mentioned objectives, the present invention further provides a terminal comprising: a memory, a processor, and a program for making decisions on binary problems for a heterogeneous robot system based on a multimodal large language model, stored in the memory and executable on the processor. When executed by the processor, the program controls the terminal to implement the steps of the method for making decisions on binary problems for a heterogeneous robot system based on a multimodal large language model.

[0030] In addition, to achieve the above-mentioned purpose, the present invention also provides a readable storage medium, which stores a binary problem decision-making program for a heterogeneous robot system based on a multimodal large language model. When the binary problem decision-making program for a heterogeneous robot system based on a multimodal large language model is executed by a processor, the steps of the binary problem decision-making method for a heterogeneous robot system based on a multimodal large language model as described above are implemented.

[0031] The present invention adopts the above technical solution to achieve the following effects:

[0032] The present invention makes judgments through a multimodal large language model. The multimodal large model can adapt to different task scenarios and the current robot form according to the task description and role description, and has high versatility. In addition, a communication mechanism is used to exchange the opinions of robots of different forms in a heterogeneous multi-machine system to make group judgments. Because different robots fully evaluate the judgments of other robots from the perspective of their own capabilities when making group judgments, this method can ensure versatility while still achieving effective utilization of robots with different capabilities in a heterogeneous multi-machine system. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a flowchart of a binary problem decision-making method for a heterogeneous robot system based on a multimodal large language model in a preferred embodiment of the present invention;

[0034] Figure 2 It is a flow chart of the debate mechanism in a preferred embodiment of the present invention;

[0035] Figure 3 This is a schematic diagram of a test scenario in a preferred embodiment of the present invention;

[0036] Figure 4 It is a structural diagram of a binary problem decision-making system of a heterogeneous robot system based on a multimodal large language model in a preferred embodiment of the present invention;

[0037] Figure 5 Schematic diagram of the operating environment of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of the present invention more clear and distinct, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0039] Example 1

[0040] In recent years, advances in robotics and artificial intelligence have enabled robots to play key social roles and interact more closely with humans and other robots to perform common tasks. As this trend continues, it is foreseeable that heterogeneous robots will increasingly cooperate and / or compete with each other to complete tasks together.

[0041] The behavior of heterogeneous multi-robot systems under different tasks and conditions remains an open problem in the existing technology. Heterogeneous multi-robot systems, which include robots with different capabilities, sensors, and physical properties, have unique advantages in handling multifaceted tasks that require multiple skills. However, due to the inherent differences in robot design and functionality, these systems face significant challenges in coordination, communication, and decision-making. The collective behavior of heterogeneous multi-robot systems is particularly important when they need to make decisions on events that have a high impact on task execution. The variability of robot hardware can significantly bias the decision-making process. For example, visual observation data collected from different camera viewpoints can lead to differentiated judgments about the environment. This hardware variability, combined with environmental changes, poses a significant challenge to traditional decision-making mechanisms, which are traditionally solved by manually formulating scene-specific rules.

[0042] Specifically, collaborative decision-making in heterogeneous multi-robot systems refers to the process of selecting an action from a set of candidate actions that maximizes the mission objective while satisfying any potential constraints. Previous approaches for decision-making in heterogeneous multi-robot systems can be categorized as centralized, decentralized, or hybrid. Centralized heterogeneous multi-robot systems typically have a fixed leader, which may be a robot or a remote server, responsible for all decision-making. Decentralized heterogeneous multi-robot systems can be further subdivided into hierarchical and distributed approaches. The former allows for local centralized decision-making, while the latter places equal responsibility for decision-making on all robots. Previous approaches for task planning in heterogeneous multi-robot systems include algorithm-based methods, such as optimization-based methods, and learning-based methods. However, these methods are typically scenario-specific and require human expertise to manually formulate scenario-specific rules when transferred to new environments.

[0043] To this end, this paper integrates Multimodal Large Language Models (MLLMs) with a structured debate mechanism to effectively address the challenges posed by hardware variability and environmental differences. By creating a series of simulated "make-or-break" scenario benchmarks, the decision success rate is significantly improved and inference time is significantly reduced.

[0044] Specifically, see Figure 1 and Figure 2 , Embodiment 1 of the present application is a binary problem decision-making method for a heterogeneous robot system based on a multimodal large language model, which comprises the steps of:

[0045] S1. Obtain environmental feedback, and make a decision judgment on a binary problem decision scenario based on the environmental feedback based on a multimodal large language model to obtain a judgment result.

[0046] In this example, the Vision Language Model (VLM) from the multimodal large language model is used as the backbone. By extending MetaGPT, a structured debate-driven decision-making framework is proposed to address the "make-or-break" problem. MetaGPT is an efficient multi-agent collaboration framework based on a large language model and features a structured communication protocol.

[0047] This paper focuses on roles, viewing heterogeneous robots as supported intelligent agents with diverse capabilities. Leveraging the commonsense reasoning capabilities of visual language models, it enables robots to make accurate decisions and reach consensus in critical tasks through a systematic debate process based on multimodal feedback.

[0048] Specifically, for a given “make-or-break” scenario with N robots, the debate process begins with each robot perceiving the environment using available actions and making a judgment, where the i-th robot prompts the visual language model to make a decision based on the environment feedback o i (e.g. RGB images) and character-specific hints p i Make an initial judgment i :

[0049] VLM(o i ,p i )→j i ,i∈{1,2,…,N};

[0050] Among them, VLM stands for visual language model.

[0051] For example, a robot tasked with assessing wreckage during a joint search and rescue mission could provide the prompt: You are responsible for assessing wreckage during a joint search and rescue mission. Analyze [environmental feedback] and provide your judgment.

[0052] The i-th robot can take the prompt p i and the corresponding environmental feedback o i Input the visual language model, and the visual language model outputs the corresponding initial judgment j i .

[0053] S2. Exchange opinions with other robots based on the multimodal large language model, update the judgment result, and obtain a final judgment result.

[0054] For details, please refer to Figure 2 Since the initial judgment is affected by the physical capabilities of the robot (such as the height of the camera), the judgments output by the visual language model are not necessarily consistent with each other. Therefore, the framework of the present invention allows robots to exchange judgments, and each robot must provide evaluations to other robots. That is, each robot obtains the initial judgments provided by other robots as reference judgments and evaluates the initial judgments, where the i-th robot provides the evaluation e to the k-th robot. i,k :

[0055]

[0056] Among them, j k is the judgment of the k-th robot.

[0057] Specifically, in this embodiment, the robot inputs the following prompt to the visual language model: You have [judgment], please evaluate [reference judgment].

[0058] The i-th robot combines the above prompts and the i-th robot's judgment j k, and the k-th robot's judgment j k By inputting the visual language model, we can get the evaluation e of the i-th robot on the k-th robot. i,k At the same time, the i-th robot also receives the evaluation e from all robots k,i .

[0059] After collecting all peer evaluations, each robot will reflect on its initial judgment and the evaluations received from its peers, make necessary adjustments, and finally use the visual language model to output a binary answer to the initial decision question. For the i-th robot, it uses the visual language model to output the answer a i :

[0060] VLM(j i ,{e k,i |k∈{1,2,…,N},k≠i})→a i ∈{0,1};

[0061] where a i The value 0 means "no", a i A value of 1 means "yes".

[0062] In this embodiment, the robot generates the following prompt: You have received [evaluation], please reflect on the feedback and make a decision: whether you should continue, yes or no.

[0063] It can be seen that the above method requires multiple rounds of discussion and feedback for the robot to reach a conclusion. When the robot receives peer evaluation, the reasoning time is measured by the number of times the visual language model is called, that is, the reasoning time is equal to the number of intelligent agents N.

[0064] In sequential observation, all robots call the same multimodal large model in turn. When conducting sequential dialogues, all dialogue histories will be input to the next dialogue. For K rounds of dialogue, the overall reasoning time from observation to decision is T seq and the total number of calls to the visual language model C seq will be T seq =C seq =K·N.

[0065] To this end, this embodiment adopts a parallel dialogue mechanism, designs a message pool, and has multiple multimodal large models. When each robot calls the multimodal large model, it only selects the required dialogue input visual language model from the message pool, allowing the robot to make multiple calls between visual language models simultaneously, reducing the overall inference time to T par =K, at the cost of increasing the total number of calls to the visual language model to C par=K·N·(N-1). However, the present invention believes that for the defined “make-or-break” tasks, inference time is a more important optimization goal than inference cost.

[0066] S3. Obtaining a group judgment based on the final judgment result, and performing work based on the group judgment.

[0067] Specifically, the consensus will be based on a weighted average of all robots’ answers:

[0068]

[0069] in, represents consensus, i.e. group judgment, w i Indicates the weight of the i-th robot. In this embodiment, set and That is, all robots have the same weight, which is equivalent to voting. The final decision is made by rounding to the nearest whole number. However, if If it is equal to 0.5 (e.g., a tie vote), human intervention is required to make the decision.

[0070] Finally, the present invention is applied in a "make-or-break" scenario, which is a binary scenario where a critical collective decision that has a significant impact on the overall mission success must be made, and the outcome of the decision is binary: either complete success or complete failure.

[0071] Due to the inherently high risk, these missions inherently require precise planning, execution, and coordination. Figure 3 ,In this embodiment, five scenarios of different cooperative tasks are constructed in a ,simulation simulator (IsaacSim) with a physics engine.,Each scenario contains nine sub-scenarios with different ,environmental or robot properties.

[0072] The heterogeneous robot teams in each cooperative task correspond to a category of heterogeneous robot systems defined in existing techniques. Specifically, heterogeneous robot systems can be divided into physical heterogeneity and behavioral heterogeneity based on the source of heterogeneity. Behavioral heterogeneity can be further divided into behavioral heterogeneity with the same goal and behavioral heterogeneity with different goals.

[0073] From the above definitions, we can get five categories of heterogeneous robot systems:

[0074] The robots are physically different but share the same behavioral model;

[0075] The robots are physically different and have different behavioral models and goals;

[0076] The robots are physically different and have different behavioral models, but share the same goals;

[0077] The robots are physically identical and share the same goals, but have different behavioral models;

[0078] The robots are physically identical but have different behavioral models and goals.

[0079] Based on the classification of heterogeneous robot systems, the five simulation scenarios are fully focused on the cooperative tasks of heterogeneous robot systems. The detailed description of each scenario is as follows:

[0080] A joint aerial and ground path planning task: A humanoid robot needs to pass through a group of working robots. A drone and a humanoid robot observe the robot cluster from the air and ground, respectively, assess the situation, and decide through debate whether the humanoid robot can pass through the group. This corresponds to a heterogeneous robotic system where the robots are physically different but share the same behavioral model.

[0081] Joint search and rescue missions: In cooperative search and rescue scenarios, humanoid robots are only responsible for clearing debris, while quadruped robots are responsible for searching for signs of life. They need to decide whether to continue the rescue based on the number of potential survivors and the size of the debris. This corresponds to a heterogeneous robot system category where the robots are physically different and have different behavioral models and goals.

[0082] A drone docking with a robotic platform task involves docking a drone with a robotic platform on the ocean surface. The robotic platform observes the drone's position through its camera, and the drone observes the robotic platform's position through its camera. They negotiate and determine whether the drone can dock with the robotic platform. This corresponds to a heterogeneous robotic system, where the robots are physically distinct and have different behavioral models, but share the same goal.

[0083] Collaborative Liquid Transport Task: Ground robots move in formation to transport liquid. One robot observes the environment, while another observes the liquid's stability. They need to decide whether to adjust their speed. This corresponds to a heterogeneous robotic system where the robots are physically identical and share the same goal, but have different behavioral models.

[0084] Collaborative Target Monitoring: Within a large area, physically identical drones work together to monitor targets. Remote monitoring drones can monitor the drone density of a sub-area, while close-range monitoring drones can monitor targets within their respective sub-areas. If a sub-area has a large number of targets, a higher density of drones may be required for further monitoring. Remote monitoring drones need to discuss with close-range monitoring drones to determine whether to adjust the layout of the close-range drones. This corresponds to a heterogeneous robotic system category where the robots are physically identical but have different behavioral models and objectives.

[0085] Based on the aforementioned scenarios, this invention sets a standard outcome (yes / no) for each task. For each large scenario, its sub-scenarios are tested five times, and a decision success rate metric is obtained to verify the performance of the invention's solution. The decision success rate metric takes a decision (yes / no) output by the debate framework and compares it with the standard outcome to calculate the decision success rate, reflecting the accuracy of the framework in completing the task.

[0086] In addition, in order to verify the effectiveness of the parallel dialogue mechanism framework in this embodiment, a horizontal comparison was conducted. The parallel dialogue mechanism framework of the present invention was compared with a two-round debate framework consisting of an initial judgment round and a final decision round, and a four-round debate framework consisting of two additional rounds of sequential observation and evaluation by peers in the middle. These frameworks adopt a sequential dialogue mechanism and serve as baselines in the evaluation. In both frameworks, historical dialogue records are stored in a message pool, from which robots in subsequent rounds can retrieve relevant information. In this embodiment, the overall reasoning time indicator is obtained to reflect the efficiency of the framework.

[0087] The overall reasoning time metric is used to obtain the overall reasoning time of the debate framework, which measures real-time performance. We tested the average reasoning time of tasks in each large scenario to demonstrate the efficiency of the framework.

[0088] In the test, the method of this embodiment showed better accuracy and stability in most cases, covered the largest area on the radar chart, and always achieved the lowest overall inference time. In the last scenario with the largest number of robots, the advantages of the solution of this embodiment were more obvious.

[0089] Therefore, the solution of this embodiment shows significant performance improvements. Specifically, as shown in the table below, compared to the two-round debate framework, it achieves a 19.6% increase in decision success rate and a 22.5% reduction in overall reasoning time. Furthermore, compared to the four-round debate framework, this framework improves decision accuracy by 15.6% and reduces overall reasoning time by 60.7%.

[0090] Table 1: Test results

[0091] This embodiment 2 rounds of debate 4 rounds of debate Decision success rate indicator 87.1 67.5 71.5 Overall inference time metrics 12.23 15.77 31.08

[0092] Example 2

[0093] See Figure 4 Based on the above method, the present invention also provides a binary problem decision system for a heterogeneous robot system based on a multimodal large language model, and the binary problem decision system for a heterogeneous robot system based on a multimodal large language model includes:

[0094] A judgment module 51 obtains environmental feedback, performs decision judgment on a binary problem decision scenario based on the environmental feedback based on a multimodal large language model, and obtains a judgment result;

[0095] The communication module 52 exchanges opinions with other robots based on the multimodal large language model, updates the judgment result, and obtains a final judgment result;

[0096] The decision module 53 obtains a group judgment according to the final judgment result and executes the group judgment.

[0097] Example 3

[0098] See Figure 5 Based on the above method, the present invention further provides a terminal, which includes a processor 10, a memory 20, and a display 30. However, it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead.

[0099] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Furthermore, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code of the installation terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a decision program 40 for a binary problem of a heterogeneous robot system based on a multimodal large language model is stored on the memory 20. The decision program 40 for a binary problem of a heterogeneous robot system based on a multimodal large language model can be executed by the processor 10, thereby realizing the terminal in this application.

[0100] In some embodiments, the processor 10 can be a central processing unit (CPU), a microprocessor or other data processing chip, used to run the program code or process data stored in the memory 20, such as executing related programs of the binary problem decision-making method of the heterogeneous robot system based on a multimodal large language model.

[0101] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light Emitting Diode) touch screen, etc. The display 30 is used to display information on the terminal and to display a visual user interface.

[0102] In one embodiment, when the processor 10 executes a binary problem decision-making program 40 for a heterogeneous robot system based on a multimodal large language model in the memory 20, the steps of the binary problem decision-making method for a heterogeneous robot system based on a multimodal large language model as described above are implemented.

[0103] Example 4

[0104] This embodiment provides a storage medium, which stores a readable storage medium. A binary problem decision program for a heterogeneous robot system based on a multimodal large language model is stored in the medium. When the binary problem decision program for a heterogeneous robot system based on a multimodal large language model is executed by a processor, the steps of the binary problem decision method for a heterogeneous robot system based on a multimodal large language model are implemented as described above.

[0105] In summary, the present invention effectively addresses the challenges brought about by hardware variability and environmental differences through a multimodal large language model and debate mechanism, can adapt to different work scenarios and different heterogeneous robot systems, and has good versatility.

[0106] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or terminal comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or terminal comprising the element.

[0107] Of course, those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware (such as a processor, controller, etc.) through a computer program. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The storage medium can be a memory, a magnetic disk, an optical disk, etc.

[0108] It should be understood that the application of the present invention is not limited to the above examples. For those skilled in the art, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A decision-making method for binary problems in a heterogeneous robot system based on a multimodal large language model, characterized in that: The decision-making method for binary problems by a heterogeneous robot system based on a multimodal large language model includes: Obtaining environmental feedback, and making a decision judgment on a binary problem decision scenario based on the environmental feedback based on a multimodal large language model to obtain a judgment result; Exchanging opinions with other robots based on the multimodal large language model, updating the judgment result, and obtaining a final judgment result; A group judgment is obtained according to the final judgment result, and the group judgment is executed.

2. The binary problem decision-making method for a heterogeneous robot system based on a multimodal large language model according to claim 1 is characterized in that: The obtaining of environmental feedback and making judgments based on the environmental feedback based on the multimodal large language model specifically include: Obtaining a task role and environmental feedback, and generating a judgment prompt word according to the task role and the environmental feedback; The judgment prompt word is input into a multimodal large language model, and a result output by the multimodal large language model is obtained as the judgment result.

3. The binary problem decision-making method for a heterogeneous robot system based on a multimodal large language model according to claim 1 is characterized in that: The exchange of opinions is performed for multiple rounds, and the updated judgment after the last round of exchange of opinions is used as the final judgment result.

4. The binary problem decision-making method for a heterogeneous robot system based on a multimodal large language model according to claim 3 is characterized in that: Each round of the exchange of views includes: Obtaining reference judgments of each other robot and evaluating each reference judgment based on the multimodal large language model; Sending the evaluation to the corresponding other robots, and obtaining the reference evaluation transmitted by each of the other robots; The judgment result is updated according to each of the reference evaluations based on a multimodal large language model.

5. The binary problem decision-making method for a heterogeneous robot system based on a multimodal large language model according to claim 4 is characterized in that: There are multiple large multimodal language models; Obtaining the reference judgment of each other robot and evaluating each reference judgment based on the multimodal large language model specifically includes: Acquire each of the reference judgments from the message pool, and generate an evaluation prompt word for each of the reference judgments according to the reference judgments, to obtain a plurality of evaluation prompt words; Each of the evaluation prompt words is input into an independent multimodal large language model to obtain an evaluation of the reference judgment corresponding to the evaluation prompt word.

6. The binary problem decision-making method for a heterogeneous robot system based on a multimodal large language model according to claim 1, characterized in that: The obtaining of a group judgment according to the final judgment result specifically includes: Obtain the final judgment results of all robots and the weights of all robots; The group judgment is obtained by performing weighted processing based on each of the final judgment results and the weights.

7. The binary problem decision-making method for a heterogeneous robot system based on a multimodal large language model according to claim 6 is characterized in that: All robots have the same weight.

8. A binary problem decision-making system for heterogeneous robot systems based on a multimodal large language model, characterized by: The decision-making system of the heterogeneous robotic system for success-or-failure scenarios includes: A judgment module obtains environmental feedback and makes a decision judgment on a binary problem decision scenario based on the environmental feedback based on a multimodal large language model to obtain a judgment result; A communication module exchanges opinions with other robots based on a multimodal large language model, updates the judgment result, and obtains a final judgment result; The decision module obtains a group judgment according to the final judgment result and executes the group judgment.

9. A terminal, characterized in that: The terminal includes: a memory, a processor, and a binary problem decision-making program for a heterogeneous robot system based on a multimodal large language model, which is stored in the memory and can be run on the processor. When the binary problem decision-making program for a heterogeneous robot system based on a multimodal large language model is executed by the processor, the terminal is controlled to implement the steps of the binary problem decision-making method for a heterogeneous robot system based on a multimodal large language model as described in any one of claims 1 to 8.

10. A readable storage medium, characterized in that: The readable storage medium stores a binary problem decision-making program for a heterogeneous robot system based on a multimodal large language model. When the binary problem decision-making program for a heterogeneous robot system based on a multimodal large language model is executed by a processor, the steps of the binary problem decision-making method for a heterogeneous robot system based on a multimodal large language model as described in any one of claims 1 to 8 are implemented.

Images