A multi-modal large model layered multi-machine system and a scheduling method

Abstract

The application discloses a multi-modal large model layered multi-machine system and a scheduling method, relates to the field of artificial intelligence, and the cloud server of the system deploys a multi-modal large model; an edge side computing device is used for acquiring task requirements; an execution end unmanned system is used for acquiring multi-modal environment situation information and task completion feedback information and transmitting the multi-modal environment situation information and the task completion feedback information to the edge side computing device; the edge side computing device comprises a multi-modal memory database; the multi-modal memory database is used for providing retrieval data to the cloud server; the edge side computing device deploys a plurality of intelligent agent nodes; the intelligent agent nodes are used for completing data interaction; and the multi-modal large model of the cloud server is used for outputting structured decision and task scheduling planning information according to the task requirements, the retrieval data, the multi-modal environment situation information and the task completion feedback information transmitted by the edge side computing device. The application can improve the environmental adaptability of the system and the collaboration of the intelligent agents.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a multimodal large-model hierarchical multi-machine system and scheduling method. Background Technology

[0002] Multi-drone unmanned systems, due to their potential collaborative characteristics, can be well applied to various task scenarios, improving task completion efficiency. Typical task scenarios include target search and security patrol. With technological advancements, the proposal and application of cloud-edge-device architecture have greatly enhanced the real-time response capabilities of multi-drone unmanned systems, reduced data transmission and storage costs, and further expanded the operational range and application scenarios of unmanned systems, enabling them to handle more complex situations and complete larger-scale tasks. Therefore, in recent years, the application of cloud-edge-device architecture in multi-drone unmanned systems has gradually become a hot topic in engineering and research.

[0003] Existing methods for applying cloud-edge-device architecture to multi-machine unmanned systems often only consider computing power allocation. They expand the computing power of the execution-end unmanned system by deploying data storage devices and servers in the cloud and offloading computing power to edge computing devices. For the application of task scenarios, they are often limited to model training to complete single targeted tasks, such as mapping in unfamiliar environments and route planning. They fail to fully utilize the advantages of cloud-edge-device architecture to expand the operational scope and application scenarios of unmanned systems.

[0004] Meanwhile, although the existing cloud-edge-device architecture has tried to modularize the execution-end unmanned system as much as possible, allowing it to focus on system control and task execution and reduce the manufacturing and maintenance costs of the execution end, it is still limited by the single nature of the tasks planned and issued by the cloud and edge, which cannot fully explore the application potential of the execution-end unmanned system in the task scenario. For example, for a single route planning task, the tasks performed by each intelligent agent on the execution end are basically to perceive the surrounding environment and track the path, which is highly repetitive and fails to effectively utilize the diversity of intelligent agent configurations, functions and other characteristics, and the collaboration among intelligent agents is not strong.

[0005] Therefore, technologies that can continuously improve environmental adaptability and agent collaboration are needed. Summary of the Invention

[0006] The purpose of this application is to provide a multimodal large-model hierarchical multi-machine system and scheduling method, which can improve the system's environmental adaptability and the cooperation of various agents.

[0007] To achieve the above objectives, this application provides the following solution:

[0008] In a first aspect, this application provides a multimodal large-scale model hierarchical multi-machine system, comprising: a cloud server, an edge computing device, and an execution-end unmanned system connected sequentially; the cloud server deploys a multimodal large-scale model; the edge computing device is located at a local site where the task is executed; the edge computing device is used to acquire task requirements; the execution-end unmanned system is used to acquire multimodal environmental situation information and task completion feedback information and transmit the multimodal environmental situation information and the task completion feedback information to the edge computing device; the edge computing device includes a multimodal memory database; the multimodal memory database is used to provide retrieval data to the cloud server; the edge computing device deploys multiple intelligent agent nodes; the intelligent agent nodes are used to complete data interaction; the multimodal large-scale model of the cloud server is used to output structured decision and task scheduling planning information based on the task requirements transmitted by the edge computing device, the retrieval data, the multimodal environmental situation information, and the task completion feedback information.

[0009] In one embodiment, the cloud server includes a computing server; the multimodal large model is deployed on the computing server.

[0010] In one embodiment, the cloud server further includes: a cloud data storage device; the cloud data storage device is used to store device information of the execution-end unmanned system; the cloud data storage device is connected to a computing server.

[0011] In one embodiment, the edge computing device includes: a ground-based data and image transmission module and multiple mobile edge computing devices; the intelligent agent node is deployed on the mobile edge computing device; the ground-based data and image transmission module is connected to a cloud server and an unmanned execution system, respectively.

[0012] In one embodiment, the edge computing device further includes an edge data storage device; the edge data storage device is connected to the ground end of the data transmission module.

[0013] In one embodiment, the execution-side unmanned system includes multiple heterogeneous drones and unmanned vehicles; the drones and unmanned vehicles are all connected to edge computing devices.

[0014] Secondly, this application provides a multimodal large-scale model hierarchical multi-machine scheduling method. This method applies the aforementioned multimodal large-scale model hierarchical multi-machine system and includes: acquiring task requirements, multimodal environmental situation information, task completion feedback information, and retrieved data transmitted from edge-side computing devices; outputting structured decision-making and task scheduling planning information based on the multimodal large-scale model according to the task requirements, multimodal environmental situation information, task completion feedback information, and retrieved data; and sending the structured decision-making and task scheduling planning information to the edge-side computing devices to schedule the unmanned execution system.

[0015] In one embodiment, before acquiring the task requirements, multimodal environment situation information, task completion feedback information and retrieved data transmitted by the edge computing device, the method further includes: deploying a large multimodal model and performing supervised fine-tuning based on LoRA technology using a pre-built training dataset.

[0016] In one embodiment, the multimodal large model hierarchical multi-machine scheduling method further includes: constructing an execution-end embodied intelligent action library in the edge computing device; the execution-end embodied intelligent action library is a MySQL key-value pair database; wherein, the key is the core action keyword in the structured decision, and the value is the atomic operation instruction value and parameter range of the execution end in the structured decision.

[0017] In one embodiment, acquiring task requirements, multimodal environmental situation information, task completion feedback information, and retrieval data transmitted by the edge computing device specifically includes: parsing the task requirements using a mobile edge computing device to obtain a structured task description and loading the structured task description into an agent node; using the agent node to integrate the multimodal environmental situation information and task completion feedback information transmitted by the unmanned system at the execution end to construct a multimodal memory database; and the multimodal memory database provides retrieval data.

[0018] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0019] This application provides a multimodal large-scale model hierarchical multi-machine system and scheduling method. It utilizes an unmanned execution system to acquire multimodal environmental situational information and task completion feedback information. The multimodal large-scale model outputs structured decision and task scheduling planning information based on task requirements transmitted from edge computing devices, the retrieved data, the multimodal environmental situational information, and the task completion feedback information. By deploying the multimodal large-scale model in a cloud server and multiple intelligent agent nodes in edge computing devices, coordination with the unmanned execution system is achieved. This fully leverages the advantages of cloud-edge computing power and utilizes the multimodal large-scale model's ability to generate continuous and reliable decisions in complex environments and task scenarios, thereby improving the system's environmental adaptability and the coordination of various intelligent agents, significantly enhancing the system's robustness and versatility. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A cloud-edge-device architecture diagram for a multimodal, large-scale, layered, multi-machine system.

[0022] Figure 2 This is a flowchart illustrating target search in a high-altitude pasture mission scenario, as shown in the example. Detailed Implementation

[0023] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] By deploying a multimodal large-scale model in the cloud and then deploying intelligent agent nodes based on this model on edge computing devices, collaboration between the execution-end unmanned system can be achieved. This approach fully leverages the advantages of cloud-edge computing power and utilizes the multimodal large-scale model's ability to generate continuous and reliable decisions in complex environments and task scenarios, significantly improving system robustness and versatility. After enhancing the system's versatility in task scenarios through multimodal large-scale model technology, the execution end can be functionally decoupled and differentiated according to the characteristics of each intelligent agent in the execution-end unmanned system, further improving the system's collaborative application in task scenarios.

[0025] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0026] like Figure 1 As shown, a multimodal large-scale hierarchical multi-machine system is provided, including a cloud server, an edge computing device, and an unmanned execution system connected in sequence.

[0027] The cloud server deploys a multimodal large model; the edge computing device is located at the local site where the task is executed; the edge computing device is used to acquire task requirements; the unmanned execution system is used to acquire multimodal environmental situation information and task completion feedback information and transmit the multimodal environmental situation information and the task completion feedback information to the edge computing device; the edge computing device includes a multimodal memory database; the multimodal memory database is used to provide retrieval data to the cloud server; the edge computing device deploys multiple intelligent agent nodes; the intelligent agent nodes are used to complete data interaction; the multimodal large model of the cloud server is used to output structured decision and task scheduling planning information based on the task requirements transmitted by the edge computing device, the retrieval data, the multimodal environmental situation information, and the task completion feedback information.

[0028] In practical applications, cloud servers and edge computing devices transmit data via the Internet, while edge computing devices and unmanned execution systems transmit data via self-organizing network data transceivers.

[0029] In one exemplary embodiment, the cloud server includes a computing server; the multimodal large model is deployed on the computing server. In practical applications, the cloud server further includes a cloud data storage device; the cloud data storage device is used to store device information of the execution-end unmanned system; the cloud data storage device is connected to the computing server.

[0030] The cloud server is deployed in a centralized server room. It consists of data storage device A and a computing server deploying a multimodal large-scale model. Data storage device A is the cloud-based data storage device. The cloud server has an interface for data interaction with edge computing devices. Data storage device A pre-stores the specific capabilities, i.e., device information, of each unmanned device at the execution end, such as unmanned vehicle configuration characteristics, flight speed, flight time, range, and the capability of its onboard actuators, as well as the speed and actuator capabilities of unmanned vehicles, which serve as one of the input bases for task scheduling and planning. The cloud server receives the following data information from the edge computing devices:

[0031] (1) The system task or task requirement in the current scenario described in natural language, such as "detect and warn of dangerous situations".

[0032] (2) Multimodal information collected by different sensors of different execution devices, including image, audio, location, depth and other information, serves as the basis for the perception input of the fusion situation.

[0033] (3) The operating data of each unmanned device at the execution end serves as the basis for judging the operating status of each device, and also as the basis for judging whether the system needs to be redundantly deployed. The deployable redundancy of the system means that, under the premise of modularizing the execution tasks as much as possible, if an execution device goes offline due to environmental or hardware problems, the large model can schedule tasks based on the functional coverage of other unmanned devices to ensure the normal operation of the system and improve the system reliability.

[0034] (4) The system's plot memory and operation sequence memory over a period of time serve as the basis for the output of the large model for subsequent reasonable task scheduling planning. Plot memory refers to the environment and internal situation information of the system in the previous time series, while operation sequence memory refers to the relevant operations that the execution end has completed in the previous time series, such as going to a certain waypoint or performing a capture operation.

[0035] After receiving the above data, the cloud server stores it in data storage device A, where the large model computing server completes the necessary data collection. Following data collection, the large model, based on system task requirements and the fusion of environmental situational information, outputs structured decision-making and task scheduling planning information, tailored to the scenario adaptability of different levels / configurations of unmanned execution devices. This includes the expected position and state of each unmanned execution device, and the environmental information requirements for sensor acquisition. The structured nature of the decision-making and task scheduling planning information output by the cloud server is required to facilitate further refinement of the information by edge computing devices down to the individual actuators of the unmanned execution devices.

[0036] The cloud server's multimodal big model receives four types of information from edge computing devices as the basis for decision-making: (1) task requirements described in natural language; (2) multimodal environmental situation information after fusion processing; (3) real-time operation data of each unmanned device at the execution end; and (4) the system's historical plot memory and operation sequence memory.

[0037] The capabilities of the multimodal large model stem from its pre-training and supervised fine-tuning. The system uses a pre-built training dataset covering various task scenarios to fine-tune the large model. This dataset includes task descriptions, simulated multimodal data, device states, and matching structured expected decisions and task scheduling information written by experts. Through this process, the multimodal large model learns the mapping relationship from input information to structured output.

[0038] Upon receiving real-time input, the multimodal large model, based on its learned capabilities through fine-tuning, and referencing system task requirements, current environmental conditions, equipment status, and historical memory, generates structured decision-making and task scheduling information.

[0039] In one exemplary embodiment, the edge computing device includes: a ground-based data transmission module and multiple mobile edge computing devices; the intelligent agent node is deployed on the mobile edge computing devices; the ground-based data transmission module is connected to a cloud server and an unmanned execution system, respectively. In practical applications, the edge computing device further includes: an edge data storage device; the edge data storage device is connected to the ground-based data transmission module.

[0040] The edge computing device is deployed at the local site where the task is executed. It consists of a data storage device B, a ground-based data transmission module, and several mobile edge computing devices. Data storage device B serves as the edge-side data storage device, while the mobile edge computing devices deploy multiple intelligent agent nodes based on large models, enabling them to interface with the cloud server and the execution-end unmanned equipment. The edge computing device has an interface for data interaction with the cloud server and interacts with the execution-end unmanned equipment via the ground-based and airborne data transmission modules. Data storage device B pre-stores the system's embodied intelligent action library, which corresponds to the decision-making and task scheduling planning information output by the cloud server and is refined to the implementation by the execution-end unmanned equipment, thus ensuring the effective execution of decisions. The edge computing device acts as a bridge between the cloud server and the execution-end unmanned equipment, interacting with both. It also receives system tasks described in natural language from system users as the initial input for system operation. After receiving initial input, the edge computing device receives multimodal environmental perception information and operational data collected by the unmanned execution device via the data transmission module. This data is stored in data storage device B, forming a multimodal memory database containing plot memories and operation sequence memories. This database is then retrieved and uploaded to the cloud server, serving as one of the bases for outputting reasonable task scheduling plans. If the system is in its initial state, the operation sequence is recorded as empty. After the system tasks, multimodal environmental perception information, unmanned execution device operational data, plot memories, and operation sequence memories are retrieved and uploaded to the cloud server via the data interface, the edge computing device receives task scheduling planning information from the cloud server. This information is then matched with the embodied intelligent action database to generate command information that can be implemented on each actuator of the unmanned execution device. This command information is sent to the execution end. After completing the task, the execution end provides feedback on its status. The edge computing device stores this process in the memory database as operation sequence memories, repeating this process to ensure system operation. Considering the capacity of data storage device B, the multimodal memory database uses a queue to store and release memories of earlier time series at fixed time intervals to avoid data overflow. The specific carrier for the above data interaction between edge computing devices is an intelligent agent node based on a large model. These nodes can also be implemented in multiple mobile edge computing devices. When an edge computing device cannot operate normally due to software or hardware issues, parallel migration of nodes and data can be achieved through edge-to-edge collaboration mechanisms, thereby improving edge autonomy. The specific data processing flow inside the intelligent agent node can be summarized as follows:

[0041] Data aggregation and fusion: As the starting point for data processing, the intelligent agent node first receives and aggregates two types of information: one is a structured task description formed by parsing the user's natural language input; the other is heterogeneous data streams transmitted in real time from the unmanned system at the execution end, including multimodal environmental situational information (images, audio, depth, etc.) and the device's own operational status data. The intelligent agent node uses timestamp alignment and device identifier association technology to effectively fuse and initially process this data from different sources and with different modalities.

[0042] Dynamic Memory Construction and Retrieval: Based on the fused data, agent nodes dynamically construct and maintain a "multimodal memory database." This database not only records the real-time evolution of the environmental situation but also includes the system's past operation sequences and episodic memories. Before needing to communicate with the cloud, the agent retrieves the most relevant historical information from this database according to the current task requirements.

[0043] Information integration and uploading: The intelligent agent packages and integrates the current structured task description, real-time multimodal environmental situation information, equipment operating status, and relevant historical data retrieved from the memory database, and uploads them to the cloud server through the data interface, providing comprehensive and context-rich input for advanced decision-making and task planning of the multimodal large model in the cloud.

[0044] Decoupling and Refinement of Cloud-Based Decision-Making: Upon receiving structured, high-level decision-making and task scheduling information from the cloud server, the intelligent agent executes its core "decoupling and differentiation" function. It queries the "embodied intelligent action library" pre-stored in the local edge data storage device B, which contains a set of atomic operation instructions executable by the unmanned device. Based on the core action keywords in the cloud-based decision, the agent matches the corresponding atomic operation instruction template and fills it with specific parameters from the decision information (such as target coordinates, flight altitude, speed, etc.), thereby refining the high-level, abstract plan into a series of low-level control instruction sequences that can be directly understood and executed by the specific unmanned device hardware.

[0045] Command Issuance and Closed-Loop Feedback: The generated sequence of low-level control commands is issued to the corresponding unmanned devices at the execution end. After completing the operation, the device will feed back its execution status and task completion to the agent node. The agent receives this feedback and stores it as a new "operation sequence memory" in the multimodal memory database, thus forming a complete "perception-planning-execution-feedback" closed loop and providing a learning basis for subsequent task iterations.

[0046] The Embodied Intelligent Action Library is a relatively static, pre-built database in the form of a MySQL key-value database. Its core function is translation. It maps the "core action keywords" (high-level instructions) in the structured decisions output by the cloud-based large model to "atomic operation instruction sets and their parameter ranges" (low-level control instructions) that can be understood and executed by specific device models at the execution end. This library serves as a bridge for implementing decisions.

[0047] A multimodal memory database is a dynamically updated database, taking the form of a vector database based on integrated data. Its function is recording and memorizing. It stores historical information about the system's operation, including environmental situation evolution, equipment operation history, task feedback, etc., and attaches metadata such as timestamps and device identifiers to these records. This memory data will serve as an important input for subsequent decision-making by large cloud-based models, providing them with historical context.

[0048] In one exemplary embodiment, the execution-side unmanned system includes multiple heterogeneous drones and unmanned vehicles; all drones and unmanned vehicles are connected to edge computing devices. The execution-side unmanned system consists of various unmanned devices that forward-deploy to perform tasks.

[0049] The execution-end unmanned system consists of multiple heterogeneous drones and unmanned vehicles. Fixed-wing drones cruise at higher altitudes, with higher speeds, longer ranges, and longer flight times. They can carry payloads such as IMUs and cameras to perform tasks such as continuous global situational awareness monitoring and signal relay in high-altitude environments. Rotary-wing drones cruise at lower altitudes and are flexible and can hover. They carry actuators such as cameras and loudspeakers to perform detailed situational awareness collection and communication tasks in low-altitude environments. Unmanned vehicles carry cameras and laser emitters to perform tasks such as ground situational awareness collection and mapping at the ground level. The tasks performed by these unmanned devices at each level are just examples; they can perform other tasks depending on the mission scenario and by changing the payloads. Both drones and unmanned vehicles are equipped with computing power and onboard data transmission modules to ensure the implementation of underlying control and the uploading of data.

[0050] This application relates to a hierarchical multi-machine system that deploys a multimodal large model on a cloud server, embodied intelligent agents on edge computing devices, and has multiple levels of execution terminals. This system can complete multiple unplanned tasks, rather than being designed for a specific task, and belongs to the fields of artificial intelligence and unmanned aerial vehicle (UAV) technology. "Embodied" refers to the concept of combining intelligence with a physical entity. It emphasizes that an intelligent system not only exists in the digital world but also possesses a physical "body," capable of perceiving the physical world through sensors and interacting with it through actuators. Specifically, after deploying intelligent agents, UAVs and unmanned vehicles can autonomously complete a series of coherent actions in the real world, such as takeoff, landing, moving to designated coordinates, and performing grasping actions. This is significant for improving the system's versatility in various task scenarios and the collaborative capabilities of multiple intelligent agents within the system. Specifically, this system, by deploying a multimodal large model on a cloud server, leverages the ability of large model technology to generate continuous and reliable decision-making and planning information for complex environments, significantly improving the system's ability to complete tasks and its reliability in different scenarios. Simultaneously, the system also utilizes multimodal large model technology for fusion perception of complex situations, enabling the system to effectively respond to unexpected situations arising during the task. This system deploys intelligent agent nodes based on large model technology on edge computing devices, acting as a bridge between the cloud and the execution end. It uploads fused situational information such as images, sound, and depth collected by the execution end to the cloud server, serving as the input for the large model to generate decision-making and planning information, fully utilizing cloud-edge computing power. Simultaneously, it effectively publishes the decision-making and planning information output from the cloud server to the execution end, achieving multi-agent collaboration and ultimately completing the scenario task. This system deploys hierarchical, heterogeneous, multi-machine unmanned equipment at the execution end. Based on the equipment configuration and functional characteristics, it fully explores the adaptability of each level of unmanned equipment to the task scenarios, thereby building an unmanned execution end system. This achieves task modularization at the execution end, improves the collaboration of multiple intelligent agents, and significantly improves system efficiency. At the same time, the deployable redundancy between the different levels of the hierarchical multi-machine system enhances system reliability.

[0051] Based on the same inventive concept, this application also provides a multimodal large-scale model hierarchical multi-machine scheduling method. This method applies the aforementioned multimodal large-scale model hierarchical multi-machine system and includes:

[0052] Acquire task requirements, multimodal environment status information, task completion feedback information, and retrieved data transmitted from edge computing devices.

[0053] Based on task requirements, multimodal environmental situation information, task completion feedback information, and retrieved data, a multimodal large model outputs structured decision-making and task scheduling planning information.

[0054] Structured decision-making and task scheduling planning information are sent to edge computing devices to schedule the execution of unmanned systems.

[0055] In an exemplary embodiment, before acquiring the task requirements, multimodal environment situation information, task completion feedback information and retrieved data transmitted by the edge computing device, the method further includes: deploying a large multimodal model and performing supervised fine-tuning based on LoRA technology using a pre-built training dataset.

[0056] In an exemplary embodiment, the multimodal large model hierarchical multi-machine scheduling method further includes: constructing an execution-end embodied intelligent action library in the edge computing device; the execution-end embodied intelligent action library is a MySQL key-value pair database; wherein, the key is the core action keyword in the structured decision, and the value is the atomic operation instruction value and parameter range of the execution end in the structured decision.

[0057] In one exemplary embodiment, acquiring task requirements, multimodal environmental situation information, task completion feedback information, and retrieval data transmitted by the edge computing device specifically includes: using a mobile edge computing device to parse the task requirements, obtaining a structured task description, and loading the structured task description into an agent node; using the agent node to integrate the multimodal environmental situation information and task completion feedback information transmitted by the unmanned system at the execution end to construct a multimodal memory database; and the multimodal memory database provides retrieval data.

[0058] In an exemplary embodiment, a method for performing tasks in a task scenario in a multimodal large-model hierarchical multi-machine system is also provided, including the following steps:

[0059] Step 1: Deploy a multimodal large model on a cloud server and store the specific capabilities of each unmanned device at the execution end in data storage device A.

[0060] Step 2: Fine-tune the large model. The fine-tuning process uses a pre-built training dataset for supervised fine-tuning based on LoRA (Low-Rank Adaptation) technology. The training dataset includes expert-written natural language task descriptions covering multiple task scenarios, corresponding multimodal perception simulation data, device status simulation data, historical memory simulation data, and matching structured expected decision information and task scheduling planning information. The structured decision information includes status instructions, perception requirement instructions, and collaborative strategies for different levels / configurations of unmanned aerial vehicles (UAVs) and uses a unified JSON structured representation with field tags. The task scheduling planning information includes: the expected position state of each UAV; the specific operation sequence to be performed by each UAV (such as takeoff, landing, hovering, moving to a specified coordinate, turning on / off specific sensors, performing grasping / deployment, etc.); the type of environmental information, accuracy, and timeliness requirements to be acquired by each UAV; the task collaboration requirements and constraints between each UAV; and redundant deployment plans triggered by changes in device status and environment.

[0061] Step 3: Based on the decision and task scheduling planning information output by the large model, construct an embodied intelligent action library for the execution end in the data storage device B of the edge computing device. This embodied intelligent action library is constructed as a MySQL key-value database. The keys are core action keywords from the structured instructions output by the large model (such as takeoff, landing, hovering, moving to a specified coordinate, turning on / off a specific sensor, performing grab / drop, etc.), and the values ​​are the atomic operation instruction sets and their parameter ranges that can be understood by the specific device model of the corresponding execution end and implemented at the underlying controller or actuator.

[0062] Step 4: System users input task requirements into the edge computing device in the form of natural language text or voice. Task requirements include task objectives, constraints, and time window requirements. After the input information is parsed by the natural language processing module, key elements are extracted and a structured task description (JSON format) is generated. The parsed task requirements are stored in data storage device B and simultaneously loaded into the intelligent agent nodes deployed on the mobile edge computing device, serving as the input basis for subsequent multimodal memory database construction and task scheduling planning.

[0063] Step 5: Each unmanned device at the execution end collects multimodal environmental situational information by carrying heterogeneous payloads. Fixed-wing UAVs carry payloads such as high-resolution multi-sensor gimbals; rotary-wing UAVs carry payloads such as LiDAR, depth cameras, directional audio collectors, and intelligent voice-activated actuators; and unmanned vehicles carry payloads such as LiDAR and depth cameras. Each unmanned device at the execution end packages information such as images, sound, location, and depth with its own operational status data and uploads it to the edge computing device via a data transmission module.

[0064] Step 6: The edge computing device integrates multimodal environmental situational awareness and task completion feedback information from the execution-end devices (initially empty). Using timestamp alignment and device identifier association technology, it fuses and associates different modal data and the operational status and task feedback information of the same device within the same spatiotemporal range. A dynamic multimodal memory database is constructed based on data storage device B. This database is a vector database based on the integrated data. Each memory record includes: timestamp, device identifier, data source type, original data content, data description, associated task ID, etc. This database dynamically records the evolution of the environmental situation and the device operation history during system operation. The edge computing device uploads task requirement information, multimodal environmental situational awareness, execution-end unmanned device operational status data, and retrieved multimodal memories to the cloud server via a data interface.

[0065] In step 6, the multimodal memory database is stored in a queue-like manner. After a certain period of time, the storage space of the earliest time-series memory data in the queue is released to avoid data overflow.

[0066] Step 7: After receiving data from the edge computing device, the cloud server completes data storage and backup. Combining the capabilities of each layer of unmanned equipment and the adaptability of the scenario, it outputs structured decision-making and task scheduling planning information based on the expert data in Step 2, and interacts with the edge computing device through the data interface.

[0067] Step 8: After receiving decision-making and task scheduling planning information, the edge computing device, in accordance with the embodied intelligent action library, decouples and differentiates the received structured decision-making and planning information according to the following steps:

[0068] Based on the action keywords, the corresponding atomic operation instruction templates and parameter ranges are searched in the embodied action library constructed in step 3. The parameters from the structured decision planning information are then populated into the atomic operation instruction templates. The instruction operation templates generate a specific sequence of low-level control instructions executable by the device.

[0069] The commands that the actuators of each unmanned device can execute are sent to each device, and this operation is recorded as an operation sequence in the multimodal memory database.

[0070] In step 8, the specific sequence of underlying control instructions is usually represented in a protocol format supported by the device manufacturer's SDK.

[0071] In step 8, if the system fails to function properly due to hardware or software issues, the edge-to-edge collaboration mechanism can be used to migrate the intelligent agent node container running on the faulty device and its associated runtime state data and cached data to a nearby healthy edge computing device in real time, thereby achieving parallel migration of nodes and data and ensuring the smooth progress of the task.

[0072] Step 9: Each unmanned device at the execution end sends the task feedback information to the edge computing device through the data transmission module and enters the default standby state to wait for the next instruction.

[0073] Then, repeat steps 5-9 until the task request made by the system user in step 4 is met. At this point, the entire process can be reactivated by repeating step 4.

[0074] The advantages and benefits of the multimodal large-model hierarchical multi-machine system and method provided in this application are as follows:

[0075] 1. Compared to the traditional application of cloud-edge-device architecture to multi-machine unmanned systems, this method, besides considering computing power allocation, fully leverages the advantages of cloud-edge computing power by deploying a multimodal large-scale model on a cloud server. It utilizes the multimodal large-scale model's ability to generate continuous and reliable decisions for complex environments and task scenarios, generating multi-stage complex tasks and completing long-sequence dynamic objectives, thus improving task completion capability and reliability. Simultaneously, based on the technical characteristics of large-scale model pre-training and few-shot learning, it significantly enhances the system's versatility for various task environments, providing solutions for various industries. Step 2 details the specific methods for achieving this technical effect.

[0076] 2. When outputting decision-making and task planning and scheduling information, the cloud server fully integrates the system's previous time-series memory and operation sequence memory, which can avoid invalid instructions and improve system efficiency. Simultaneously, by combining multimodal situational information and the versatility of the large model, the system can effectively handle, respond to, and provide feedback on unexpected situations during task execution. Furthermore, the edge-to-edge collaboration mechanism enhances the system's edge autonomy capabilities, and combined with the system's deployable redundancy mechanism, significantly improves the system's robustness and reliability.

[0077] 3. The system fully explores and considers the task adaptability of the unmanned equipment at the execution end. By effectively utilizing the diversity of equipment configuration and functions, the system decouples and differentiates the execution end at different functional levels, thereby improving the coordination between different layers of the unmanned equipment at the execution end and the efficiency of the system in completing tasks.

[0078] In this application, the cloud server consists of a data storage device A and a computing server deploying a multimodal large model. Data storage device A pre-stores the specific capabilities of each unmanned device at the execution end, such as unmanned vehicle configuration characteristics, flight speed, flight time, range, and the ability to carry actuators, as well as the speed and actuator carrying capabilities of unmanned vehicles, serving as one of the input bases for task scheduling and planning. The edge computing device consists of a data storage device B, a ground-based data transmission module, and several mobile edge computing devices. Multiple intelligent agent nodes based on the large model are deployed on the mobile edge computing devices, enabling them to interface with the cloud server and the execution end unmanned devices. The edge computing devices have interfaces for data interaction with the cloud server and interact with the execution end unmanned devices through the ground-based and airborne data transmission modules. Data storage device B pre-stores the system's embodied intelligent action library and multimodal memory database, ensuring the rationality of decisions and their effective execution. The execution-end unmanned system consists of multiple heterogeneous drones and unmanned vehicles, capable of completing corresponding tasks according to the mission scenario and changes in payload. Both drones and unmanned vehicles are equipped with devices possessing certain computing power and onboard data transmission modules to ensure the implementation of underlying control and the uploading of data information. The following detailed description of the system's implementation process in a high-altitude pasture mission scenario is provided.

[0079] The following is an example:

[0080] S1 deploys multimodal multi-models on a cloud server and stores the specific capabilities of each unmanned device at the execution end in data storage device A. The selected drone flies at an altitude of 20m and a speed of 5m / s, and is equipped with forward-looking and downward-looking vision cameras; the selected unmanned vehicle has a maximum speed of 0.3m / s, off-road capability, and is equipped with a forward-looking vision camera.

[0081] S2 utilizes the open-source multimodal large-scale model LLaVa, which can output structured decision-making and task scheduling planning information. Corresponding to the decision-making and task scheduling planning information output by the large-scale model, an embodied intelligent action library is constructed in the data storage device B of the edge computing device. The embodied intelligent action library is a MySQL database, built with functions that each unmanned device can execute, with parameters derived from expert data, specifically including movement, turning, and image uploading.

[0082] S3, input the task requirement "A certain number of livestock have gone missing from the ranch. Please search for the missing livestock and guide them back to the ranch" to the edge computing device, and input the image information of the missing livestock.

[0083] S4, after receiving the task requirements, receives situational information from the execution end, forms a multimodal memory database, and uploads the integrated information to the cloud, where the cloud-based multimodal big model generates instructions.

[0084] (1) The execution end collects situational information such as pasture pictures and uploads them to the edge computing device through the data transmission module. At the same time, the location information and operating status information of each unmanned device are also uploaded.

[0085] (2) After receiving the situation information, task requirements and images of lost livestock from the execution end, the edge end stores the data and forms a multimodal memory database, and then integrates and uploads the information to the cloud server.

[0086] (3) After receiving the information, the cloud server multimodal big model combines the task requirements, the specific capabilities of the execution terminal unmanned equipment and the current situation information to generate an instruction: the high-altitude UAV goes to the surrounding area to conduct reconnaissance and confirm the approximate coordinates of the lost livestock, laying the foundation for the subsequent confirmation of the target location of the lost livestock.

[0087] After receiving cloud commands, the S5 edge device matches the commands with the embodied intelligent action library and transmits the commands to the various devices of the high-altitude UAV at the execution end. The various devices of the high-altitude UAV conduct reconnaissance and search in the surrounding area and transmit image information back.

[0088] S6: After the high-altitude unmanned equipment reaches the designated area, it enters a default standby state before receiving the next instruction. The cloud big data model identifies the lost target based on the returned situation information and the multimodal memory database.

[0089] (1) After each device of the high-altitude UAV reaches the designated area, it continuously hovers and waits at high altitude, and the collected image information is continuously uploaded to the edge computing device.

[0090] (2) The edge computing device uploads the memory data retrieved from the situation information to the cloud.

[0091] (3) The cloud big data model identifies the search target and its approximate area coordinates in the back-transmitted situation information, and gives instructions based on this: the high-altitude UAV continues to hover and monitor, the low-altitude UAV goes to the target area and approaches to confirm, and the ground-level unmanned vehicle goes to the target area to guide the target back.

[0092] S7: After receiving cloud commands, edge computing devices decouple and distribute the commands to unmanned devices at each execution layer. At the execution layer, high-altitude drones continuously circle and monitor the target area, transmitting image information and synchronizing the target's approximate location in real time; low-altitude drones approach the target for confirmation, and the transmitted image information is compared with the input image information of lost livestock by the cloud server; at the ground layer, unmanned vehicles reach the target area and activate their onboard loudspeakers to guide the target.

[0093] S8, under the command of the large model, the unmanned equipment at each layer of the execution end guides the target back to its home. During the return journey, the large model detects sudden vegetation and soil erosion based on the situation and performs task planning and issues commands in real time.

[0094] (1) The high-altitude UAV provides the overall situation for the return trip; the low-altitude UAV tracks and monitors the target status to prevent the target from getting lost again; the ground-level unmanned vehicle guides the target back to confirm the feasibility of ground travel on the return route. All of the above devices will upload situation information and their own status information during the execution of the mission.

[0095] (2) The cloud big model discovers that there may be vegetation and soil erosion in the pasture environment through situational information. Based on situational information and multimodal memory database information, the big model forms scheduling instructions: the high-altitude UAV goes to the soil erosion area to preliminarily confirm the area coordinate range, the low-altitude UAV goes to the area to confirm the situation and return the precise location information of the area, and the ground UAV continues to guide the target back.

[0096] S9: The high-altitude drone collects image information and returns the approximate coordinate range of the area; the low-altitude drone approaches the target, collects image information, and returns precise coordinate information; the ground-level unmanned vehicle guides the target back.

[0097] S10: Once the task requirements are completed, the cloud issues a standby command to the equipment, and the unmanned equipment on each layer enters standby mode, waiting for the input of the next task requirements.

[0098] Target search and emergency response procedures are as follows: Figure 2 As shown in the example, this embodiment demonstrates that the system proposed in this application deploys a multimodal large model in the cloud, enabling it to generate continuous and reliable scheduling and decision-making capabilities to cope with complex environments and task scenarios. It can generate complex tasks in multiple stages and effectively handle, respond to, and provide feedback on unexpected situations not mentioned in the task requirements but beneficial to the task scenario, exhibiting high system robustness. Furthermore, each layer of unmanned equipment can adapt to the tasks at each stage of the task scenario based on its inherent capabilities, improving the coordination between different layers of unmanned equipment at the execution end and the system's efficiency in completing tasks.

[0099] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0100] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0101] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0102] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A multi-modal large model layered multi-machine system, characterized in that, Comprise: The cloud server, edge side computing device and execution end unmanned system connected in turn; The cloud server deploys a multi-modal large model; the edge side computing device is arranged at a local site performing a task; the edge side computing device is used to obtain a task demand; The execution end unmanned system is used to obtain multi-modal environment situation information and task completion feedback information and transmit the multi-modal environment situation information and the task completion feedback information to the edge side computing device; the edge side computing device comprises a multi-modal memory database; the multi-modal memory database is used to provide retrieval data to the cloud server; the edge side computing device deploys a plurality of intelligent agent nodes; the intelligent agent nodes are used to complete data interaction; the multi-modal large model of the cloud server is used to output structured decision and task scheduling planning information according to the task demand transmitted by the edge side computing device, the retrieval data, the multi-modal environment situation information and the task completion feedback information; Considering the capacity of the data storage device, the multi-modal memory database stores and releases the memory of the earlier time sequence in a queue form at a fixed time interval to avoid data overflow; the specific carrier of the above data interaction of the edge side computing device is an intelligent agent node based on a large model, and these nodes are also implemented in a plurality of mobile edge computing devices; when a certain edge computing device cannot normally operate due to software and hardware problems, the node and data are parallelly migrated through edge-edge cooperation mechanism to improve the edge autonomy ability; the specific data processing process inside the intelligent agent node is summarized as the following steps: Data aggregation and fusion: the intelligent agent node, as the starting point of data processing, first receives and aggregates two types of information: one is the structured task description formed after analyzing the user natural language input; The second is the heterogeneous data stream transmitted in real time from the execution end unmanned system, including multi-modal environment situation information and device self-operation state data; The intelligent agent node uses timestamp alignment and device identification association technology to effectively fuse and preliminarily process these data with different sources and modalities; Dynamic memory construction and retrieval: based on the fused data, the intelligent agent node dynamically constructs and maintains a "multi-modal memory database"; the database not only records the real-time evolution of the environment situation, but also contains the past operation sequence and episode memory of the system; before communicating with the cloud, the intelligent agent retrieves the most relevant historical information from the database according to the current task demand.

2. The multi-modal large model hierarchical multi-machine system of claim 1, wherein, The cloud server comprises: The multi-modal large model is deployed on the computing server.

3. The multi-modal large model hierarchical multi-machine system of claim 2, wherein, The cloud server further comprises: The cloud end data storage device is used to store device information of the execution end unmanned system; the cloud end data storage device is connected with the computing server.

4. The multi-modal large model hierarchical multi-machine system of claim 1, wherein, The edge side computing device comprises: The intelligent agent node is deployed on the mobile edge side computing device; the number image transmission module ground end is connected with the cloud server and the execution end unmanned system respectively.

5. The multi-modal large model hierarchical multi-machine system of claim 4, wherein, The edge side computing device further comprises: The edge side data storage device is connected with the ground end of the digital map transmission module.

6. The multi-modal large model hierarchical multi-machine system of claim 1, wherein, The execution end unmanned system includes a plurality of heterogeneous unmanned aerial vehicles and unmanned vehicles. The unmanned aerial vehicles and the unmanned vehicles are connected with the edge side computing device.

7. A multi-modal large model hierarchical multi-machine scheduling method, characterized in that, The multi-modal large model hierarchical multi-machine scheduling method applies the multi-modal large model hierarchical multi-machine system of any one of claims 1-6, and the multi-modal large model hierarchical multi-machine scheduling method comprises: Obtaining task requirements, multi-modal environment situation information, task completion feedback information and search data transmitted by the edge side computing device; Outputting structured decision and task scheduling planning information based on the multi-modal large model according to the task requirements, the multi-modal environment situation information, the task completion feedback information and the search data; Sending the structured decision and the task scheduling planning information to the edge side computing device to schedule the execution end unmanned system.

8. The multi-modal large model hierarchical multi-machine scheduling method according to claim 7, characterized in that, Before obtaining the task requirements, the multi-modal environment situation information, the task completion feedback information and the search data transmitted by the edge side computing device, further comprising: Deploying the multi-modal large model and performing supervised fine-tuning based on the LoRA technology based on the pre-constructed training data set.

9. The multi-modal large model hierarchical multi-machine scheduling method of claim 7, wherein, Further comprising: Constructing an execution end body intelligent action library in the edge side computing device; the execution end body intelligent action library is a MySQL key-value pair database; wherein the key is a core action keyword in the structured decision, and the value is an atomic operation instruction value and a parameter range of the execution end in the structured decision.

10. The multi-modal large model hierarchical multi-machine scheduling method of claim 7, wherein, Obtaining the task requirements, the multi-modal environment situation information, the task completion feedback information and the search data transmitted by the edge side computing device, specifically comprising: Using the mobile edge side computing device to analyze the task requirements to obtain a structured task description and load the structured task description to an intelligent agent node; Using the intelligent agent node to integrate the multi-modal environment situation information and the task completion feedback information transmitted by the execution end unmanned system to construct a multi-modal memory database; the multi-modal memory database provides search data.

Images