Multi-mode cross-view-angle air-ground collaborative large model system with body

Abstract

The invention, which belongs to the technical field of artificial intelligence, discloses a multi-modal cross-view-angle air-ground cooperation big model system comprising a heterogeneous agent module, a multi-modal data processing module, a cross-view-angle fusion module, a big language model reasoning module and an action decision module. The heterogeneous intelligent agent module comprises an unmanned aerial vehicle, a ground robot and an unmanned vehicle, the unmanned aerial vehicle is provided with a high-resolution camera, an infrared imaging device, a laser radar and a meteorological sensor, the ground robot is provided with the laser radar, a camera and a force sensor, and the unmanned vehicle is provided with a camera, a radar and an inertial measurement unit. The multi-view features are fused through a cross attention mechanism, precise mapping from the features to real coordinates is realized in combination with a ScaleToRealClip model, and scale and coordinate differences of air and ground data are eliminated; meanwhile, the cross-view fusion module can dynamically update environment coordinates, help an intelligent agent to perceive temporary obstacles in real time, and construct a comprehensive and accurate environment cognition model.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, specifically to a multimodal, cross-perspective, air-ground collaborative embodied large model system. Background Technology

[0002] In the fields of robotics and intelligent navigation, multi-view fusion and heterogeneous agent collaboration have become important directions for improving environmental perception capabilities and task execution efficiency. With the increasing demands of complex scenarios such as disaster relief and autonomous driving, single-agent or single-modal perception technologies are no longer sufficient to meet task requirements. The industry is gradually moving towards multi-agent collaboration and multimodal data fusion. Currently, multi-agent systems are beginning to combine large language models and large visual models to achieve unified encoding and feature alignment of cross-view information, aiming to build a more comprehensive environmental cognition model to support autonomous navigation and task execution in complex scenarios.

[0003] However, existing technologies still have many unresolved issues:

[0004] Multi-agent collaboration often faces the problem of "air-ground information asymmetry". Due to differences in the acquisition scale and coordinate system, it is difficult to achieve accurate matching of data from the global perspective in the air and the local perspective on the ground, resulting in incomplete environmental cognition. Furthermore, communication between agents is easily affected by complex environments, resulting in data transmission delays or interruptions, which affect collaboration efficiency.

[0005] Path planning and deviation control lack dynamic adaptability and cannot adjust monitoring parameters according to scene characteristics, making it difficult to balance accuracy and efficiency in both open and narrow scenes.

[0006] The mapping between large model inference and actual physical space is insufficient, the abstract strategies output by the model are difficult to accurately translate into the actual actions of the agent, and there is a lack of effective use of historical data, making it difficult to continuously optimize task execution performance.

[0007] In addition, the existing system has shortcomings in application verification and engineering implementation in complex scenarios. The experimental process and operation specifications are not perfect, and the visualization interaction and manual intervention mechanisms need to be optimized, which affects the practicality and promotion of the system. Summary of the Invention

[0008] The purpose of this invention is to provide a multimodal cross-view air-ground cooperative embodied large model system to solve the existing multimodal cross-view air-ground cooperative problems mentioned in the background art.

[0009] To achieve the above objectives, the present invention provides the following technical solution: including a heterogeneous intelligent agent module, a multimodal data processing module, a cross-perspective fusion module, a large language model reasoning module, and an action decision module;

[0010] The heterogeneous intelligent agent module includes a drone, a ground robot, and an unmanned vehicle. The drone is equipped with a high-resolution camera, infrared imaging equipment, lidar, and weather sensors. The ground robot is equipped with lidar, a camera, and force sensors. The unmanned vehicle is equipped with a camera, radar, and an inertial measurement unit.

[0011] The multimodal data processing module can receive images, 3D point clouds, infrared heat sources, and meteorological data collected by the heterogeneous intelligent agent module. It performs semantic extraction and structured processing on the multimodal data using a visual language model, where the text encoding satisfies the formula... , Natural language instruction word segmentation sequence For text feature vectors, For the number of word segments, For feature dimensions;

[0012] The cross-view fusion module employs a Cross-Attention mechanism to fuse feature information from the UAV's global top-down view and the ground agent's local view. The attention calculation satisfies the formula... , For text feature queries, For multi-view image features, Key and Value, Using the Key dimension and combining it with the ScaleToRealClip model to map the feature space to real physical coordinates, the coordinate mapping satisfies the formula... As a feature of fusion, These are the actual geographical coordinates;

[0013] The large language model inference module generates structured task sequences and action strategies based on the fused multimodal information and the input natural language instructions. The strategy generation satisfies the formula... This is a set of action strategies; the action decision module transforms the reasoning results into control commands for heterogeneous intelligent agents, and can be implemented through path deviation formulas. Monitor the agent's path deviation, among which This is the path deviation value. For the real-time position and heading angle of the intelligent agent, To plan the position and heading angle, For heading deviation weighting coefficient ( This enables air-to-ground collaborative mission execution and dynamic correction of deviations.

[0014] As a further preferred embodiment of this technical solution: the action decision module includes a status monitoring unit, which collects the position, heading angle, and motion status data of the ground robot and the unmanned vehicle in real time, and uses the path deviation formula described in claim 1. Calculate the path deviation value; when Greater than the preset deviation threshold ( When this occurs, the state monitoring unit triggers the large language model inference module to regenerate an adapted action strategy, and simultaneously sends temporary correction instructions to the heterogeneous agent until... This ensures that the intelligent agent executes its tasks along the planned path.

[0015] As a further preferred embodiment of this technical solution: the cross-view fusion module can also provide dynamic environmental coordinate data for path planning, combining the global terrain information collected by the UAV with the local obstacle data fed back by the ground agent to update the planned path. Parameters; When temporary obstacles appear in the environment, the cross-view fusion module corrects the target area coordinates in real time and synchronizes them to the action decision module. The action decision module then uses the path deviation formula... ,in The deviation is recalculated for the updated planning coordinates and heading angle to achieve obstacle avoidance and dynamic path adjustment.

[0016] As a further preferred embodiment of this technical solution: the multimodal data processing module further includes an image encoding unit to process the data of the drone, ground robot, and unmanned vehicle at different times. The acquired images are encoded, and the ground image sequence is... The aerial image sequence is The encoding satisfies the formula , , , Ground and aerial images respectively The feature vector at each moment; the encoded image feature vector is used to assist in correcting path planning parameters. When the location of a landmark identified in the image deviates significantly from the planned coordinates, adjustments are made... Optimize the planning benchmark in the path deviation formula to improve the accuracy of deviation monitoring;

[0017] As a further preferred embodiment of this technical solution: when generating a path planning strategy, the large language model inference module dynamically adjusts the heading deviation weight coefficient λ of the path deviation formula based on historical task data.

[0018] In open and flat scenarios, λ is set to 0.1-0.2 to reduce the impact of heading deviation on the overall path.

[0019] In complex and narrow scenarios, λ is set to 0.3-0.5 to enhance the accuracy of heading and ensure that the agent travels in the preset direction and avoids collisions with obstacles.

[0020] As a further preferred embodiment of this technical solution: the preset deviation threshold The timing of ground robots performing precision tasks can be dynamically adjusted based on the type of intelligent agent and the task scenario. Set as To ensure positioning accuracy; when unmanned vehicles perform long-distance transportation tasks, Set as Balancing accuracy and traffic efficiency; adjusted Working in conjunction with the path deviation formula, it enables differentiated deviation monitoring and control in different scenarios;

[0021] As a further preferred embodiment of this technical solution, it also includes a communication coordination module, which is capable of real-time synchronization of heterogeneous intelligent agents. Data and planning parameters When the agent is in a weak signal area causing data transmission delay, the communication coordination module will cache historical path deviation data and use the path deviation formula after the signal is restored.

[0022] Calculate the deviation value to avoid the failure of deviation monitoring due to data interruption;

[0023] As a further preferred embodiment of this technical solution: the ScaleToRealClip model optimizes the mapping of fused features to real physical coordinates through a coordinate calibration algorithm. The acquisition accuracy is calibrated using the following formula: , ,in , The original collected coordinates, , For calibration bias based on landmark matching results; after calibration Substituting the path deviation formula further reduces the impact of coordinate acquisition errors on deviation calculation and improves the reliability of path monitoring;

[0024] As a further preferred embodiment of this technical solution: the large language model inference module further includes a strategy optimization unit. This strategy optimization unit calculates the deviation distribution patterns under different scenarios based on the path deviation formula and generates a deviation correction model. When similar deviation patterns appear in subsequent tasks, the strategy optimization unit can adjust the planned path parameters in advance. ,reduce The probability of occurrence, for example, in curves prone to deviation, can be optimized in advance. This value enables the agent to turn smoothly and reduces heading deviation;

[0025] As a further preferred embodiment of this technical solution, the following steps are included:

[0026] Step 1: Activate the heterogeneous intelligent agent module. The drone, ground robot, and unmanned vehicle activate their sensors to collect multimodal data and transmit it to the multimodal data processing module.

[0027] Step two: The multimodal data processing module completes text encoding and image encoding; the cross-view fusion module generates fused features and maps them to real coordinates; and the large language model inference module generates features including planning parameters. Action strategy;

[0028] Step three: The action decision module translates the strategy into control commands, the agent executes the task, and the status monitoring unit collects data in real time. Through the path deviation formula Calculate the deviation value;

[0029] Step four: The communication and coordination module synchronizes the updated parameters and real-time data to ensure that the agent executes according to the corrected path;

[0030] Step 5: The communication and coordination module synchronizes the updated parameters and real-time data to ensure that the agent executes according to the corrected path;

[0031] Step Six: After the task is completed, the feedback learning unit optimizes the input and output based on the deviation data. This will improve the ability to control deviations in subsequent tasks.

[0032] Compared with the prior art, the beneficial effects of the present invention are:

[0033] 1. In this invention, multi-view features are fused through a cross-attention mechanism and combined with the ScaleToRealClip model to achieve accurate mapping of features to real coordinates, eliminating the scale and coordinate differences between aerial and ground data; at the same time, the cross-view fusion module can dynamically update environmental coordinates, helping the agent to perceive temporary obstacles in real time and build a comprehensive and accurate environmental cognition model.

[0034] 2. In this invention, the planning parameters and real-time data between intelligent agents are efficiently synchronized through the communication and coordination module. Even in areas with weak signals, deviations can be recalculated by caching historical data to ensure uninterrupted deviation monitoring and guarantee the stability and continuity of multi-agent collaboration.

[0035] 3. In this invention, the large language model inference module can dynamically adjust the heading deviation weight coefficient based on scene characteristics, reducing the heading impact in open scenes and enhancing heading accuracy in narrow scenes; at the same time, the preset deviation threshold can be flexibly adjusted according to the type of intelligent agent and task requirements, realizing differentiated deviation monitoring and control, and balancing task accuracy and execution efficiency.

[0036] 4. In this invention, the ScaleToRealClip model optimizes the acquisition accuracy through a coordinate calibration algorithm, reducing the impact of coordinate errors on deviation calculation; the feedback learning unit optimizes model parameters and planning strategies based on task execution data, enabling the system to continuously improve the planning ability and execution accuracy of subsequent tasks and adapt to more complex scenarios.

[0037] 5. In this invention, the system has clear application procedures and experimental verification specifications in both disaster relief and unmanned autonomous driving scenarios, ensuring that the technology can be implemented. At the same time, the visual interface can intuitively display the status of the intelligent agent, the landmark recognition results and the navigation route. Operators can conveniently input commands and view results through a tablet, and can also intervene in tasks in emergency situations, enhancing the ease of use and safety of the system. Attached Figure Description

[0038] Figure 1 This invention provides a CVF-LLM overall algorithm framework for a multimodal, cross-view air-ground collaborative embodied large model system. Figure 1 ;

[0039] Figure 2 This invention provides a CVF-LLM overall algorithm framework for a multimodal, cross-view air-ground collaborative embodied large model system. Figure 2 ;

[0040] Figure 3 This invention provides an onboard perspective for an unmanned vehicle within a multimodal, cross-viewpoint, air-ground collaborative embodied large model system. Figure 1 ;

[0041] Figure 4 This invention provides an onboard perspective for an unmanned vehicle within a multimodal, cross-viewpoint, air-ground collaborative embodied large model system. Figure 2 ;

[0042] Figure 5 This is a view of a vehicle-mounted UAV (Universal Air Vehicle) of a multimodal, cross-view air-ground collaborative embodied large model system according to the present invention.

[0043] Figure 6 This is a ready-to-use diagram of a ground robot for a multimodal, cross-view, air-ground collaborative embodied large model system according to the present invention.

[0044] Figure 7 This invention provides an aerial UAV takeoff and flight path setting for a multimodal, cross-view air-ground collaborative embodied large model system.

[0045] Figure 8 The robot dog of the multimodal, cross-view air-ground collaborative embodied large model system of the present invention begins to move;

[0046] Figure 9 This is the result of landmark recognition from the perspective of an aerial UAV in a multimodal, cross-view air-ground collaborative embodied large model system according to the present invention.

[0047] Figure 10 This is the landmark recognition result from the perspective of a robot dog in a multimodal, cross-view air-ground collaborative embodied large model system of the present invention;

[0048] Figure 11 This is a ScaleToRealCLIP alignment effect diagram of a multimodal cross-view air-ground collaborative embodied large model system of the present invention;

[0049] Figure 12 After the mission of the multimodal cross-view air-ground collaborative embodied large model system of the present invention is completed, the drone robot dog returns to the starting point;

[0050] Figure 13 This is a module relationship diagram of a multimodal, cross-perspective, air-to-ground collaborative embodied large model system according to the present invention;

[0051] Figure 14 This is a flowchart illustrating the task execution process of a multimodal, cross-perspective, air-ground collaborative embodied large model system according to the present invention. Detailed Implementation

[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] Example

[0054] Please see Figures 1-14 As shown, the present invention provides a technical solution for a multimodal, cross-perspective, air-ground collaborative embodied large model system, including a heterogeneous intelligent agent module, a multimodal data processing module, a cross-perspective fusion module, a large language model reasoning module, and an action decision module;

[0055] The heterogeneous intelligent agent module includes a drone, a ground robot, and an unmanned vehicle. The drone is equipped with a high-resolution camera, infrared imaging equipment, lidar, and weather sensors. The ground robot is equipped with lidar, a camera, and force sensors. The unmanned vehicle is equipped with a camera, radar, and an inertial measurement unit.

[0056] The multimodal data processing module can receive images, 3D point clouds, infrared heat sources, and meteorological data collected by the heterogeneous intelligent agent module. It performs semantic extraction and structured processing on the multimodal data using a visual language model, where the text encoding satisfies the formula... , Natural language instruction word segmentation sequence For text feature vectors, For the number of word segments, For feature dimensions;

[0057] The cross-view fusion module employs a Cross-Attention mechanism to fuse feature information from the UAV's global top-down view and the ground agent's local view. The attention calculation satisfies the formula... , For text feature queries, For multi-view image features, Key and Value, Using the Key dimension and combining it with the ScaleToRealClip model to map the feature space to real physical coordinates, the coordinate mapping satisfies the formula... As a feature of fusion, These are the actual geographical coordinates;

[0058] The large language model inference module generates structured task sequences and action strategies based on the fused multimodal information and the input natural language instructions. The strategy generation satisfies the formula... This is a set of action strategies; the action decision module transforms the reasoning results into control commands for heterogeneous intelligent agents, and can be implemented through path deviation formulas. Monitor the agent's path deviation, among which This is the path deviation value. For the real-time position and heading angle of the intelligent agent, To plan the position and heading angle, For heading deviation weighting coefficient ( This enables air-to-ground collaborative mission execution and dynamic correction of deviations.

[0059] In this embodiment, the action decision module includes a status monitoring unit, which collects the position, heading angle, and motion status data of the ground robot and the unmanned vehicle in real time, and uses the path deviation formula described in claim 1. Calculate the path deviation value; when Greater than the preset deviation threshold ( When this occurs, the state monitoring unit triggers the large language model inference module to regenerate an adapted action strategy, and simultaneously sends temporary correction instructions to the heterogeneous agent until... This ensures that the intelligent agent performs its tasks along the planned path.

[0060] In this embodiment, the cross-view fusion module can also provide dynamic environmental coordinate data for path planning, combining global terrain information collected by the UAV with local obstacle data fed back by the ground agent to update the planned path. Parameters; When temporary obstacles appear in the environment, the cross-view fusion module corrects the target area coordinates in real time and synchronizes them to the action decision module. The action decision module then uses the path deviation formula... ,in The deviation is recalculated for the updated planning coordinates and heading angle to achieve obstacle avoidance and dynamic path adjustment.

[0061] Specifically, the multimodal data processing module also includes an image encoding unit to process data from drones, ground robots, and unmanned vehicles at different times. The acquired images are encoded, and the ground image sequence is... The aerial image sequence is The encoding satisfies the formula , , , Ground and aerial images respectively The feature vector at each moment; the encoded image feature vector is used to assist in correcting path planning parameters. When the location of a landmark identified in the image deviates significantly from the planned coordinates, adjustments are made... Optimize the planning benchmark in the path deviation formula to improve the accuracy of deviation monitoring.

[0062] In this embodiment, when generating a path planning strategy, the large language model inference module dynamically adjusts the heading deviation weight coefficient λ of the path deviation formula based on historical task data.

[0063] In open and flat scenarios, λ is set to 0.1-0.2 to reduce the impact of heading deviation on the overall path.

[0064] In complex and narrow scenarios, λ is set to 0.3-0.5 to enhance the accuracy of heading and ensure that the agent travels in the preset direction and avoids collisions with obstacles.

[0065] Specifically, the preset deviation threshold The timing of ground robots performing precision tasks can be dynamically adjusted based on the type of intelligent agent and the task scenario. Set as To ensure positioning accuracy; when unmanned vehicles perform long-distance transportation tasks, Set as Balancing accuracy and traffic efficiency; adjusted In conjunction with the path deviation formula, it enables differentiated deviation monitoring and control in different scenarios.

[0066] In this embodiment, a communication coordination module is also included, which can synchronize heterogeneous intelligent agents in real time. Data and planning parameters When the agent is in a weak signal area causing data transmission delay, the communication coordination module will cache historical path deviation data and use the path deviation formula after the signal is restored.

[0067] Calculate the deviation value to avoid the failure of deviation monitoring due to data interruption.

[0068] Specifically, the ScaleToRealClip model optimizes the mapping of fused features to real physical coordinates using a coordinate calibration algorithm. The acquisition accuracy is calibrated using the following formula: , ,in , The original collected coordinates, , For calibration bias based on landmark matching results; after calibration Substituting the path deviation formula further reduces the impact of coordinate acquisition errors on deviation calculation and improves the reliability of path monitoring.

[0069] In this embodiment, the large language model inference module further includes a strategy optimization unit. This unit calculates the path deviation formula, statistically analyzes the deviation distribution patterns under different scenarios, and generates a deviation correction model. When similar deviation patterns appear in subsequent tasks, the strategy optimization unit can adjust the planned path parameters in advance. ,reduce The probability of occurrence, for example, in curves prone to deviation, can be optimized in advance. This value enables the agent to turn smoothly and reduce heading deviation.

[0070] Specifically, the system's air-ground collaborative task execution method includes the following steps:

[0071] Step 1: Activate the heterogeneous intelligent agent module. The drone, ground robot, and unmanned vehicle activate their sensors to collect multimodal data and transmit it to the multimodal data processing module.

[0072] Step two: The multimodal data processing module completes text encoding and image encoding; the cross-view fusion module generates fused features and maps them to real coordinates; and the large language model inference module generates features including planning parameters. Action strategy;

[0073] Step three: The action decision module translates the strategy into control commands, the agent executes the task, and the status monitoring unit collects data in real time. Through the path deviation formula Calculate the deviation value;

[0074] Step four: The communication and coordination module synchronizes the updated parameters and real-time data to ensure that the agent executes according to the corrected path;

[0075] Step 5: The communication and coordination module synchronizes the updated parameters and real-time data to ensure that the agent executes according to the corrected path;

[0076] Step Six: After the task is completed, the feedback learning unit optimizes the input and output based on the deviation data. This will improve the ability to control deviations in subsequent tasks.

[0077] It should be noted that this system has clear and feasible application scenarios in the fields of complex environment navigation and unmanned autonomous driving. The specific application methods and experimental verification process are as follows:

[0078] In complex disaster environments such as forest fires and floods, this system can support cross-view air-ground embodied intelligent navigation tasks. After a disaster occurs, UAVs can quickly take off to acquire global overhead view data of the disaster area, while ground robots can penetrate deep into the core disaster area to collect detailed local environmental information. Through multimodal data fusion and cross-view collaboration, the two can achieve disaster area topographic mapping, location of trapped personnel, and rescue route planning, providing accurate environmental perception and mission execution support for disaster relief.

[0079] In the field of autonomous driving, this system conducts special application verification for urban road navigation scenarios. The experiment uses the DAIR-V2X dataset produced by the Institute for Intelligent Industry of Tsinghua University (AIR). This dataset is a large-scale multimodal and multi-view dataset in the field of vehicle-road cooperative autonomous driving research. It contains synchronously labeled data of vehicle view and vehicle-mounted drone aerial photography view, totaling 72,890 frames of images, which can comprehensively simulate complex traffic scenarios in urban roads.

[0080] The experimental procedure strictly followed a standardized process: First, the input images were preprocessed, and the LANCZOS interpolation algorithm was used to uniformly scale the longest side of all images to a preset maximum size, reducing GPU memory usage while preserving image details to the maximum extent. Then, the results were filtered. First, confidence filtering was performed, and then non-maximum suppression was performed after classification. The preliminary detection results were grouped by category, and the NMS algorithm was applied to the overlapping masks in each group to retain only the mask with the highest score in that category, thus avoiding the multiple recognition of a single object.

[0081] The system ultimately generates two standardized outputs: a detailed score report that records the category, confidence score, bounding box coordinates, and similarity score list of all candidate text descriptions for each identified landmark; and a visualized labeled image that displays the outline of landmark objects in semi-transparent color highlighting and labels the category name and confidence score, providing a clear basis for the system's performance evaluation in unmanned autonomous driving scenarios.

[0082] The algorithm engineering implementation of this system follows a clear process logic, covering data collection, processing and computation, task planning, collaborative execution, and post-operative optimization. The specific steps are as follows:

[0083] Multimodal data collection: After the task starts, each intelligent agent simultaneously activates its corresponding sensor module to collect environmental data. The UAV activates its high-resolution camera, infrared imaging equipment, lidar, and weather sensor; the ground robot activates its lidar, camera, and force sensor; and the unmanned vehicle activates its camera, radar, and inertial measurement unit. Each device collects data at a preset frequency to ensure the timeliness and completeness of the data.

[0084] Task data processing and cross-perspective computing: Each agent transmits the collected raw data to the central control system and distributed edge robot nodes through communication links, realizing distributed storage and computing of data; the Visual Language Model (VLM) is used to fuse multimodal data, including converting rescue task information in text form into structured task descriptions (language description generation), and identifying landmarks, dynamic obstacles and task targets in the scene (visual information extraction); the fused multimodal information is parsed based on the Large Language Model (LLM) to generate preliminary task-related instruction text, unify the information expression format between agents and the API interface call specifications for robot capabilities, and ensure the consistency of information transmission;

[0085] Task planning comprises two parts: path planning and task allocation. In the path planning stage, based on a pre-established topology map and node information within a known scenario, the optimal task path is generated using a visual navigation model (VNM). This considers environmental dynamics (such as temporarily appearing obstacles) and collaborative requirements (such as path coordination between drones and ground robots), and the path is corrected in real-time based on dynamically updated landmark information. In the task allocation stage, the real-time status of each agent is dynamically analyzed (such as remaining battery power, current load, task execution progress, and device health status). The optimal allocation of subtasks is calculated using LLM to ensure efficient resource utilization and task execution.

[0086] Collaborative Execution: Each agent executes specific sub-tasks according to the task allocation results. UAVs conduct wide-area reconnaissance (such as global mapping of disaster areas), real-time data transmission (synchronizing global environmental information to the ground control terminal and other agents), and environmental monitoring (such as monitoring the spread of fire). Ground robots complete ground navigation (moving according to planned paths), material transportation (such as delivering rescue supplies to trapped personnel), and precision operations in complex terrain (such as searching for trapped personnel in rubble). Unmanned vehicles perform long-distance transportation (such as transferring rescue equipment) or patrol (such as safety monitoring on urban roads). Close collaboration is formed among agents. UAVs provide global environmental maps for ground agents (ground robots and unmanned vehicles) to assist in optimizing ground navigation paths. Ground robots provide local terrain data (such as road damage) to unmanned vehicles to optimize their path decisions. During task execution, each agent transmits the task completion status back to the rescue center or control terminal in real time, allowing human operators to monitor the task progress.

[0087] Post-operative optimization: After the task is completed, all agents will upload the execution log (including task execution steps, parameter adjustment records, and fault occurrences), environmental data (collected multimodal raw data and fusion processing results), and optimization suggestions (such as improvement directions for path planning and suggestions for adjusting sensor parameters) to cloud storage. Based on the uploaded historical data, the LLM model will be optimized, and the model parameters will be updated through incremental training to improve the model's planning ability and intelligence level for similar tasks, laying the foundation for improving the efficiency and accuracy of subsequent task execution.

[0088] This system features comprehensive visualization and human-computer interaction capabilities. It presents the system's operational status through an intuitive interface and supports convenient command input and result viewing. Specific details are as follows:

[0089] Visualizations and Explanations: The system operation process includes visualizations of multiple key states, as shown in the following figures and explanations:

[0090] Ground robot readiness status diagram: Displays the robot's real-time status parameters (such as front_range, soc, x, y, etc. mentioned above) in JSON data format, clearly showing whether the robot has the conditions to perform the task.

[0091] Aerial UAV Takeoff and Flight Route Setting Map: Displays the UAV's takeoff position, preset waypoints, and flight route (such as latitude and longitude coordinates based on the WGS84 coordinate system) in a map interface, while also displaying flight altitude (such as ASL: 201.5m, HAE: 163.5m), execution time, and other information to facilitate operator monitoring of the UAV's flight trajectory;

[0092] Robot dog starts moving log information diagram: The robot dog's motion control commands and status feedback are recorded in the form of text log (such as "[INFO][1732003918.391152279][sport_control_client]:---0---"), reflecting the robot dog's motion control process;

[0093] Landmark recognition results: The results show the landmark recognition from the perspective of an aerial drone and the ground perspective of a robot dog, respectively. The recognized landmark categories (such as "car", "road", "TreeLine", "sky") and corresponding confidence scores (such as 0.24, 0.25, 0.27) are labeled, intuitively presenting the landmark positioning effect.

[0094] ScaleToRealCLIP alignment effect diagram: Show the cross-view correlation and localization results of landmark descriptions (such as "dummy", "swimming ring", "medical kit" and "roadblock") in natural language task instructions in the drone top view and robot dog first view image. The top view presents the overall distribution and relative position of landmarks, and the ground plan view refines the accuracy of bounding boxes and the target localization of occluded areas.

[0095] The image shows the return of the intelligent agent to the starting point after the task is completed: This indicates that the drone and robot dog have returned to their initial positions according to the planned path after completing the task, marking the end of the task execution cycle.

[0096] Human-computer interaction: Operators interact with the system via tablet devices, inputting natural language commands on the tablet interface (such as "Search along the main road, find the red fire extinguisher and return to the supply vehicle," or "Start from the starting point, move forward until you see the black chair, pass the flower pots until you see the fire point"). The commands are directly transmitted to the system's text encoding unit for processing. During task execution, the system saves real-time generated visual annotation images (such as drawing bounding boxes around landmark objects, labeling category names and confidence scores) to a designated path. Operators can view the annotation images and detailed score reports on the tablet interface to monitor task progress and results. In case of emergencies (such as agent failure or change of task objectives), operators can input intervention commands on the tablet to adjust the system's task planning and ensure safe and effective task execution.

[0097] The ScaleToRealClip model and LLM real device planning process in this system have refined execution logic. The specific steps and examples are as follows:

[0098] ScaleToRealClip model execution steps:

[0099] Step 1: Image loading. Read image files in PIL or OpenCV format. If loading fails, return directly to ensure the validity of the data source for subsequent processing.

[0100] Step 2: Candidate region generation. Create a SelectiveSearch instance, input the OpenCV image into the instance and configure the generation mode (fast mode or high quality mode). Obtain the initial rectangular region from the instance, and then obtain the candidate bounding box and candidate cropped image according to the filtering conditions such as area ratio and aspect ratio. If the candidate cropped image is empty, save the original image and return.

[0101] Step 3: Feature Extraction and Similarity Scoring (Improved CLIP Method). The landmark text list and general negative prompts are encoded as text input. The improved CLIP model is used to encode the text input, obtaining normalized text features. Candidate cropped images are batch-processed, pre-processed, and stacked as image input batches. Normalized image features are obtained by encoding using the CLIP model. The dot product of the normalized image features and normalized text features is calculated to obtain the cosine similarity. For each image in the batch, the corresponding target landmark score and general negative score are extracted. If the target landmark score exceeds the initial confidence threshold and the difference between the target landmark score and the general negative score exceeds the general negative score difference threshold, then (boundary box, score, landmark name) is added to the list of all potential detection results. If the list is empty, the original image is saved and returned.

[0102] Step 4: Non-maximum suppression (NMS) Group all potential detection results by landmark name. For each landmark category, extract the candidate box list and score list. Call the non_max_suppression function (based on IoU threshold) to calculate the retention index. Add the detection result with the highest score in the retention index to the detection result list after NMS. If the list is empty, save the original image and return.

[0103] Step 5: Specific Landmark Re-verification (Optional). If this function is enabled, initialize the final drawing detection result list. For each detection result in the NMS detection results, crop the corresponding region from the PIL image to obtain a cropped image, construct specific negative prompt text for the landmark, calculate the CLIP similarity between the cropped image and the specific negative prompt text, and if the detection result score exceeds (specific negative score + specific re-verification score difference threshold), add the detection result to the final drawing detection result list.

[0104] Step 6: Coordinate Alignment and Result Saving. Align the coordinate system with the real space according to the pixel scaling rule, draw the bounding box, name and score of the final detection result on the OpenCV image, and save the labeled image to the specified output path.

[0105] Example of LLM device planning steps:

[0106] Step 1: Command Input and Parsing. The operator inputs natural language commands (e.g., "The robot team moves forward, sees a roadblock, moves forward; sees a gray stone, moves right; sees a tree, turns left until it sees the fire, observes if there is a fire extinguisher, and calls for manual rescue"). The LLM first performs word segmentation and semantic parsing on the commands and extracts a list of landmarks (e.g., ['roadblock', 'gray stone', 'fire', 'fire extinguisher']).

[0107] Step 2: Waypoint generation. Based on the extracted list of landmarks and environmental topology information, waypoints are generated (e.g., the number of waypoints is 5), and the path navigation process is started.

[0108] Step 3: Action Command Output. Based on the real-time acquired agent state and environmental perception data, LLM outputs action commands step by step, as shown in the following example: "Benben Dog Action: moveforward" (controls the robot dog to move forward). When the robot dog "observes: roadblock", it outputs "Benben Dog Action: right" (controls the robot dog to turn right); when it "observes: gray stone", it outputs "Benben Dog Action: left" (controls the robot dog to turn left); when it "observes: fire point", it outputs "Benben Dog Action: moveforward" (controls the robot dog to continue forward); when it "observes: fire extinguisher", it outputs "Benben Dog Action: moveforward (reaches the final destination)", finally completing the "task completed" status feedback, realizing the complete process from command parsing to action execution.

[0109] Working principle or structural principle: The working process of this multimodal cross-view air-ground collaborative embodied large model system relies on the coordinated linkage of various modules to form a closed-loop process of "data acquisition - processing and fusion - reasoning and decision-making - execution feedback". First, the UAV, ground robot and unmanned vehicle in the heterogeneous intelligent agent module simultaneously start their respective configured sensors to collect images, 3D point clouds, infrared heat sources, meteorological data and motion status data from the global top view and local ground view, respectively, and transmit the multi-type raw data to the multimodal data processing module;

[0110] After receiving the data, the multimodal data processing module segments and encodes the natural language commands, converting them into computer-processable text feature vectors. Simultaneously, it encodes aerial and ground image sequences acquired at different times, extracting visual features from the images. Subsequently, the cross-view fusion module uses text features as the query basis and fuses text features with multi-view image features through a cross-attention mechanism to generate fused features containing global and local correlation information. Then, using the ScaleToRealClip model, the fused features are mapped to real physical coordinates, providing accurate spatial references for subsequent path planning.

[0111] The large language model inference module generates a structured task sequence and action strategy containing parameters such as planned position and heading angle based on the fused coordinate information and natural language instructions. The action decision module translates the strategy into control instructions for each agent, driving the agents to execute tasks. At the same time, the state monitoring unit collects the real-time position and heading angle of the agents in real time, and determines whether the agents have deviated from the planned path by calculating path deviation. If the deviation exceeds the threshold, the large language model inference module is triggered to replan the strategy and send a correction command.

[0112] The communication and coordination module maintains data synchronization among agents throughout the process, ensuring efficient transmission of planning parameters and real-time status information. Even in areas with weak signals, it can compensate for deviations by caching historical data to avoid monitoring interruptions. During task execution, the cross-view fusion module will also dynamically update the planning coordinates according to environmental changes to help agents avoid temporary obstacles. After the task is completed, the feedback learning unit optimizes the model parameters based on the execution data to improve performance for subsequent tasks.

[0113] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0114] The present invention and its embodiments have been described above. This description is not restrictive, and the accompanying drawings are only one embodiment of the invention; the actual structure is not limited thereto. In conclusion, if those skilled in the art, inspired by this description, design similar structures and embodiments without departing from the spirit of the invention, such designs should fall within the scope of protection of this invention.

[0115] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multimodal, cross-viewpoint, air-ground collaborative embodied large model system, characterized in that: It includes a heterogeneous intelligent agent module, a multimodal data processing module, a cross-perspective fusion module, a large language model reasoning module, and an action decision module; The heterogeneous intelligent agent module includes a drone, a ground robot, and an unmanned vehicle. The drone is equipped with a high-resolution camera, infrared imaging equipment, lidar, and weather sensors. The ground robot is equipped with lidar, a camera, and force sensors. The unmanned vehicle is equipped with a camera, radar, and an inertial measurement unit. The multimodal data processing module can receive images, 3D point clouds, infrared heat sources, and meteorological data collected by the heterogeneous intelligent agent module. It performs semantic extraction and structured processing on the multimodal data using a visual language model, where the text encoding satisfies the formula... , Natural language instruction word segmentation sequence For text feature vectors, For the number of word segments, For feature dimensions; The cross-view fusion module employs a Cross-Attention mechanism to fuse feature information from the UAV's global top-down view and the ground agent's local view. The attention calculation satisfies the formula... , For text feature queries, For multi-view image features, Key and Value, Using the Key dimension and combining it with the ScaleToRealClip model to map the feature space to real physical coordinates, the coordinate mapping satisfies the formula... As a feature of fusion, These are the actual geographical coordinates; The large language model inference module generates structured task sequences and action strategies based on the fused multimodal information and the input natural language instructions. The strategy generation satisfies the formula... This is a set of action strategies; the action decision module transforms the reasoning results into control commands for heterogeneous intelligent agents, and can be implemented through path deviation formulas. Monitor the agent's path deviation, among which This is the path deviation value. For the real-time position and heading angle of the intelligent agent, To plan the position and heading angle, For heading deviation weighting coefficient ( This enables air-to-ground collaborative mission execution and dynamic correction of deviations.

2. The multimodal cross-view air-ground collaborative embodied large model system according to claim 1, characterized in that: The action decision module includes a status monitoring unit, which collects real-time data on the position, heading angle, and motion status of the ground robot and the unmanned vehicle, and uses the path deviation formula described in claim 1. Calculate the path deviation value; when Greater than the preset deviation threshold ( When this occurs, the state monitoring unit triggers the large language model inference module to regenerate an adapted action strategy, and simultaneously sends temporary correction instructions to the heterogeneous agent until... This ensures that the intelligent agent performs its tasks along the planned path.

3. The multimodal cross-view air-ground collaborative embodied large model system according to claim 2, characterized in that: The cross-view fusion module can also provide dynamic environmental coordinate data for path planning, combining global terrain information collected by the UAV with local obstacle data fed back by the ground agent to update the planned path. Parameters; When temporary obstacles appear in the environment, the cross-view fusion module corrects the target area coordinates in real time and synchronizes them to the action decision module. The action decision module then uses the path deviation formula... ,in The deviation is recalculated for the updated planning coordinates and heading angle to achieve obstacle avoidance and dynamic path adjustment.

4. The multimodal cross-view air-ground collaborative embodied large model system according to claim 3, characterized in that: The multimodal data processing module also includes an image encoding unit to process data from drones, ground robots, and unmanned vehicles at different times. The acquired images are encoded, and the ground image sequence is... The aerial image sequence is The encoding satisfies the formula , , , Ground and aerial images respectively The feature vector at each moment; the encoded image feature vector is used to assist in correcting path planning parameters. When the location of a landmark identified in the image deviates significantly from the planned coordinates, adjustments are made... Optimize the planning benchmark in the path deviation formula to improve the accuracy of deviation monitoring.

5. A multimodal, cross-view air-ground collaborative embodied large model system according to claim 4, characterized in that: When generating path planning strategies, the large language model inference module dynamically adjusts the heading deviation weight coefficient λ in the path deviation formula based on historical task data. In open and flat scenarios, λ is set to 0.1-0.2 to reduce the impact of heading deviation on the overall path. In complex and narrow scenarios, λ is set to 0.3-0.5 to enhance the accuracy of heading and ensure that the agent travels in the preset direction and avoids collisions with obstacles.

6. A multimodal, cross-view air-ground collaborative embodied large model system according to claim 5, characterized in that: The preset deviation threshold The timing of ground robots performing precision tasks can be dynamically adjusted based on the type of intelligent agent and the task scenario. Set as To ensure positioning accuracy; when unmanned vehicles perform long-distance transportation tasks, Set as Balancing accuracy and traffic efficiency; adjusted In conjunction with the path deviation formula, it enables differentiated deviation monitoring and control in different scenarios.

7. A multimodal, cross-view air-ground collaborative embodied large model system according to claim 6, characterized in that: It also includes a communication and coordination module, which can synchronize heterogeneous intelligent agents in real time. Data and planning parameters When the agent is in a weak signal area causing data transmission delay, the communication coordination module will cache historical path deviation data and use the path deviation formula after the signal is restored. Calculate the deviation value to avoid the failure of deviation monitoring due to data interruption.

8. A multimodal cross-view air-ground collaborative embodied large model system according to claim 7, characterized in that: The ScaleToRealClip model optimizes the mapping of fused features to real physical coordinates using a coordinate calibration algorithm. The acquisition accuracy is calibrated using the following formula: , ,in , The original collected coordinates, , For calibration bias based on landmark matching results; after calibration Substituting the path deviation formula further reduces the impact of coordinate acquisition errors on deviation calculation and improves the reliability of path monitoring.

9. A multimodal cross-view air-ground collaborative embodied large model system according to claim 8, characterized in that: The large language model inference module also includes a strategy optimization unit. This unit calculates the path deviation formula, statistically analyzes the deviation distribution patterns under different scenarios, and generates a deviation correction model. When similar deviation patterns appear in subsequent tasks, the strategy optimization unit can adjust the planned path parameters in advance. ,reduce The probability of occurrence, for example, in curves prone to deviation, can be optimized in advance. This value enables the agent to turn smoothly and reduce heading deviation.

10. A method for air-ground cooperative task execution based on the system described in any one of claims 1-9, characterized in that, Includes the following steps: Step 1: Activate the heterogeneous intelligent agent module. The drone, ground robot, and unmanned vehicle activate their sensors to collect multimodal data and transmit it to the multimodal data processing module. Step two: The multimodal data processing module completes text encoding and image encoding; the cross-view fusion module generates fused features and maps them to real coordinates; and the large language model inference module generates features including planning parameters. Action strategy; Step three: The action decision module translates the strategy into control commands, the agent executes the task, and the state monitoring unit collects data in real time. Through the path deviation formula Calculate the deviation value; Step four: The communication and coordination module synchronizes the updated parameters and real-time data to ensure that the agent executes according to the corrected path; Step 5: The communication and coordination module synchronizes the updated parameters and real-time data to ensure that the agent executes according to the corrected path; Step Six: After the task is completed, the feedback learning unit optimizes the input and output based on the deviation data. This will improve the ability to control deviations in subsequent tasks.

Images