Man-machine fusion unmanned system autonomous control method and system based on large language model

Abstract

This application discloses an autonomous control method and system for a human-machine fusion unmanned system based on a large language model, relating to the application of large language models and the field of unmanned systems. The method includes acquiring an unstructured high-level semantic instruction sequence, and executing the following steps for each instruction in the sequence until all tasks are completed: acquiring multi-sensor environmental data at the current moment and constructing a current local environment topology map; using the current instruction and the topology map as a fusion query vector to retrieve an expert policy database, and selecting a preset number of historical expert policies as a candidate policy set; based on the current instruction, the topology map, and the candidate policy set, generating the optimal policy at the current moment through a large language model and controlling its execution; if the task is not completed, updating the topology map and repeating the above steps until the current task is completed. This application enables unmanned systems to simultaneously build maps, retrieve information, make decisions, and execute, improving the unmanned system's response capability, task adaptability, and execution reliability in complex and unknown environments.

Description

Technical Field

[0001] This application relates to the fields of large language model applications and unmanned systems, and in particular to an autonomous control method and system for human-machine fusion unmanned systems based on large language models. Background Technology

[0002] With the continuous iteration and upgrading of unmanned systems technology, their mission scenarios are becoming increasingly diverse. Unmanned systems have rapidly expanded from early preset trajectory flight and basic fixed-point operations to highly unstructured and uncertain dynamic environments such as emergency search and rescue, exploration of unknown airspace, and situational awareness in complex environments. In such scenarios, relying solely on low-level flight control capabilities is no longer sufficient; unmanned systems urgently need the ability to understand intentions and integrate expert knowledge and experience to make logical decisions.

[0003] Given the strengths of Large Language Models (LLMs) in semantic understanding and logical reasoning, integrating them with unmanned systems to enhance high-level cognition and decision-making capabilities has become an important direction in related technologies. However, the following challenges remain: First, high-level instructions are often difficult for unmanned systems to reliably understand and translate into executable strategies. Related unmanned systems typically use GPS (Global Positioning System) waypoints, trajectory curves, or velocity vectors as inputs and constraints, making them better suited to executing explicit navigation and maneuvering instructions. However, in complex tasks such as search and rescue and reconnaissance, related technologies struggle to translate unstructured high-level intentions into specific task strategies, thus failing to generate stable and executable low-level control instructions. Second, related methods are mostly focused on low-level obstacle avoidance and local path planning based on geometric features such as point clouds, lacking semantic understanding and experience invocation mechanisms at the policy layer. During execution, it is difficult to quickly align the current environmental semantic information with expert experience and complete policy matching, resulting in delayed or incompatible policy selection. This makes it difficult for unmanned systems to adjust their task execution strategies in a timely manner, leading to decreased task completion efficiency.

[0004] Therefore, there is an urgent need for an autonomous control method for unmanned systems based on a large language model that integrates human and machine, enabling unmanned systems to understand task intent, invoke expert experience, and adaptively generate executable actions in unknown environments, thereby improving the autonomous adaptability and task completion efficiency of unmanned systems in unstructured dynamic environments. Summary of the Invention

[0005] The purpose of this application is to provide an autonomous control method and system for human-machine fusion unmanned systems based on a large language model, which can solve the technical problems of traditional unmanned systems such as insufficient understanding of high-level instructions in complex and unstructured environments, lack of strategy selection mechanism supported by expert experience, and poor mission adaptability due to over-reliance on preset waypoints.

[0006] To achieve the above objectives, this application provides the following solution:

[0007] Firstly, this application provides an autonomous control method for human-machine integrated unmanned systems based on a large language model, including:

[0008] Obtain an unstructured high-level semantic instruction sequence; the unstructured high-level semantic instruction sequence includes multiple unstructured high-level semantic instructions;

[0009] Traverse the sequence of unstructured high-level semantic instructions, and for each unstructured high-level semantic instruction, execute the following steps until all tasks corresponding to all unstructured high-level semantic instructions in the sequence have been completed, then terminate control:

[0010] The starting time corresponding to the current unstructured high-level semantic instruction in the unstructured high-level semantic instruction sequence is taken as the current time, and the multi-sensor environmental data at the current time is obtained; the multi-sensor environmental data includes lidar data, visual sensor data, and inertial measurement unit data;

[0011] Construct a local environmental topology map based on the current multi-sensor environmental data.

[0012] The current unstructured high-level semantic instructions and the current local environment topology are used as fusion query vectors to retrieve expert strategy databases, and a preset number of historical expert strategies are selected as the current candidate strategy set; the expert strategy database includes historical fusion query vectors and corresponding historical expert strategies;

[0013] Based on the current unstructured high-level semantic instructions, the current local environment topology, and the current candidate policy set, the optimal policy at the current moment is obtained through a large language model.

[0014] Based on the optimal strategy at the current moment, control the unmanned system to execute the current task corresponding to the current unstructured high-level semantic instruction;

[0015] If the current task is not completed, the multi-sensor environmental data of the next moment is obtained to update the current local environmental topology map, and the above steps are repeated until the current task is completed.

[0016] Secondly, this application provides an autonomous control system for a human-machine integrated unmanned system based on a large language model, including:

[0017] The instruction acquisition module is used to acquire an unstructured high-level semantic instruction sequence; the unstructured high-level semantic instruction sequence includes multiple unstructured high-level semantic instructions;

[0018] The task scheduling module iterates through the sequence of unstructured high-level semantic instructions, executing the following steps for each instruction until all tasks corresponding to the unstructured high-level semantic instructions in the sequence have been completed, at which point control terminates:

[0019] The environment perception module is used to take the start time corresponding to the current unstructured high-level semantic instruction in the unstructured high-level semantic instruction sequence as the current time, and acquire the multi-sensor environment data at the current time; the multi-sensor environment data includes lidar data, visual sensor data, and inertial measurement unit data;

[0020] The topology building module is used to construct a local environmental topology map based on the multi-sensor environmental data at the current moment.

[0021] The strategy retrieval module is used to retrieve the expert strategy database by using the current unstructured high-level semantic instructions and the current local environment topology as fusion query vectors, and select a preset number of historical expert strategies as the current candidate strategy set; the expert strategy database includes historical fusion query vectors and corresponding historical expert strategies;

[0022] The decision generation module is used to obtain the optimal policy at the current moment based on the current unstructured high-level semantic instructions, the current local environment topology graph, and the current candidate policy set, through a large language model.

[0023] The task execution module is used to control the unmanned system to execute the current task corresponding to the current unstructured high-level semantic instruction based on the optimal strategy at the current moment.

[0024] The loop control module is used to obtain multi-sensor environmental data at the next moment to update the current local environmental topology map if the current task is not completed, and repeat the above steps until the current task is completed.

[0025] According to the specific embodiments provided in this application, this application has the following technical effects:

[0026] This application provides an autonomous control method and system for human-machine fusion unmanned systems based on a large language model. By acquiring and traversing unstructured high-level semantic instruction sequences, it solves the technical problem in related technologies where unmanned systems struggle to continuously handle complex multi-stage tasks, achieving autonomous planning and coherent execution of multiple instruction sequences and improving the autonomous decision-making capability of unmanned systems in long-term tasks. By iteratively executing an inner loop of "real-time perception - topology mapping - policy retrieval - policy generation - task execution" for each unstructured high-level semantic instruction until the current task is completed, it solves the technical problem in related technologies where high-level instructions are difficult to reliably understand and convert into executable policies, achieving stable conversion of unstructured high-level semantic instructions into specific task policies and improving the consistency between instruction understanding and policy generation. By constructing a current local environment topology map based on multi-sensor environmental data at the current moment, and using the current unstructured high-level semantic instructions and the current local environment topology map as fusion query vectors to retrieve an expert policy database, it addresses the lack of policy-level language models in related technologies. This paper addresses the technical challenges of semantic understanding and experience retrieval mechanisms, enabling rapid alignment and strategy matching between the current environmental situation and historical expert experience. This reduces response latency in strategy selection and improves task execution efficiency in complex dynamic environments. By generating the optimal strategy for the current moment using a large language model based on current unstructured high-level semantic instructions, the current local environment topology, and the current candidate strategy set, it solves the technical problem in related technologies where unmanned systems struggle to integrate high-level intent and expert experience for adaptive decision-making. This achieves intelligent decision-making through human-machine integration, enhancing the autonomous adaptability and execution reliability of unmanned systems in unknown environments. Furthermore, by updating the current local environment topology in real-time within the inner loop and iteratively executing until the current task is completed and the next instruction is entered into the outer loop, it solves the technical problem in related technologies where unmanned systems over-rely on preset waypoints, leading to poor task adaptability. This achieves an online closed-loop mechanism of simultaneous mapping, retrieval, and decision-making, enabling the unmanned system to dynamically adjust its execution strategy according to the real-time environmental situation, thus improving its autonomous adaptability and task completion efficiency in complex and unknown environments. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is an application environment diagram of an autonomous control method for a human-machine fusion unmanned system based on a large language model, according to one embodiment of this application.

[0029] Figure 2This is a flowchart illustrating an autonomous control method for a human-machine fusion unmanned system based on a large language model, provided as an embodiment of this application.

[0030] Figure 3 This is a flowchart illustrating an autonomous control method for a human-machine fusion unmanned system based on a large language model, provided as another embodiment of this application.

[0031] Figure 4 This is a schematic diagram of the process of constructing an environment topology graph and mapping a strategy based on the FAST-LIVO2 algorithm, which is an embodiment of this application.

[0032] Figure 5 This is a schematic diagram of the retrieval and adaptive decision-making logic of the retrieval enhancement generation strategy during the online task execution phase, as provided in an embodiment of this application.

[0033] Figure 6 This is a visual schematic diagram of a local environment topology structure provided in an embodiment of this application.

[0034] Figure 7 This is a schematic diagram of the functional modules of an autonomous control device for a human-machine fusion unmanned system based on a large language model, provided in an embodiment of this application.

[0035] Figure 8 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0036] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0037] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0038] The autonomous control method for human-machine fusion unmanned systems based on large language models provided in this application can be applied to, for example... Figure 1In the application environment shown, terminal 101 communicates with server 102 via a network. A data storage system can store the data that server 102 needs to process. The data storage system can be set up independently, integrated into server 102, or placed in the cloud or on other servers. Terminal 101 can send unstructured high-level semantic instruction sequences to server 102. After receiving the unstructured high-level semantic instruction sequences, server 102 iterates through the sequences and executes the following steps for each instruction until all tasks corresponding to the unstructured high-level semantic instructions in the sequence have been completed, ending the control: The starting time corresponding to the current unstructured high-level semantic instruction in the sequence is taken as the current time; multi-sensor environmental data at the current time is acquired; the multi-sensor environmental data includes lidar data, visual sensor data, and inertial measurement unit data; based on the current multi-sensor environmental data, a current... The system first obtains a local environment topology map; then, it uses the current unstructured high-level semantic instruction and the current local environment topology map as a fusion query vector to retrieve an expert strategy database, selecting a preset number of historical expert strategies as the current candidate strategy set; the expert strategy database includes the fusion query vector and the corresponding historical expert strategies; based on the current unstructured high-level semantic instruction, the current local environment topology map, and the current candidate strategy set, it obtains the optimal strategy at the current moment through a large language model; based on the optimal strategy at the current moment, it controls the unmanned system to execute the current task corresponding to the current unstructured high-level semantic instruction; if the current task is not completed, it obtains multi-sensor environmental data at the next moment to update the current local environment topology map, repeating the above steps until the current task is completed. The server 102 can feed back the obtained optimal strategy at the current moment to the terminal 101. In addition, in some embodiments, the human-machine fusion unmanned system autonomous control method based on the large language model can also be implemented by the server 102 or the terminal 101 alone. For example, the terminal 101 can directly perform autonomous control processing on the unstructured high-level semantic instruction sequence, or the server 102 can obtain the unstructured high-level semantic instruction sequence from the data storage system and perform autonomous control processing on the unstructured high-level semantic instruction sequence.

[0039] The terminal 101 can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. The server 102 can be implemented using a standalone server or a server cluster consisting of multiple servers, or it can be a cloud server.

[0040] In one exemplary embodiment, such as Figure 2As shown, an autonomous control method for a human-machine integrated unmanned system based on a large language model is provided. This method is executed by a computer device, specifically by a terminal or server alone, or by both a terminal and a server. In this embodiment, the method is applied to... Figure 1 Taking server 102 as an example, the explanation includes the following steps 201 to 208. Wherein:

[0041] Step 201: Obtain an unstructured high-level semantic instruction sequence; the unstructured high-level semantic instruction sequence includes multiple unstructured high-level semantic instructions. Each unstructured high-level semantic instruction corresponds to a task.

[0042] Step 202: Traverse the unstructured high-level semantic instruction sequence, and execute the following steps for each unstructured high-level semantic instruction until all tasks corresponding to all unstructured high-level semantic instructions in the unstructured high-level semantic instruction sequence have been completed, then end control:

[0043] Step 203: Take the start time corresponding to the current unstructured high-level semantic instruction in the unstructured high-level semantic instruction sequence as the current time, and obtain the multi-sensor environment data at the current time; the multi-sensor environment data includes lidar data, visual sensor data and inertial measurement unit data.

[0044] Step 204: Construct the current local environment topology map based on the multi-sensor environmental data at the current moment.

[0045] Step 205: Use the current unstructured high-level semantic instructions and the current local environment topology as fusion query vectors to retrieve expert strategy databases, and select a preset number of historical expert strategies as the current candidate strategy set; the expert strategy database includes historical fusion query vectors and corresponding historical expert strategies.

[0046] Step 206: Based on the current unstructured high-level semantic instructions, the current local environment topology graph, and the current candidate policy set, the optimal policy at the current moment is obtained through the large language model.

[0047] Step 207: Based on the optimal strategy at the current moment, control the unmanned system to execute the current task corresponding to the current unstructured high-level semantic instruction.

[0048] Step 208: If the current task is not completed, obtain the multi-sensor environmental data of the next moment to update the current local environmental topology map, and repeat the above steps until the current task is completed.

[0049] By implementing steps 201 to 208 above, this application can solve the technical problems of traditional unmanned systems, such as insufficient understanding of high-level instructions in complex and unstructured environments, lack of strategy selection mechanisms supported by expert experience, and poor mission adaptability due to over-reliance on preset waypoints. By constructing an expert policy database indexed by the environmental topology map, and combining retrieval-enhanced generation, it achieves rapid matching and policy output between real-time environmental topology and historical experience, thereby realizing the effective transformation of high-level intentions into executable action primitive sequences for the unmanned system, and improving mission execution efficiency in complex dynamic environments.

[0050] In another exemplary embodiment of this application, step 204 specifically includes:

[0051] Based on the multi-sensor environmental data at the current moment, state estimation and local mapping operations are performed through a preset fusion mapping algorithm to obtain the pose information of the unmanned system and the local environmental point cloud map at the current moment; the preset fusion mapping algorithm includes the FAST-LIVO2 algorithm or a similar multi-sensor fusion mapping algorithm.

[0052] Based on the current pose information of the unmanned system and the local environment point cloud map, construct the current local environment topology map.

[0053] As an optional implementation, based on the multi-sensor environmental data at the current moment, state estimation and local mapping operations are performed using the FAST-LIVO2 algorithm to obtain the pose information of the unmanned system and the local environmental point cloud map at the current moment, specifically including:

[0054] Step 204-1: Multi-sensor hard synchronization and scan reconstruction.

[0055] The system scans and reassembles data from the LiDAR, vision sensor, and inertial measurement unit (IMU) to achieve hard synchronization between the LiDAR, IMU, and vision sensor. This step ensures precise alignment of LiDAR point cloud scanning, image frame acquisition, and IMU sampling on the time axis through a unified clock reference, eliminating timestamp deviations caused by asynchronous sensor acquisition and providing a time-consistent data foundation for subsequent tightly coupled state estimation.

[0056] Step 204-2: Motion distortion correction based on IMU pre-integration.

[0057] Motion distortion correction is performed on the lidar point cloud using pre-integration of the inertial measurement unit (IMU) to obtain corrected point cloud data. Since the unmanned system undergoes continuous motion during lidar scanning, the original point cloud will be distorted. This step estimates the motion trajectory during point cloud acquisition by integrating high-frequency IMU data, compensating for the acquisition pose of each lidar point, and restoring the true spatial distribution of the point cloud.

[0058] Step 204-3: Tightly coupled state estimation.

[0059] Tightly coupled state estimation is performed based on a sequentially updated error state iterative Kalman filter to calculate the pose information of the unmanned system at the current moment. This process includes two sub-steps:

[0060] Step 204-3-1: LiDAR geometric residual update: Project the corrected LiDAR point cloud onto the local map, calculate the geometric residual from the point to the plane, update the state estimate through a Kalman filter, and obtain the preliminary optimized pose of the unmanned system.

[0061] Step 204-3-2: Visual photometric residual optimization: Based on the LiDAR update, the photometric residual from the direct visual method is introduced, and the state estimation is further optimized by using image grayscale consistency constraints to improve the accuracy and robustness of pose calculation.

[0062] Step 204-4: Adaptive voxel map construction.

[0063] The optimized point cloud data and image data are fused into an adaptive voxel map managed by a hash table and an octree to generate a local environment point cloud map containing texture information. This map uses a hash table for fast spatial indexing and an octree structure for multi-resolution voxel management. The voxel resolution is adaptively adjusted according to the environmental complexity, ensuring map accuracy while controlling computational overhead, and simultaneously storing point cloud geometric information and image texture information.

[0064] Step 204-5: Structured generation of the environment topology graph.

[0065] The local environment point cloud map is subjected to semantic segmentation and topological analysis to extract environmental entity nodes and spatial relationships between entities, and a current local environment topology map containing a multimodal feature matrix is ​​constructed. Specifically, key entities in the environment (such as obstacles, target areas, structural boundaries, etc.) are identified through semantic segmentation, topological connections between entities are constructed based on spatial adjacency relationships, and the geometric position information and semantic labels of each entity are fused to form a multimodal feature matrix. Finally, a formally defined triple graph structure is output as the current local environment topology map for subsequent strategy retrieval and matching.

[0066] In addition to the FAST-LIVO2 algorithm, this application can also use similar multi-sensor fusion mapping algorithms. Although there are differences in the specific implementation details, they all solve high-precision pose and construct point cloud maps with texture information through a tight coupling of laser-inertial-vision, and can achieve the same mapping function as the FAST-LIVO2 algorithm.

[0067] In another exemplary embodiment of this application, step 205 specifically includes:

[0068] The current unstructured high-level semantic instruction and the current local environment topology graph are input into the cross-modal fusion representation model to obtain the fused query vector corresponding to the current unstructured high-level semantic instruction and the current local environment topology graph. The cross-modal fusion representation model includes a text encoder, a graph encoder, and a modal fusion layer. The text encoder is used to extract the instruction semantic features of the unstructured high-level semantic instruction. The graph encoder is used to extract the topology graph structure features of the local environment topology graph. The modal fusion layer is used to fuse the instruction semantic features and the corresponding topology graph structure features to obtain the fused query vector.

[0069] Calculate the cosine similarity between the fusion query vector corresponding to the current unstructured high-level semantic instruction and the current local environment topology graph and all historical fusion query vectors in the expert strategy database.

[0070] Sort all cosine similarities from highest to lowest to obtain a similarity ranking set.

[0071] Select the historical expert strategies corresponding to the top preset number of cosine similarities in the similarity ranking set as the current candidate strategy set.

[0072] As an optional implementation, step 205 fuses the current unstructured high-level semantic instructions with the local environment topology graph using a cross-modal encoder to generate a unified fusion embedding vector, thereby achieving end-to-end similarity retrieval in the vector space. The specific implementation steps are as follows:

[0073] Step 205-1: Construct a cross-modal fusion representation model.

[0074] A multimodal encoder architecture similar to CLIP (Contrastive Language-Image Pre-training) is adopted, with a text encoder designed to extract semantic features of instructions and a graph encoder designed to extract structural features of the topological graph. The features of the two modalities are mapped to the same fusion embedding space. Through contrastive learning, pre-training or fine-tuning is performed on a corpus containing a large amount of (instruction, topological graph) pairing data, so that the vector distances of matching instruction-graph pairs in this space are close, while the distances of mismatched pairs are far apart.

[0075] Step 2: Generate the current fusion query vector.

[0076] The current unstructured high-level semantic instructions and the current local environment topology map The cross-modal encoder in the modal fusion layer is input to obtain their respective modal features, and then the fused query vector is obtained through the modal fusion layer. :

[0077] .

[0078] in, This refers to a cross-modal encoder, used to map data from two different modalities—text instructions and topology graphs—to the same fusion embedding space. express 3D real vector; To integrate the dimensions of the embedded space.

[0079] Step 3: Calculate the fusion similarity with historical strategies.

[0080] The expert strategy database stores the fusion query vector corresponding to each historical strategy. (Pre-calculated and stored using the same cross-modal encoder during data entry). Cosine similarity is used as the metric. With each Similarity between them:

[0081] .

[0082] in, Represents the current fusion query vector Fusion query vector with historical data at time t in the expert strategy database Cosine similarity; The norm of a vector.

[0083] Step 4: Select a set of candidate strategies.

[0084] The obtained cosine similarity scores are sorted in descending order, and the K historical expert strategies corresponding to the highest cosine similarity scores are selected as the current candidate strategy set for the current context, which is used for subsequent decision fusion or adaptive generation.

[0085] In summary, cross-modal fusion representation directly captures the deep relationship between instruction semantics and environmental structure, avoiding the information loss that may be introduced when calculating similarity separately and then fusing, resulting in higher retrieval accuracy. However, model fine-tuning is required for specific task domains to achieve the best results.

[0086] In another exemplary embodiment of this application, after obtaining the optimal strategy at the current moment, the method further includes:

[0087] The optimal strategy at the current moment is subjected to strategy constraint verification; the strategy constraint verification includes security constraints and dynamic constraints.

[0088] If all constraints in the policy constraint check are satisfied, the policy constraint check passes.

[0089] If any constraint in the policy constraint check is not met, the policy constraint check fails.

[0090] In another exemplary embodiment of this application, step 207 specifically includes:

[0091] Map the optimal strategy at the current moment to the preset action primitive database to obtain the current underlying executable action primitive.

[0092] The current low-level executable action primitives are converted into low-level control commands; the low-level control commands include speed, heading, attitude, and altitude.

[0093] The low-level control commands are input into the low-level controller of the unmanned system to control the unmanned system to execute the current task corresponding to the current unstructured high-level semantic command.

[0094] In another exemplary embodiment of this application, the security constraints specifically include:

[0095] The optimal strategy at the current moment is projected onto the current local environment topology to obtain the predicted path of the unmanned system.

[0096] Based on the predicted path and the current local environment topology, the minimum distance between the unmanned system and each obstacle is calculated.

[0097] If the minimum distance between the unmanned system and each obstacle is greater than or equal to the preset safety threshold, then the safety constraint is satisfied.

[0098] If the minimum distance between the unmanned system and any obstacle is less than a preset safety threshold, the safety constraint is not met.

[0099] Dynamic constraints, specifically including:

[0100] Based on the optimal strategy at the current moment, extract the rate of change of velocity, acceleration, and yaw rate.

[0101] If the speed is greater than the preset maximum permissible speed, or the acceleration is greater than the preset maximum permissible acceleration, or the rate of change of yaw rate is greater than the preset maximum permissible rate of change of yaw rate, then the dynamic constraints are not met.

[0102] If the speed is less than or equal to the preset maximum permissible speed, the acceleration is less than or equal to the preset maximum permissible acceleration, and the rate of change of yaw rate is less than or equal to the preset maximum permissible rate of change of yaw rate, then the dynamic constraints are satisfied.

[0103] In another exemplary embodiment of this application, if the cosine similarity of all historical expert strategies in the current candidate strategy set is lower than a preset similarity threshold, or if the strategy constraint verification fails, a security mechanism is triggered; the security mechanism includes switching to an autonomous navigation mode based on geometric obstacle avoidance to control the unmanned system or controlling the unmanned system to return to a preset safe point.

[0104] In another exemplary embodiment of this application, during the process of inputting the low-level control command into the low-level controller of the unmanned system and controlling the unmanned system to execute the current task corresponding to the current unstructured high-level semantic command, the low-level control command is locally adjusted through the autonomous obstacle avoidance algorithm of the lidar to achieve real-time obstacle avoidance.

[0105] In another exemplary embodiment of this application, a current local environment topology map is constructed based on the current pose information of the unmanned system and the local environment point cloud map, specifically including:

[0106] The geometric location information and semantic labels of key entities in the environment are extracted by point cloud semantic segmentation algorithm, and the spatial connection relationship between key entities in the environment is constructed by topological relationship analysis algorithm to obtain the current local environment topology map; the key entities include obstacles, target objects and unmanned systems.

[0107] The current local environment topology is as follows:

[0108] .

[0109] .

[0110] in, Indicates the current time The current local environment topology; Indicates the current time The set of key entity nodes in the environment; Indicates the current time Spatial topological connections between entities; Indicates the current time The multimodal feature matrix; express The multimodal feature matrix of the i-th node; This represents a vector concatenation operation; express The geometric position information of the i-th node; express The semantic label of the i-th node.

[0111] The following example illustrates this application using a specific autonomous control process of a human-machine integrated unmanned system based on a large language model.

[0112] In one exemplary embodiment, this application constructs an expert policy database and an action primitive database based on multi-sensor data and high-fidelity simulation data from unmanned systems, and establishes an index structure for retrieving enhanced generation. For example... Figure 3As shown, a human-machine fusion autonomous control method for unmanned systems based on a large language model is provided. This method includes: In the expert policy database construction phase, based on multi-sensor environmental data, a pre-defined fusion mapping algorithm is used to generate a local environmental topology map. The local environmental topology map is semantically aligned with the expert's high-level policies to establish an "environmental topology—expert policy" mapping relationship, forming a policy database for retrieval-enhanced generation. Simultaneously, an action primitive database is constructed to support underlying execution. In the execution phase, after receiving unstructured high-level semantic instructions, the unmanned system constructs the current local environmental topology map in real time, which, together with the unstructured high-level semantics, constitutes a retrieval query. Through a retrieval-enhanced generation mechanism, similar topologies and their corresponding policies are quickly matched in the policy database. Then, the large language model integrates the unstructured high-level semantic instructions, real-time topology, and retrieval policies to output a task-level policy, which is mapped into a sequence of control commands such as speed, heading, attitude, and altitude to drive flight execution, forming a closed-loop process of "perception mapping—policy retrieval—decision generation—action execution," thereby improving the unmanned system's responsiveness, task adaptability, and execution reliability in complex and unknown environments. Specifically:

[0113] Step 1: Formalization of unstructured high-level semantic instruction intents and construction of a policy library.

[0114] By utilizing multi-source data collected by unmanned systems in historical missions and high-fidelity simulation training, a knowledge base is constructed that includes environmental situation characteristics and corresponding expert strategies. The specific process includes:

[0115] Local environment topology graph construction: The FAST-LIVO2 (Fast, Direct LiDAR-Inertial-VisualOdometry) algorithm (or similar multi-sensor fusion mapping algorithms) is used to process data from LiDAR, Inertial Measurement Unit (IMU), and visual sensors. Simultaneously, high-precision state estimation and mapping are performed, and semantic segmentation and topology analysis generate an environment topology graph structure containing both geometric and semantic information. To quantify the deep semantic situational awareness of the environment, this graph structure is formally defined as a triple containing nodes, edges, and a multimodal feature matrix.

[0116] .

[0117] in, Represents the local environment topology at time t; This represents the set of key entity nodes in the environment at time t (such as obstacles, target objects, target areas, etc.). Indicates the spatial topological connectivity (such as adjacent, contained) between entities at time t. Let be the multimodal feature matrix at time t, the calculation of which incorporates the geometric position information of the entity. With high-dimensional semantic tags , represented as:

[0118] .

[0119] in, express The multimodal feature matrix of the i-th node; This represents a vector concatenation operation; express The geometric position information of the i-th node; express The semantic label of the i-th node.

[0120] Expert policy database construction: The historical local environment topology graph structure generated at each sampling time and the corresponding unstructured high-level semantic instructions at each sampling time are input into the cross-modal fusion representation model to obtain the fusion query vector, and finally construct the expert policy database. It can be represented as:

[0121] .

[0122] in, This represents the historical fusion query vector at time t in the expert strategy database; This represents the historical expert policies at time t in the expert policy database, including the macro-level task intent.

[0123] In database storage and retrieval mechanisms This indicates that it is used as a key. It is stored as a value. In actual operation, the system inputs the current unstructured high-level semantic instructions and the local environment topology map into the cross-modal encoder in the modal fusion layer to obtain the current fusion query vector. By calculating its similarity with each key in the database, it supports subsequent RAG (Retrieval-Augmented Generation) retrieval, thereby extracting the most matching historical expert strategy.

[0124] Action Primitive Database Establishment: A database of executable action primitives for the unmanned system's underlying layer is constructed, decomposing the mission strategy into standardized sequences of control commands. This database stores and manages the executable action primitives for the unmanned system's underlying layer. Each action primitive corresponds to a set of control command parameters or control sequences, including speed, heading, attitude, and altitude control parameters.

[0125] Step 2: Receive high-level semantic instructions and build the real-time environment.

[0126] Unmanned systems receive unstructured high-level semantic commands. (For example, searching for red buildings ahead), and during the task execution, a local point cloud map of the current environment and state estimation results are constructed in real time, while a real-time environment graph structure diagram is generated. .

[0127] Step 3: Create the current local environment topology graph structure. With unstructured high-level semantic instructions As a fusion query vector representation, a retrieval enhancement generation task is performed in the expert strategy database:

[0128] Similarity retrieval: This involves retrieving the current unstructured high-level semantic instructions. and the current local environment topology map The cross-modal encoder in the modal fusion layer is input to obtain the modal features of each, and the current fusion query vector is obtained through the modal fusion layer. The cosine similarity between the current fusion query vector and all historical fusion query vectors in the expert policy database is calculated, thereby retrieving the Top-K most similar environmental topology nodes from the expert policy database.

[0129] Policy generation: Utilizing a large language model as the core of reasoning, it integrates current unstructured high-level semantic instructions. Current local environment topology structure And the retrieved set of historical expert strategies (current candidate strategy set). Generate the optimal strategy at the current moment. This process simulates human decision-making intuition in similar scenarios, and can be represented as:

[0130] .

[0131] in, This represents the structured action execution sequence generated by the large language model, which will serve as the input for subsequent action sequence mapping and execution steps. S represents the expert policy database.

[0132] As an optional implementation method, to enable the large language model to possess the following core capabilities: (1) understanding cross-modal semantics of unstructured high-level semantic instructions and environmental topology graphs; (2) fusing retrieved historical expert strategies for comprehensive reasoning; (3) outputting structured multi-step action execution sequences and calling underlying control tools; this application adopts a supervised fine-tuning method based on human expert teaching data to perform domain-adaptive training on the large language model. The specific training process is as follows:

[0133] 1. Training data construction.

[0134] Construct a supervised fine-tuning dataset containing a large number of input-output pairings. The input portion of each training sample includes:

[0135] ① Unstructured high-level semantic instructions: Natural language instructions issued by human experts during task execution, such as "Search the red building ahead" or "Go around the obstacle and reach the target point".

[0136] ② Local environment topology graph: Convert the local environment topology graph into a structured text description or graph embedding representation that the model can understand, including key entity nodes (obstacles, target areas, channels, etc.) and their spatial topological relationships.

[0137] ③ Retrieved historical expert strategies: As contextual examples, provide historical expert strategies similar to the current scenario to enhance the model's ability to utilize expert experience.

[0138] The output of each training sample is:

[0139] The sequence of actions actually performed by human experts in this scenario is represented in a structured format, such as a JSON dictionary, containing action types (such as "forward", "hover", "turn left", "climb", "scan", etc.) and their corresponding execution parameters (such as speed value, heading angle, duration, target position, etc.).

[0140] 2. Cross-modal semantic alignment.

[0141] During fine-tuning, the large language model gradually builds cross-modal semantic alignment capabilities by learning the association between the environmental topology description in the input and high-level semantic instructions. Specifically, the model learns to implicitly align key entities extracted from the environmental topology with semantic concepts in the instructions through an attention mechanism. For example, it aligns "red building nodes" in the topology with the concept of "red building" in the instructions, and "channel nodes" with the intention of "detour," thereby understanding the mapping relationship between the current situation and the task objective.

[0142] 3. Structured output training.

[0143] The fine-tuning phase employs a supervised learning objective of next-label prediction, meaning the model progressively generates a structured output sequence based on the input sequence. Through training with a large amount of expert teaching data, the model learns to decompose macro-level tactical intentions into multi-step action invocation commands. An example output format is as follows:

[0144] 4. Enhanced tool access capabilities.

[0145] To enable the model to interact with the underlying flight control system, a tool call format was introduced into the fine-tuning data. The model was trained to identify the underlying control tools to be invoked using a predefined function call format when outputting action sequences. During the inference phase, the system parses the tool call instructions output by the model, maps them to the corresponding underlying control instructions in the action primitive database, and drives the unmanned system to execute specific flight actions.

[0146] 5. Training Implementation Methods.

[0147] The fine-tuning process can be implemented in any of the following ways:

[0148] ① Full parameter fine-tuning: Update all parameters of the basic large language model to make the model fully adaptable to the decision-making task of the unmanned system and obtain the best domain adaptability.

[0149] Efficient parameter fine-tuning: Employing techniques such as LoRA (Low-Rank Adaptation) or Adapter, only a small number of additional parameters are updated, achieving domain adaptation while maintaining the general capabilities of the basic model, significantly reducing training costs and deployment resource consumption, and avoiding catastrophic forgetting problems.

[0150] 6. Training effect.

[0151] After the aforementioned supervised fine-tuning, the large language model receives current unstructured high-level semantic instructions, real-time environment topology descriptions, and retrieved historical expert policies during the inference phase, enabling it to generate a structured multi-step action execution sequence. This sequence is then parsed by the system and mapped line by line to the underlying control instructions in the action primitive database, realizing an interpretable and executable end-to-end decision-making link from high-level intent to low-level control.

[0152] Step 4: Action sequence mapping and execution.

[0153] The optimal strategy generated by the large language model at the current moment Mapping to the action primitive database Generate the control input sequence at time t (currently executable low-level action primitives). This information is then converted into low-level control commands (including speed, heading, attitude, or altitude control parameters) to achieve human-like maneuverability. The mapping relationship between policy and control input can be expressed as:

[0154] .

[0155] in, To translate high-level strategies into mapping functions for the control of low-level unmanned systems, This provides feedback on the current status.

[0156] As an optional implementation, after obtaining the optimal strategy at the current moment, the following steps are also included:

[0157] The optimal strategy at the current moment is subjected to strategy constraint verification, which includes security constraints and dynamic constraints. The verification process follows a veto principle, that is, if any constraint in the strategy constraint verification is not satisfied, the system determines that the current strategy has insufficient confidence or is risky, and the strategy constraint verification fails.

[0158] If the cosine similarity of all historical expert strategies in the current candidate strategy set is lower than a preset similarity threshold, or if the strategy constraint verification fails, a safety assurance mechanism is automatically triggered. This safety assurance mechanism includes switching to a geometry-based obstacle avoidance-based autonomous navigation mode to control the unmanned system or controlling the unmanned system to return to a preset safety point to ensure the system does not collide, thus guaranteeing flight safety. The system continuously cycles through the above perception, retrieval, and decision-making steps during execution until the mission is completed.

[0159] Implementation Method 1: Historical Strategy Knowledge Base Construction Method (Offline Stage)

[0160] The core of this implementation lies in constructing an expert strategy database "based on expert experience and indexed by environmental topology maps," providing unmanned systems with searchable and recallable strategy memories. The process is as follows: Figures 3-4 As shown.

[0161] Step S1-1: Acquisition and preprocessing of multi-source flight data.

[0162] The system acquires high-fidelity mission data from a preset data source, including: operational logs of the unmanned system in historical real-world missions, and operational demonstration data from experts in simulated environments (such as AirSim and Gazebo). The data undergoes timestamp alignment, anomaly removal, and noise filtering to obtain standardized multimodal sensor sequences and corresponding control commands / operation records, providing input for subsequent mapping and strategy annotation.

[0163] Step S1-2: Construction and feature extraction of the environmental topology map.

[0164] like Figure 4 As shown, for each frame of data acquired in step S1-1, the system runs the FAST-LIVO2 algorithm (or a similar multi-sensor fusion mapping algorithm) to perform state estimation and local mapping, and further generates an environmental situation map structure (local environmental topology map) through semantic segmentation and spatial topology analysis. This graph structure is formally defined as a set of nodes. Edge set With multimodal feature matrix The triplet. For example... Figure 6As shown, the environment graph structure is presented in the form of a topological network, where nodes (e.g., V1, V2, V3) represent key entities in the environment (e.g., drones, walls, windows), and edges represent spatial semantic relationships between entities (e.g., "distance," "physical connection," "line-of-sight access"). Each node is associated with a multimodal feature matrix, such as... Figure 4 As shown, Deeply integrates the geometric position information of entities With high-dimensional semantic tags This step ensures that the environment is not merely a collection of point clouds, but a topological network that can be understood by the drone system.

[0165] Step S1-3: Strategy mapping and database generation.

[0166] A cross-modal fusion representation model is constructed by inputting the local environment topology map and unstructured high-level semantic instructions generated in steps S1-2 into the model to obtain a fusion query vector. The high-level strategy intent of experts at the same time is then associated and labeled with the fusion query vector to form a mapping pair. The final expert policy database is then constructed. It consists of the union of mapping pairs generated at all moments in N historical tasks. Simultaneously, an action primitive database is constructed, decomposing high-level policies into low-level executable action sequences, such as control primitives for heading adjustment, speed control, and attitude maintenance, to support the implementation of policy decisions as control commands during the online phase.

[0167] Implementation Method 2: Human-Machine Fusion Online Decision Execution Method Based on Retrieval Enhancement Generation (Online Phase).

[0168] Implementation method two is used to realize an online closed-loop decision-making process of "real-time topology construction—historical policy retrieval—policy generation—action execution" based on the policy database and action primitive database constructed in implementation method one during actual task execution. The process is as follows: Figure 3 and Figure 5 As shown.

[0169] Step S2-1: Real-time perception and graph structure generation.

[0170] After receiving unstructured high-level semantic commands, such as "search for red buildings ahead," the unmanned system completes real-time localization and mapping during flight, and dynamically constructs a real-time environmental situation map structure representing the current environmental topology and semantic relationships. Its data structure and Figure 6Consistency is ensured to maintain consistency with the feature representation during the offline phase. Specifically, taking the command "Search for the red building ahead" as an example, the expert strategy database pre-stores similar historical scenarios, such as data A (command: "Reconnoiter the red target"; topology: large obstacles exist in the open area) and data B (command: "Approach the building"; topology: includes corridors and corners). During execution, the unmanned system extracts the features of the current environmental situation map and performs a joint search with the command: the system calculates that the similarity between the current state and data A is 0.91, and the similarity with data B is 0.75.

[0171] Subsequently, the system selects the top-2 policies with the highest similarity to input into the large language model, and generates the optimal policy sequence for the current time step through inference. Specifically, this includes: 1. Maintaining a 3m offset distance, flying towards the center area of ​​the red pixel; 2. Performing a "bow-shaped" sweep across the building facade. This strategy is further translated into underlying control commands. This is manifested as the desired velocity vector v=[1.5,0,0.2]m / s and the pitch angle increment.

[0172] The above perception, retrieval, and generation steps are executed cyclically until the airborne vision sensor identifies detailed features of the red building for 10 consecutive frames with a confidence level greater than 0.98, or the node marked as "red building" in the environmental topology map has been completely covered by the sweep path. At this point, the system determines that the "search for red buildings ahead" command has been executed and switches to hover mode.

[0173] Step S2-2: Search for matches based on the strategy generated by search enhancement.

[0174] The system will display the current local environment topology structure. With unstructured high-level semantic instructions As a query vector, in the expert strategy database Cosine similarity matching is performed to retrieve the Top-K most similar historical scene nodes. The historical strategies corresponding to these retrieved nodes are used as retrieval enhancement information for subsequent strategy generation.

[0175] Step S2-3: Strategy integration and action execution.

[0176] The system incorporates a large language model (LLM) as its inference core, fusing retrieved historical expert strategies, current unstructured high-level semantic instructions, and current local environment topology information. LLM inference generates the optimal task decision for the current moment. It is then mapped to the action primitive database, outputting an executable sequence of low-level control instructions, which are then driven by the unmanned system control and execution module to complete the actions.

[0177] The optimal strategy for the current moment is obtained, followed by strategy constraint verification, which includes safety constraints and dynamic constraints. When the fusion similarity of all historical expert strategies in the current candidate strategy set is lower than a preset similarity threshold, or the strategy constraint verification fails, the system switches to a geometric obstacle avoidance-based safety mode or returns to a preset safety point to ensure the safe operation of the unmanned system. The system executes steps S2-1 to S2-3 cyclically according to a preset period until the task is completed or the termination condition is met.

[0178] By implementing the aforementioned offline database construction and online retrieval approach to enhance decision-making, this application achieves structured storage and online retrieval of historical expert strategy experience. During the task execution phase, the system can quickly retrieve and match similar strategies from the database based on a real-time environmental situation map, and generate task-level decisions under the inference of a large language model, mapping them to action primitives for execution. This improves the real-time performance, adaptability, and execution reliability of the unmanned system's autonomous decision-making in complex and unknown environments.

[0179] Based on the same inventive concept, this application also provides an autonomous control system for a human-machine fusion unmanned system based on a large language model, used to implement the aforementioned autonomous control method for a human-machine fusion unmanned system based on a large language model. The solution provided by this system is similar to the solution described in the above method. Therefore, the specific limitations of one or more embodiments of the autonomous control system for a human-machine fusion unmanned system based on a large language model provided below can be found in the limitations of the autonomous control method for a human-machine fusion unmanned system based on a large language model described above, and will not be repeated here.

[0180] In one exemplary embodiment, such as Figure 7 As shown, an autonomous control system for a human-machine integrated unmanned system based on a large language model is provided, including:

[0181] The instruction acquisition module 301 is used to acquire an unstructured high-level semantic instruction sequence; the unstructured high-level semantic instruction sequence includes multiple unstructured high-level semantic instructions.

[0182] The task scheduling module 302 is used to traverse the unstructured high-level semantic instruction sequence and execute the following steps for each unstructured high-level semantic instruction until all tasks corresponding to all unstructured high-level semantic instructions in the unstructured high-level semantic instruction sequence have been completed, and then the control ends.

[0183] The environment perception module 303 is used to take the start time corresponding to the current unstructured high-level semantic instruction in the unstructured high-level semantic instruction sequence as the current time and acquire the multi-sensor environment data at the current time; the multi-sensor environment data includes lidar data, visual sensor data and inertial measurement unit data.

[0184] Topology building module 304 is used to build a current local environment topology map based on the multi-sensor environmental data at the current moment.

[0185] The strategy retrieval module 305 is used to retrieve the expert strategy database by using the current unstructured high-level semantic instructions and the current local environment topology as fusion query vectors, and select a preset number of historical expert strategies as the current candidate strategy set; the expert strategy database includes historical fusion query vectors and corresponding historical expert strategies.

[0186] The decision generation module 306 is used to obtain the optimal policy at the current moment based on the current unstructured high-level semantic instructions, the current local environment topology graph, and the current candidate policy set through a large language model.

[0187] The task execution module 307 is used to control the unmanned system to execute the current task corresponding to the current unstructured high-level semantic instruction based on the optimal strategy at the current moment.

[0188] The loop control module 308 is used to obtain the multi-sensor environmental data of the next moment to update the current local environmental topology map if the current task is not completed, and repeat the above steps until the current task is completed.

[0189] In one exemplary embodiment, this application provides an autonomous task execution system for unmanned aerial vehicles (UAVs) based on retrieval-enhanced generation and human-machine policy fusion. This system, upon receiving instructions from a higher level, retrieves historical expert policies based on real-time environmental situation and generates executable action sequences, thereby enabling autonomous task execution by the UAV in unknown environments. The system includes:

[0190] Multimodal perception and mapping module: Equipped with lidar, camera and IMU sensors, and built-in mapping algorithm unit, it is used to perform real-time state estimation, high-precision mapping and environmental topology generation during the flight of unmanned systems.

[0191] The knowledge base management module stores and manages the expert strategy database and the action primitive database. The expert strategy database stores the mapping relationship between historical fusion query vectors and historical expert strategies, while the action primitive database stores standardized underlying executable action primitives.

[0192] The RAG decision-making and reasoning module includes a retrieval unit and a large language model unit. The retrieval unit is used to retrieve similar historical situation samples and their associated strategies from the policy database based on the current environment topology. The large language model unit is used to fuse user commands with retrieved historical expert strategies to generate adaptive task-level policy decisions (during execution, real-time local adjustments are made through the autonomous obstacle avoidance algorithm of the LiDAR).

[0193] Flight control execution module: It is used to translate high-level decisions into a sequence of executable actions and drive the unmanned system's actuators to perform specific flight actions through the low-level controller.

[0194] In summary, this application has the following beneficial effects:

[0195] 1. Achieved online policy retrieval and decision selection based on environmental topology graph: This application completes high-precision mapping during the execution of unmanned system tasks and dynamically constructs an environmental topology graph structure representing environmental topology and semantic relationships; using real-time environmental topology graph and high-level instructions as fusion queries, a retrieval enhancement generation mechanism is used to retrieve and match similar historical situations and their expert policies in the policy database, thereby realizing online selection and output of task-level policies, reducing decision computation overhead and response latency.

[0196] 2. Achieved a closed-loop implementation of human-machine fusion strategy from instruction understanding to action execution: This application constructs a two-layer mapping mechanism of "expert strategy database - action primitive database", and uses a large language model to integrate high-level instructions and historical expert strategies under the constraints of retrieval results to generate interpretable task-level strategy decisions, which are further mapped to the execution of underlying action sequences, thereby improving the consistency and controllability between instruction understanding, strategy generation and action execution.

[0197] 3. Improved adaptability and reliability of mission execution in complex and unknown environments: Compared with traditional fixed waypoint planning, this application supports an online closed-loop mechanism for unmanned systems to build maps, search, and make decisions during flight; when the real-time fused query vector is similar to the historical fused query vector in the expert strategy database, the system can reuse the corresponding expert strategy and make action adjustments, thereby improving the autonomous adaptability and execution safety in complex dynamic scenarios.

[0198] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 8As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data for autonomous control processing of the unmanned system. The I / O interfaces are used for information exchange between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a human-machine fusion autonomous control method for an unmanned system based on a large language model.

[0199] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer equipment to which the present application is applied. Specific computer equipment may include, for example, [the following is a list of possible additional structures]. Figure 8 The embodiments show more or fewer components, combinations of certain components, or different component arrangements. In one exemplary embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, which the processor executes to implement the steps in the above-described method embodiments.

[0200] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0201] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0202] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0203] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0204] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0205] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0206] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An autonomous control method for a human-machine integrated unmanned system based on a large language model, characterized in that, include: Obtain an unstructured high-level semantic instruction sequence; the unstructured high-level semantic instruction sequence includes multiple unstructured high-level semantic instructions; Traverse the sequence of unstructured high-level semantic instructions, and for each unstructured high-level semantic instruction, execute the following steps until all tasks corresponding to all unstructured high-level semantic instructions in the sequence have been completed, then terminate control: The starting time corresponding to the current unstructured high-level semantic instruction in the unstructured high-level semantic instruction sequence is taken as the current time, and the multi-sensor environmental data at the current time is obtained; the multi-sensor environmental data includes lidar data, visual sensor data, and inertial measurement unit data; Based on the multi-sensor environmental data at the current moment, a local environmental topology map is constructed. Specifically, this includes: based on the multi-sensor environmental data at the current moment, state estimation and local mapping operations are performed through a preset fusion mapping algorithm to obtain the pose information of the unmanned system and the local environmental point cloud map at the current moment; the preset fusion mapping algorithm includes the FAST-LIVO2 algorithm or a similar multi-sensor fusion mapping algorithm. Based on the current pose information of the unmanned system and the local environment point cloud map, a topology map of the current local environment is constructed, specifically including: The geometric location information and semantic labels of key entities in the environment are extracted by point cloud semantic segmentation algorithm, and the spatial connection relationship between key entities in the environment is constructed by topological relationship analysis algorithm to obtain the current local environment topology map; the key entities include obstacles, target objects and unmanned systems; The current local environment topology is as follows: ； ； in, Indicates the current time Local environment topology map; Indicates the current time The set of key entity nodes in the environment; Indicates the current time Spatial topological connections between entities; Indicates the current time The multimodal feature matrix; express The multimodal feature matrix of the i-th node; This represents a vector concatenation operation; express The geometric position information of the i-th node; express The semantic label of the i-th node in the dataset; The current unstructured high-level semantic instructions and the current local environment topology are used as fusion query vectors to retrieve expert strategy databases, and a preset number of historical expert strategies are selected as the current candidate strategy set; the expert strategy database includes historical fusion query vectors and corresponding historical expert strategies; Based on the current unstructured high-level semantic instructions, the current local environment topology, and the current candidate policy set, the optimal policy at the current moment is obtained through a large language model. Based on the optimal strategy at the current moment, control the unmanned system to execute the current task corresponding to the current unstructured high-level semantic instruction; If the current task is not completed, the multi-sensor environmental data of the next moment is obtained to update the current local environmental topology map, and the above steps are repeated until the current task is completed.

2. The autonomous control method for a human-machine integrated unmanned system based on a large language model according to claim 1, characterized in that, The current unstructured high-level semantic instructions and the current local environment topology are used as fusion query vectors to retrieve expert strategy databases. A preset number of historical expert strategies are selected as the current candidate strategy set, specifically including: The current unstructured high-level semantic instruction and the current local environment topology graph are input into the cross-modal fusion representation model to obtain the fused query vector corresponding to the current unstructured high-level semantic instruction and the current local environment topology graph. The cross-modal fusion representation model includes a text encoder, a graph encoder, and a modal fusion layer. The text encoder is used to extract the instruction semantic features of the unstructured high-level semantic instruction. The graph encoder is used to extract the topology graph structure features of the local environment topology graph. The modal fusion layer is used to fuse the instruction semantic features and the corresponding topology graph structure features to obtain the fused query vector. Calculate the cosine similarity between the fusion query vector corresponding to the current unstructured high-level semantic instruction and the current local environment topology graph and all historical fusion query vectors in the expert strategy database. Sort all cosine similarities from highest to lowest to obtain a similarity ranking set; Select the historical expert strategies corresponding to the top preset number of cosine similarities in the similarity ranking set as the current candidate strategy set.

3. The autonomous control method for a human-machine integrated unmanned system based on a large language model according to claim 1, characterized in that, After obtaining the optimal strategy for the current moment, the following steps are also included: Perform policy constraint verification on the optimal policy at the current moment; the policy constraint verification includes security constraints and dynamic constraints. If all constraints in the policy constraint check are satisfied, the policy constraint check passes. If any constraint in the policy constraint check is not met, the policy constraint check fails.

4. The autonomous control method for a human-machine integrated unmanned system based on a large language model according to claim 1, characterized in that, Based on the optimal strategy at the current moment, control the unmanned system to execute the current task corresponding to the current unstructured high-level semantic instruction, specifically including: Map the optimal strategy at the current moment to the preset action primitive database to obtain the current underlying executable action primitive; The current low-level executable action primitives are converted into low-level control commands; the low-level control commands include speed, heading, attitude, and altitude; The low-level control commands are input into the low-level controller of the unmanned system to control the unmanned system to execute the current task corresponding to the current unstructured high-level semantic command.

5. The autonomous control method for a human-machine integrated unmanned system based on a large language model according to claim 3, characterized in that, Security constraints, specifically including: Projecting the optimal strategy at the current moment onto the current local environment topology map yields the predicted path of the unmanned system; Based on the predicted path and the current local environment topology, the minimum distance between the unmanned system and each obstacle is calculated. If the minimum distance between the unmanned system and each obstacle is greater than or equal to the preset safety threshold, then the safety constraint is satisfied. If the minimum distance between the unmanned system and any obstacle is less than the preset safety threshold, the safety constraint is not met. Dynamic constraints, specifically including: Based on the optimal strategy at the current moment, extract the velocity, acceleration, and yaw rate of change; If the speed is greater than the preset maximum permissible speed, or the acceleration is greater than the preset maximum permissible acceleration, or the rate of change of yaw rate is greater than the preset maximum permissible rate of change of yaw rate, then the dynamic constraints are not met. If the speed is less than or equal to the preset maximum permissible speed, the acceleration is less than or equal to the preset maximum permissible acceleration, and the rate of change of yaw rate is less than or equal to the preset maximum permissible rate of change of yaw rate, then the dynamic constraints are satisfied.

6. The autonomous control method for a human-machine integrated unmanned system based on a large language model according to claim 5, characterized in that, If the cosine similarity of all historical expert strategies in the current candidate strategy set is lower than the preset similarity threshold, or if the strategy constraint verification fails, a safety guarantee mechanism is triggered. The safety guarantee mechanism includes switching to an autonomous navigation mode based on geometric obstacle avoidance to control the unmanned system or controlling the unmanned system to return to a preset safety point.

7. The autonomous control method for a human-machine integrated unmanned system based on a large language model according to claim 4, characterized in that, The autonomous control method for human-machine fusion unmanned systems based on large language models further includes: during the process of inputting low-level control commands into the low-level controller of the unmanned system and controlling the unmanned system to execute the current task corresponding to the current unstructured high-level semantic command, the low-level control commands are locally adjusted through the autonomous obstacle avoidance algorithm of the lidar to achieve real-time obstacle avoidance.

8. An autonomous control system for a human-machine integrated unmanned system based on a large language model, characterized in that, The autonomous control system for the human-machine fusion unmanned system based on a large language model applies the autonomous control method for the human-machine fusion unmanned system based on any one of claims 1-7, and the autonomous control system for the human-machine fusion unmanned system based on a large language model includes: The instruction acquisition module is used to acquire an unstructured high-level semantic instruction sequence; the unstructured high-level semantic instruction sequence includes multiple unstructured high-level semantic instructions; The task scheduling module iterates through the sequence of unstructured high-level semantic instructions, executing the following steps for each instruction until all tasks corresponding to the unstructured high-level semantic instructions in the sequence have been completed, at which point control terminates: The environment perception module is used to take the start time corresponding to the current unstructured high-level semantic instruction in the unstructured high-level semantic instruction sequence as the current time, and acquire the multi-sensor environment data at the current time; the multi-sensor environment data includes lidar data, visual sensor data, and inertial measurement unit data; The topology building module is used to construct a local environmental topology map based on the multi-sensor environmental data at the current moment. The strategy retrieval module is used to retrieve the expert strategy database by using the current unstructured high-level semantic instructions and the current local environment topology as fusion query vectors, and select a preset number of historical expert strategies as the current candidate strategy set; the expert strategy database includes historical fusion query vectors and corresponding historical expert strategies; The decision generation module is used to obtain the optimal policy at the current moment based on the current unstructured high-level semantic instructions, the current local environment topology graph, and the current candidate policy set, through a large language model. The task execution module is used to control the unmanned system to execute the current task corresponding to the current unstructured high-level semantic instruction based on the optimal strategy at the current moment. The loop control module is used to obtain multi-sensor environmental data at the next moment to update the current local environmental topology map if the current task is not completed, and repeat the above steps until the current task is completed.

Images