Heterogeneous unmanned aerial vehicle cluster communication message processing method and system

Abstract

The invention discloses a heterogeneous unmanned aerial vehicle cluster communication message processing method and system. The method comprises the following steps that a user uploads a communication protocol document and scene description of an unmanned aerial vehicle; the method comprises the following steps: designing a multi-modal analysis mechanism on the basis of a retrieval enhancement generation (RAG) technology endowed by a large language model (LLM), automatically identifying and processing a communication protocol document, and generating a knowledge graph and a communication message specification library; based on the knowledge graph and the communication message specification library, generating a structured definition according to a scene, and converting the structured definition into a multi-format message instance; and simulating a communication scene, verifying message transmission validity, and outputting an optimization suggestion. By adopting the technical scheme, the heterogeneous unmanned aerial vehicle protocol multi-modal analysis method based on LLM + RAG improves the time for analyzing the standard document and the message instance generation efficiency.

Description

Technical Field

[0001] This invention belongs to the field of heterogeneous unmanned aerial vehicle (UAV) technology, and relates to a method and system for processing communication messages in heterogeneous UAV swarms. Background Technology

[0002] With the rapid development of heterogeneous drone technology, its application scenarios have expanded to multiple fields such as aerial surveying and mapping, logistics and distribution, agricultural plant protection, and emergency rescue. Communication between drones and ground stations, as well as between drones themselves, relies on standardized message exchange, involving the transmission of various types of messages, including flight status data (such as position, speed, and attitude), mission commands (such as takeoff, cruise, and return), and equipment status (such as battery level and sensor health).

[0003] To ensure communication consistency, the industry has developed a number of protocol standards (such as ASTM F3411 heterogeneous UAV system communication standard and GB / T 38940 UAV remote identification specification). These standard documents contain multimodal content such as text descriptions, table definitions, flowcharts, and mathematical formulas, and are the core basis for message interaction.

[0004] However, the complexity of heterogeneous UAV communication protocols (such as multi-scenario adaptation and multi-format compatibility) places high demands on the standardized generation and testing of messages. Traditional protocol verification relies on manual document parsing and message encoding, which is difficult to adapt to the rapid iteration needs of the industry. Therefore, automated tools are urgently needed to support the efficient implementation and verification of protocols.

[0005] Currently, the processing of heterogeneous UAV communication messages mainly relies on the following methods:

[0006] 1. Manual parsing and coding: Domain experts interpret the protocol documents line by line (such as the message field definitions and format constraints in ASTM F3411), manually write code to generate message instances, or use general programming tools (such as Python scripts) to generate simple format messages (such as JSON) in batches.

[0007] 2. General document parsing tools: These tools use libraries such as Apache PDFBox and Python-docx to parse document text and tables, but they cannot handle non-text content such as images, flowcharts, and mathematical formulas, and they lack the ability to understand protocol semantics.

[0008] 3. Dedicated message generation tools: Some drone manufacturers provide customized tools (such as the message debugging tool that comes with the DJI SDK), but these only support the private protocols of specific brand devices and have limited functionality (such as only generating fixed-format messages and not supporting outlier testing or multi-format conversion).

[0009] The existing technology still has the following problems:

[0010] 1. Inefficiency: Manually parsing protocol documents of more than 3,000 pages takes weeks or even months, and it is difficult to keep up with the updates of protocol standards (such as version iterations of ASTM F3411).

[0011] 2. Insufficient precision: Natural language interpretation is prone to ambiguity, leading to missing message fields and format errors (such as incorrect nesting of XML tags), which affects communication compatibility.

[0012] 3. Functional limitations: General-purpose tools cannot handle multimodal content (such as message interaction sequence in flowcharts), and dedicated tools do not support cross-protocol format conversion (such as JSON and XML conversion), and lack the ability to generate abnormal messages (such as boundary value and structural defect messages). Summary of the Invention

[0013] The purpose of this invention is to provide a method and system for processing heterogeneous UAV swarm communication messages, which improves the time for parsing standard documents and the efficiency of message instance generation, and reduces formatting errors caused by manual interpretation.

[0014] To achieve the above objectives, the basic solution of the present invention is: a heterogeneous unmanned aerial vehicle (UAV) swarm communication message processing method, comprising the following steps:

[0015] Users upload drone communication protocol documents and scenario descriptions;

[0016] Based on the retrieval enhancement generation technology empowered by the Large Language Model (LLM), a multimodal parsing mechanism is designed to automatically identify and process communication protocol documents, and generate a knowledge graph and a communication message specification library.

[0017] Based on knowledge graphs and communication message specification libraries, structured definitions are generated according to scenarios and converted into multi-format message instances;

[0018] Simulate communication scenarios, verify the effectiveness of message transmission, and output optimization suggestions.

[0019] The working principle and beneficial effects of this basic solution are as follows: This technical solution is based on LLM+RAG, which automatically parses multimodal content (text, tables, images, formulas, etc.) in heterogeneous UAV communication protocols to achieve structured extraction of domain knowledge. Based on the extracted knowledge, it automatically generates standard-compliant message definitions and supports multi-format conversion, significantly improving the time for parsing standard documents, increasing the efficiency of message instance generation, and ensuring high consistency between the message dictionary and protocol standards, thus reducing format errors caused by manual interpretation.

[0020] Furthermore, based on the Retrieval Enhancement Generation (RAG) technology empowered by Large Language Model (LLM), the method for designing a multimodal parsing mechanism is as follows:

[0021] Multimodal inputs are converted into text descriptions and embedding vectors in a unified semantic space, and a multimodal knowledge base containing these embedding vectors is constructed.

[0022] When processing user requests, the query intent is first parsed. The method for parsing the query intent is as follows:

[0023] The query is split into text and non-text parts. The non-text parts are converted into text descriptions. That is, the logical nesting relationship between elements is sorted out by category to clarify the correspondence. Finally, it is integrated into unambiguous structured text. After integration, relevant context is retrieved from the multimodal knowledge base through LLM combined with RAG to clarify the query intent.

[0024] LLM generates a retrieval vector based on this query intent, specifically as follows:

[0025] Establish a fixed-dimensional array of floating-point numbers, where each value corresponds to an abstract quantization encoding of the text semantics. This numerical form transforms the "query intent" into a computable semantic identifier.

[0026] Since all information in the multimodal knowledge base is converted into vectors of the same dimension, the retrieval vector can locate semantically matching content by calculating the similarity with other vectors;

[0027] The semantically matched content filtered through similarity calculation is input into the LLM+RAG retrieval knowledge base to ultimately determine the query intent.

[0028] Based on the Retrieval Enhancement Generation (RAG) technology enabled by Large Language Model (LLM), a multimodal parsing mechanism is designed to facilitate subsequent use.

[0029] Furthermore, the method for automatically identifying and processing communication protocol documents and generating a knowledge graph and communication message specification base using a multimodal parsing mechanism is as follows:

[0030] Protocol text rule decoding: Based on a multimodal parsing mechanism, leveraging the semantic understanding capabilities of a large language model combined with domain knowledge retrieval, it locates natural language content related to message interaction in the protocol, identifies natural language clauses in heterogeneous UAV communication protocols, and extracts rule elements directly related to message interaction, core rules for message transmission, and communication representations of device anomalies. It transforms fragmented text into communication rule entries composed of rules and message types. For example, if a user-uploaded protocol states "send status report message when UAV is low on battery", it is decoded into the rule: anomaly "low battery" → message type "status report" and stored in the communication message specification library.

[0031] Protocol table structure deconstruction: For the structured tables defining message formats in the protocol, leveraging LLM's semantic understanding capabilities, the hierarchical nesting between fields is identified to extract the technical attributes of each field, namely: basic identification attributes: the core information of the field, such as "field name" (e.g., "device ID" or "battery level"). Data format attributes: the storage and representation rules of the field, including "data type" (e.g., int, float, string, etc.), value constraint attributes: the legal range and restrictions of the field, such as "value range" (e.g., "battery level" is 0-100%), and logical association attributes: the binding relationship between the field and other fields or rules, such as "dependency relationship" (e.g., the "target coordinates" field must appear simultaneously with the "coordinate system type" field).

[0032] The specific principle behind identifying the nested relationships between fields is achieved through semantic association using LLM combined with explicit table identifiers and domain knowledge: LLM first pays attention to format clues, such as the indentation of child fields relative to parent fields (e.g., "Data Frame" is flush left, and the "Header" below it is indented by 2 spaces), hierarchical separators in naming (e.g., "." in "Data Field.Task Code"), or direct annotations in the dedicated column for "Parent Field Name"; at the same time, it combines the common structural knowledge of communication protocols (e.g., "Data Frame" usually contains a fixed hierarchy such as "Header", "Data Field", and "Check Digit") to finally clarify the nested chain (e.g., "Data Frame contains Header, and Header contains Device ID and Frame Identifier");

[0033] Protocol graphical interactive modeling: This involves transforming the visual graphics within the protocol into communication logic, specifically:

[0034] When deconstructing the visual diagrams in the protocol, the key elements are first identified, including participants (such as drones, ground stations, and other communication entities), directional message arrows (with text next to the arrows describing the message content, such as "task instructions"), a timeline running downwards along the participants (reflecting the order in which messages were sent), and special annotations. During parsing, the transmission order of the message arrows is traced from top to bottom along the timeline to clarify the interaction order. Timeout thresholds are extracted from annotations or protocol text. Based on the identifiers and the protocol, the triggering conditions for retransmission (such as no response after timeout) and the specific number of times are determined. Finally, using LLM combined with RAG, the parsed information on participants, interaction order, timeout thresholds, and retransmission rules is verified for accuracy against the protocol specifications retrieved by RAG. The LLM then generates a standardized and coherent logical description. For example: If a user uploads a picture and a drone sends a "landing request," and there is no response within 5 seconds, it retransmits twice. This can be parsed as: The drone sent the request first, and after a 5-second timeout, it retransmitted twice.

[0035] If it is a state transition diagram, extract the mandatory binding relationship between the state triggering conditions and the messages, and transform the graphical logic into an executable set of communication rules;

[0036] The knowledge graph and communication message specification library are constructed based on the results of protocol parsing: The communication message specification library integrates the rules of protocol text decoding, the field constraints of table deconstruction, the encoded semantic relationships of task mapping tables, and the interactive logic of graph transformation. It unifies and structures these scattered rules to form a standardized set covering message types, field attributes, constraints, and exception handling.

[0037] Building upon this foundation, knowledge graphs construct a semantic association network by associating various protocol elements within the communication message specification library. These elements, including message types, rule elements, and field attributes, are set as nodes, and relationships between nodes, including subordinate and triggering relationships, are defined. This enables visualization and reasonable invocation of relationships between rules.

[0038] By utilizing a multimodal parsing mechanism, communication protocol documents are automatically identified and processed to generate a knowledge graph and a communication message specification library.

[0039] Furthermore, the knowledge graph and communication message specification library includes major core message types: flight status, mission instructions, equipment status, sensor data, fault alarms, and environmental perception, as well as subcategories of messages: flight status, real-time location messages, velocity vector messages, and attitude angle messages.

[0040] Each message category's dictionary entry must clearly specify the field name, data type, value range, unit, constraints, and semantic description.

[0041] The generated annotation library has a consistency rate of ≥90% with the protocol standard, providing a unified reference standard for subsequent automatic message generation and format verification.

[0042] Based on the analysis results, a communication message specification library conforming to the heterogeneous UAV communication protocol specification is generated.

[0043] Furthermore, the method for generating structured definitions based on the scenario and converting them into multi-format message instances is as follows:

[0044] Semantic decomposition of unstructured scene descriptions is performed as follows:

[0045] The heterogeneous UAV domain corpus (including entities such as task type, coordinates, and parameters) is labeled. Through the LLM+RAG framework, prompt text is input to the LLM to adapt to the heterogeneous UAV scenario and guide the Large Language Model (LLM) to extract domain content. That is, the civil aviation domain customized prompt. The LLM extracts the corresponding key messages from the text based on semantic understanding. Then, the RAG module searches the communication message specification library (such as message element dictionary and parameter constraint rules) to verify the legality of the entities extracted by the LLM. Finally, it can recognize the core entities such as "task allocation message", "target area coordinates", "shooting resolution" and "UAV number".

[0046] Association protocol rules: Retrieve a standard structure template that perfectly matches the "task allocation message" from the communication message specification library, and then recursively traverse the hierarchical relationship of the template. The communication message specification library defines that each level stores the same type of message (such as the "target information" level corresponding to coordinates, parameters, etc.) to realize the mapping of the extracted entities to the corresponding level, while parsing the data type and value constraints, and finally inheriting the enumeration, length, and range constraints of the fields in the template.

[0047] Generate messages in multiple formats including JSON, XML, and Domain-Specific Language for Message Sharing (DSL), and implement bidirectional conversion between different formats, specifically:

[0048] Build syntax templates for JSON and XML, i.e., placeholders for built-in protocol fields;

[0049] Extract relevant protocol rules from the communication message specification library, substitute the rules into the syntax template to replace placeholders, and generate message instances containing random values, abnormal values, and boundary values ​​within the normal range. The default values ​​of the message header fields in the communication message specification library are the default values ​​for filling the message header. Users can manually modify the fields and field values. Abnormal message instances with structural defects can be generated through configuration strategies.

[0050] It utilizes an automatic conversion engine for multi-format messages (JSON / XML / DSL), especially the syntax mapping rules of custom domain languages, to support multimodal parsing, multi-format conversion, and exception testing, covering all scenarios of protocol verification, and has comprehensive functions.

[0051] Furthermore, simulated communication scenarios were used to verify the effectiveness of message transmission and optimization suggestions were generated. The specific methods are as follows:

[0052] When performing message reachability testing, the generated multi-format message instances (covering random values, outliers, boundary values, and structural defect messages within the normal range) are invoked, and the simulation scenario parameters (including the communication entity identifiers between the UAV and the ground station, the UAV and the communication between the UAVs, the transmission protocol, and the network environment) are configured.

[0053] Based on user-uploaded usage scenarios, the test simulates message transmission to drones or ground base stations, and checks whether the message is transmitted to the designated drone or ground base station. Combining the user-uploaded usage scenarios (e.g., drone-to-drone message transmission), the test determines whether the message conforms to the communication protocol rules for that scenario. Based on the protocol rules in the communication message specification library, the test calculates the proportion of message types in the test relative to the total types in the library, generating a test report. Specifically:

[0054] RAG retrieves protocol rules related to the current scenario from the communication message specification library (such as the total number of message types allowed in the scenario, constraints on sending and receiving entities, etc.), and at the same time retrieves the raw data recorded during the test (transmission success / failure results, details of violations, and a list of message types actually used);

[0055] LLM receives these retrieved rule data and test data, and first sorts out the information (such as classifying it according to "transmission detection results", "rule verification conclusions" and "type coverage");

[0056] LLM leverages natural language generation capabilities to transform structured information into a report format that conforms to industry standards. This includes scenario descriptions, detailed test items (such as "Did the message reach the target: Success / Failure" and "Does it comply with the protocol: Yes / No + Reason for violation"), type coverage calculation process (number of test types / total number of types × 100%), and summary conclusions. The final output is a test report that is both accurate and readable, supporting prototype system verification.

[0057] The present invention also provides a heterogeneous UAV swarm communication message processing system based on the method described in the present invention, including a human-computer interaction module, a multimodal parsing module, a message definition module and a testing module;

[0058] The human-computer interaction module is used to upload the communication protocol document and scene description of the heterogeneous UAV and transmit them to the multimodal parsing module;

[0059] The multimodal parsing module is used to automatically identify and process communication protocol documents, and generate knowledge graphs and communication message specification libraries;

[0060] The input of the message definition module is connected to the output of the multimodal parsing module. The message definition module generates a structured definition according to the scenario and converts it into a multi-format message instance.

[0061] The input of the test module is connected to the output of the message definition module. The test module is used to simulate communication scenarios, verify the effectiveness of message transmission, and output optimization suggestions.

[0062] This system integrates message reachability testing functionality, which can simulate communication scenarios between drones and ground stations, as well as between drones themselves. It performs transmission reachability tests on generated message instances, and the entire process requires minimal manual intervention, achieving an automated closed loop from protocol document parsing to message testing. Attached Figure Description

[0063] Figure 1 This is a flowchart illustrating the heterogeneous UAV swarm communication message processing method of the present invention;

[0064] Figure 2This is a flowchart illustrating the multimodal parsing mechanism of the heterogeneous UAV swarm communication message processing method of the present invention. Detailed Implementation

[0065] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0066] In the description of this invention, it should be understood that the terms "longitudinal", "lateral", "up", "down", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0067] In the description of this invention, unless otherwise specified and limited, it should be noted that the terms "installation", "connection" and "linking" should be interpreted broadly. For example, they can refer to mechanical or electrical connections, or internal connections between two components. They can be direct connections or indirect connections through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms according to the specific circumstances.

[0068] This invention discloses a heterogeneous UAV swarm communication message processing method, which revolves around the core architecture of "protocol semantic deconstruction - message rule modeling - communication scenario instantiation". This invention aims to solve the following problems:

[0069] How to automatically parse multimodal content (text, tables, images, formulas, etc.) in UAV communication protocols to achieve structured extraction of domain knowledge;

[0070] How to automatically generate standard-compliant message definitions based on extracted knowledge, and support conversion of multiple formats such as JSON, XML, and custom DSL;

[0071] How to generate message instances containing normal values, abnormal values, and boundary values ​​to support robustness testing of communication protocols.

[0072] like Figure 1 As shown, the heterogeneous UAV swarm communication message processing method includes the following steps:

[0073] User input: The user uploads the drone's communication protocol document (such as ASTM F3411.pdf) and scenario description (such as "Logistics drone delivery route update");

[0074] Semantic Mapping Engine: Based on the Retrieval Enhancement Generation (RAG) technology empowered by Large Language Model (LLM), a multimodal parsing mechanism is designed to automatically identify and process communication protocol documents, and generate knowledge graphs and communication message specification libraries;

[0075] Message generation: Based on knowledge graphs and communication message specification libraries, structured definitions are generated according to the scenario and converted into multi-format message instances;

[0076] Simulate communication scenarios, verify the effectiveness of message transmission, and output optimization suggestions.

[0077] In a preferred embodiment of the present invention, based on the Large Language Model (LLM)-enabled Retrieval Enhancement Generation (RAG) technology, the method for designing a multimodal parsing mechanism is as follows:

[0078] Multimodal inputs are converted into text descriptions and embedding vectors in a unified semantic space, and a multimodal knowledge base containing these embedding vectors is constructed, specifically as follows:

[0079] When processing user requests, the query intent is first parsed. The method for parsing the query intent is as follows:

[0080] The query is split into text and non-text parts. The non-text parts are converted into text descriptions. That is, the logical relationship between elements is sorted out by category to clarify the correspondence. Finally, it is integrated into unambiguous structured text. After integration, relevant context is retrieved from the multimodal knowledge base through LLM combined with RAG to clarify the query intent.

[0081] LLM generates a retrieval vector based on the query intent. The specific method is as follows: a fixed-dimensional array of floating-point numbers (such as 768-dimensional or 1024-dimensional) is established, and each value corresponds to an abstract quantization encoding of the text semantics. In this digital form, the "query intent" is transformed into a computable semantic identifier.

[0082] This retrieval vector is used to extract relevant content from a multimodal knowledge base. Since all information (text, image descriptions, etc.) in the multimodal knowledge base has been converted into vectors of the same dimension, the retrieval vector can locate semantically matching content by calculating its similarity to other vectors. Cosine similarity calculation is preferred.

[0083] Calculate the dot product of the retrieval vector and the target vector, which is to multiply the corresponding dimension values ​​and then sum them;

[0084] Calculate the magnitude of each vector, which is the square root of the sum of the squares of the values ​​in each dimension;

[0085] Finally, divide the dot product by the product of the two moduli. The closer the result (ranging from -1 to 1) is to 1, the more similar the two are semantically.

[0086] The semantically matched content filtered through similarity calculation is input into LLM+RAG to retrieve the knowledge base and finally determine the query intent.

[0087] In a preferred embodiment of the present invention, such as Figure 2 As shown, the method for automatically identifying and processing communication protocol documents and generating knowledge graphs and communication message specification libraries using a multimodal parsing mechanism is as follows: Protocol text rule decoding: Based on the multimodal parsing mechanism, leveraging the semantic understanding capabilities of large language models combined with domain knowledge retrieval, the natural language content related to message interaction in the protocol is located, the natural language clauses in heterogeneous UAV communication protocols are identified, and rule elements directly related to message interaction, core rules of message transmission, and communication representations of device anomalies are extracted in a targeted manner. The scattered text is transformed into communication rule entries composed of rules and message types. For example, if a user-uploaded protocol states "send status report message when UAV is low on battery", it is decoded into the rule: anomaly "low battery" → message type "status report" and stored in the communication message specification library.

[0088] Protocol table structure deconstruction: For the structured tables defining message formats in the protocol, combined with the semantic understanding capabilities of LLM, the hierarchical nesting between fields is identified to extract the technical attributes of each field, including basic identification attributes: the core information of the field, such as "field name" (e.g., "device ID" "battery level"); data format attributes: the storage and representation rules of the field, including "data type" (e.g., int, float, string (computer data format int represents positive numbers, float represents floating-point numbers, string refers to characters), etc.); value constraint attributes: the legal range and restrictions of the field, such as "value range" (e.g., "battery level" is 0-100%); logical association attributes: the binding relationship between the field and other fields or rules, such as "dependency relationship" (e.g., the "target coordinates" field must appear simultaneously with the "coordinate system type" field).

[0089] The specific principle behind identifying the nested relationships between fields is achieved through semantic association using LLM combined with explicit table identifiers and domain knowledge: LLM first pays attention to format clues, such as the indentation of child fields relative to parent fields (e.g., "Data Frame" is flush left, and the "Header" below it is indented by 2 spaces), hierarchical separators in naming (e.g., "." in "Data Field.Task Code"), or direct annotations in the dedicated column for "Parent Field Name"; at the same time, it combines the common structural knowledge of communication protocols (e.g., "Data Frame" usually contains a fixed hierarchy such as "Header", "Data Field", and "Check Digit") to finally clarify the nested chain (e.g., "Data Frame contains Header, and Header contains Device ID and Frame Identifier");

[0090] Extract the correspondence rules between encoding and semantics from the "Task Type Mapping Table" (e.g., "encoding '0x01' corresponds to 'Aerial Photography Task', '0x02' corresponds to 'Cruise Task'") to form rigid constraints for message construction; for the "Task Type Mapping Table" in the protocol, extract the unique correspondence between the encoded value and the task semantics (e.g., 0x01 is fixedly mapped to the "Fixed Point Photography" task), and synchronously associate the parameter format constraints corresponding to the encoding (e.g., "Fixed Point Photography" must carry latitude and longitude fields, and the data type is FLOAT64), and encapsulate the encoding, semantics, and parameter rules into rigid constraints for message construction (the task field in the message must match the encoding, and missing parameters or incorrect format are considered illegal);

[0091] Protocol graphical interactive modeling: This involves transforming the visual graphics within the protocol into communication logic, specifically:

[0092] When deconstructing the visual diagrams in the protocol, the key elements are first identified, including participants (such as drones, ground stations, and other communication entities), directional message arrows (with text next to the arrows describing the message content, such as "task instructions"), a timeline running downwards along the participants (reflecting the order in which messages were sent), and special annotations. During parsing, the transmission order of the message arrows is traced from top to bottom along the timeline to clarify the interaction order. Timeout thresholds are extracted from annotations or protocol text. Based on the identifiers and the protocol, the triggering conditions for retransmission (such as no response after timeout) and the specific number of times are determined. Finally, using LLM combined with RAG, the parsed information on participants, interaction order, timeout thresholds, and retransmission rules is verified for accuracy against the protocol specifications retrieved by RAG. The LLM then generates a standardized and coherent logical description. For example: If a user uploads a picture and a drone sends a "landing request," and there is no response within 5 seconds, it retransmits twice. This can be parsed as: The drone sent the request first, and after a 5-second timeout, it retransmitted twice.

[0093] If it is a state transition diagram, extract the mandatory binding relationship between the state triggering conditions (such as battery level <10%) and the message, and transform the graphical logic into an executable set of communication rules;

[0094] The knowledge graph and communication message specification library are constructed based on the results of protocol parsing: The communication message specification library integrates the rules of protocol text decoding, the field constraints of table deconstruction, the encoded semantic relationships of task mapping tables, and the interactive logic of graph transformation. It unifies and structures these scattered rules (such as JSON format) to form a standardized set covering message types, field attributes, constraints, and exception handling.

[0095] Building upon this foundation, the knowledge graph, by associating various protocol elements in the communication message specification library (such as associating "flight status message" with "timestamp field constraint" and "5-second reporting rule"), sets the basic elements in the communication message specification library, including message type, rule elements, and field attributes, as nodes, defines the relationships between nodes, including subordinate relationships and triggering relationships, constructs a semantic association network, and realizes the visualization and reasonable invocation of relationships between rules;

[0096] For example, the "takeoff command interaction sequence diagram" can be broken down into a step-by-step communication process: "ground station sends final command → UAV executes and reports results"; the "sensor data flow diagram" can be transformed into a data processing link to realize the conversion of graphics into executable interaction rules.

[0097] In a preferred embodiment of the present invention, based on the parsing results, the knowledge graph and communication message specification library includes 15 major categories of core message types: flight status, mission instructions, equipment status, sensor data, fault alarms, environmental perception, etc., and 200 subcategories of detailed messages: flight status, real-time location messages, velocity vector messages, attitude angle messages, etc.

[0098] Each message category's dictionary entries must clearly specify the field name, data type (int / float / string, etc.), value range (e.g., "battery power" is 0-100%), unit (e.g., meters, kilometers per hour), constraints (e.g., "latitude and longitude" must conform to the WGS84 coordinate system format), and semantic description information (e.g., "device ID" is a unique identifier for the drone).

[0099] The generated annotation library has a consistency rate of ≥90% with the protocol standard, supports dynamic expansion (such as adding the "5G communication status message" type), and provides a unified reference standard for subsequent automatic message generation and format verification, significantly reducing the workload and error rate of manual dictionary compilation.

[0100] In a preferred embodiment of the present invention, in message generation and processing, the system can automatically generate structured message definitions conforming to protocol specifications based on unstructured heterogeneous scenario descriptions, clearly defining field hierarchy, data types, and value constraints, while also supporting manual adjustments by users through a visual interface. The method for generating structured definitions based on scenarios and converting them into multi-format message instances is as follows:

[0101] Semantic decomposition of unstructured scene descriptions (e.g., "generating task allocation messages for multi-aircraft collaborative aerial photography, which must include target area coordinates, shooting resolution, and aircraft numbers") is performed, specifically as follows:

[0102] The UAV domain corpus (including entities such as task type, coordinates, and parameters) is labeled. Through the LLM+RAG framework, prompt text is input to the LLM to adapt to heterogeneous UAV scenarios and guide the Large Language Model (LLM) to extract domain content. That is, the civil aviation domain customized prompt. The LLM extracts the corresponding key messages from the text based on semantic understanding. Then, the LLM+RAG module searches the communication message specification library (such as message element dictionary and parameter constraint rules) to verify the legality of the entities extracted by the LLM. Finally, it can recognize the core entities such as "task allocation message", "target area coordinates", "shooting resolution" and "UAV number".

[0103] Association Protocol Rules: Retrieve a standard structure template that perfectly matches the "task allocation message" from the communication message specification library, and then recursively traverse the hierarchical relationship of the template. The communication message specification library defines that each level stores the same type of message (such as the "target information" level corresponding to coordinates and parameters). This enables the extraction of entities (such as UAV number = 001) to the corresponding level (task parameters → UAV number). At the same time, it parses the data type and value constraints, and finally inherits the enumeration, length, and range constraints of the fields in the template.

[0104] Generate messages in multiple formats including JSON, XML, and Domain-Specific Language for Message Sharing (DSL), and implement bidirectional conversion between different formats, specifically:

[0105] Build syntax templates for JSON and XML, i.e., placeholders for built-in protocol fields;

[0106] Extract relevant protocol rules from the communication message specification library, substitute the rules into the syntax template to replace placeholders, and generate message instances containing random values, abnormal values, and boundary values ​​within the normal range with one click. The default values ​​of the message header fields in the communication message specification library are the default values ​​for filling the message header. Users can manually modify the fields and field values. Abnormal message instances with structural defects can be generated through configuration strategies.

[0107] In a preferred embodiment of the present invention, a communication scenario is simulated to verify the effectiveness of message transmission and to output optimization suggestions. The specific method is as follows:

[0108] When performing message reachability testing, the generated multi-format message instances (covering random values, outliers, boundary values, and structural defect messages within the normal range) are invoked, and the simulation scenario parameters (including the communication entity identifiers between the UAV and the ground station, the UAV and the communication between the UAVs, the transmission protocol, and the network environment) are configured.

[0109] Based on user-uploaded usage scenarios, the test simulates message transmission to drones or ground base stations, and checks whether the message is transmitted to the designated drone or ground base station. Combining the user-uploaded usage scenarios (e.g., drone-to-drone message transmission), the test determines whether the message conforms to the communication protocol rules for that scenario. Based on the protocol rules in the communication message specification library, the test calculates the proportion of message types in the test relative to the total types in the library, generating a test report. Specifically:

[0110] RAG retrieves protocol rules related to the current scenario from the communication message specification library (such as the total number of message types allowed in the scenario, constraints on sending and receiving entities, etc.), and at the same time retrieves the raw data recorded during the test (transmission success / failure results, details of violations, and a list of message types actually used);

[0111] LLM receives these retrieved rule data and test data, and first sorts out the information (such as classifying it according to "transmission detection results", "rule verification conclusions" and "type coverage");

[0112] LLM leverages natural language generation capabilities to transform structured information into a report format that conforms to industry standards—including scenario descriptions, detailed test items (such as "Did the message reach the target: Success / Failure" and "Does it comply with the protocol: Yes / No + Reason for violation"), type coverage calculation process (number of test types / total number of types × 100%), and summary conclusions. The final output is a test report that is both accurate and readable, supporting prototype system verification.

[0113] This invention presents a heterogeneous UAV protocol multimodal parsing method based on LLM+RAG (including structured conversion algorithms for images and flowcharts). It supports dynamic dictionary generation technology for 15 major categories and 200 subcategories of messages, as well as an adaptation mechanism with standards such as ASTM F3411. It features an automatic conversion engine for multiple message formats (JSON / XML / DSL), especially with custom domain language syntax mapping rules. This significantly improves the time for parsing standard documents and increases the efficiency of message instance generation. The message dictionary maintains high consistency with protocol standards, reducing formatting errors caused by manual interpretation. Furthermore, it supports multimodal parsing, multi-format conversion, and anomaly testing, covering the entire protocol verification scenario.

[0114] This invention also provides a heterogeneous UAV swarm communication message processing system based on the method described herein, including a human-computer interaction module, a multimodal parsing module (semantic mapping engine), a message definition module (message generation module), and a testing module. The human-computer interaction module is used to upload the UAV's communication protocol document and scene description, and transmit them to the multimodal parsing module.

[0115] The multimodal parsing module automatically identifies and processes communication protocol documents, generating a knowledge graph and a communication message specification library. The message definition module's input is connected to the multimodal parsing module's output; the message definition module generates structured definitions based on the scenario and converts them into multi-format message instances.

[0116] The input of the test module is connected to the output of the message definition module. The test module is used to simulate communication scenarios, verify the effectiveness of message transmission, and output optimization suggestions.

[0117] This system can be developed using languages ​​such as Java and C++, with some functions developed using Python. It is equipped with a visual user interface and can automate the entire process of processing UAV communication protocol standard documents.

[0118] Regarding the semantic structure extraction of protocols: In view of the complexity of UAV communication protocol standard documents (such as ASTM F3411, GB / T38940, etc.), a 'semantic mapping engine' is used. This engine is designed based on a multimodal parsing mechanism that integrates large language model (LLM) and retrieval-enhanced generation (RAG).

[0119] Preferably, message format conversion can be achieved by replacing XSLT with XSD Schema to generate XML while maintaining functionality; the parsing module can be replaced by other AI models (such as LLaMA 2).

[0120] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0121] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A method for processing heterogeneous unmanned aerial vehicle (UAV) swarm communication messages, characterized in that, Includes the following steps: Users upload drone communication protocol documents and scenario descriptions; Based on the retrieval enhancement generation technology empowered by the Large Language Model (LLM), a multimodal parsing mechanism is designed to automatically identify and process communication protocol documents, and generate a knowledge graph and a communication message specification library. Based on knowledge graphs and communication message specification libraries, structured definitions are generated according to scenarios and converted into multi-format message instances; Simulate communication scenarios, verify the effectiveness of message transmission, and output optimization suggestions.

2. The heterogeneous UAV swarm communication message processing method as described in claim 1, characterized in that, Based on the retrieval enhancement generation technology empowered by the Large Language Model (LLM), the method for designing a multimodal parsing mechanism is as follows: Multimodal inputs are converted into text descriptions and embedding vectors in a unified semantic space, and a multimodal knowledge base containing these embedding vectors is constructed. When processing user requests, the query intent is first parsed. The method for parsing the query intent is as follows: The query is split into text and non-text parts. The non-text parts are converted into text descriptions. That is, the logical nesting relationship between elements is sorted out by category to clarify the correspondence. Finally, it is integrated into unambiguous structured text. After integration, relevant context is retrieved from the multimodal knowledge base through LLM combined with RAG to clarify the query intent. LLM generates a retrieval vector based on this query intent, specifically as follows: Establish a fixed-dimensional array of floating-point numbers, where each value corresponds to an abstract quantization encoding of the text semantics. This numerical form transforms the "query intent" into a computable semantic identifier. Since all information in the multimodal knowledge base is converted into vectors of the same dimension, the retrieval vector can locate semantically matching content by calculating the similarity with other vectors; The semantically matched content filtered through similarity calculation is input into the LLM+RAG retrieval knowledge base to ultimately determine the query intent.

3. The heterogeneous UAV swarm communication message processing method as described in claim 2, characterized in that, The method for automatically identifying and processing communication protocol documents and generating knowledge graphs and communication message specification bases using a multimodal parsing mechanism is as follows: Protocol text rule decoding: Based on the multimodal parsing mechanism, leveraging the semantic understanding capabilities of the large language model combined with domain knowledge retrieval, it locates the natural language content related to message interaction in the protocol, identifies the natural language clauses in the UAV communication protocol, and extracts the rule elements directly related to message interaction, the core rules of message transmission, and the communication characteristics of device anomalies, transforming scattered text into communication rule entries composed of rules and message types. Protocol table structure deconstruction: For the structured table that defines the message format in the protocol, combined with the semantic understanding capability of LLM, the hierarchical nesting between fields is identified to extract the attributes of each field, including basic identification attributes: the core information of the field, data format attributes: the storage and representation rules of the field, value constraint attributes: the legal range and restrictions of the field, and logical association attributes: the binding relationship between the field and other fields or rules. By combining explicit table identifiers with domain knowledge through LLM, the nested relationship of fields can be identified. Specifically, LLM first pays attention to format clues, including the indentation of child fields relative to parent fields, hierarchical separators in naming, or direct annotations in the "parent field name" column; at the same time, it combines the common structural knowledge of communication protocols to finally clarify the nested chain. Protocol graphical interactive modeling, that is, transforming the visual graphics in the protocol into communication logic, specifically: When deconstructing the visual graph of the protocol, the key elements in the graph are first identified, including participants, directional message arrows, timelines downwards along the participants, and special annotations. During parsing, the transmission order of message arrows is sorted out from top to bottom along the timeline to clarify the interaction order. Timeout thresholds are extracted from annotations or protocol text. Based on the identifiers and the protocol, the triggering conditions and specific number of retransmissions are determined. Finally, by combining LLM with RAG, the parsed participants, interaction order, timeout thresholds, and retransmission rules information are verified for accuracy by referring to the protocol specifications retrieved by RAG. Then, LLM generates a standardized and coherent logical description. If it is a state transition diagram, extract the mandatory binding relationship between the state triggering conditions and the messages, and transform the graphical logic into an executable set of communication rules; The knowledge graph and communication message specification library are constructed based on the results of protocol parsing: The communication message specification library integrates the rules of protocol text decoding, the field constraints of table deconstruction, the encoded semantic relationships of task mapping tables, and the interactive logic of graph transformation. It unifies and structures these scattered rules to form a standardized set covering message types, field attributes, constraints, and exception handling. Building upon this foundation, knowledge graphs construct a semantic association network by associating various protocol elements within the communication message specification library. These elements, including message types, rule elements, and field attributes, are set as nodes, and relationships between nodes, including subordinate and triggering relationships, are defined. This enables visualization and reasonable invocation of relationships between rules.

4. The heterogeneous UAV swarm communication message processing method as described in claim 3, characterized in that, The knowledge graph and communication message specification library includes major core message types: flight status, mission instructions, equipment status, sensor data, fault alarms, and environmental perception, as well as subcategories of messages: flight status, real-time location messages, velocity vector messages, and attitude angle messages. Each message category's dictionary entry must clearly specify the field name, data type, value range, unit, constraints, and semantic description. The generated annotation library has a consistency rate of ≥90% with the protocol standard, providing a unified reference standard for subsequent automatic message generation and format verification.

5. The heterogeneous UAV swarm communication message processing method as described in claim 1, characterized in that, The method for generating structured definitions based on the scenario and converting them into multi-format message instances is as follows: Semantic decomposition of unstructured scene descriptions is performed as follows: The UAV domain corpus is labeled, and the LLM+RAG framework is used to input prompt text into the LLM to adapt to heterogeneous UAV scenarios and guide the large language model LLM to extract domain content. Specifically, the civil aviation domain customized prompt is used. The LLM extracts the corresponding key messages from the text based on semantic understanding. The RAG module searches the communication message specification library and performs legality verification on the entities extracted by the LLM. Finally, it can recognize core entities such as "task allocation message", "target area coordinates", "shooting resolution" and "UAV number". Association protocol rules: Retrieve a standard structure template that perfectly matches the "task allocation message" from the communication message specification library, then recursively traverse the hierarchical relationship of the template. The communication message specification library defines that each level stores the same type of message, thus mapping the extracted entities to the corresponding level. At the same time, it parses the data type and value constraints, and finally inherits the enumeration, length, and range constraints of the fields in the template. Generate messages in multiple formats, including JSON, XML, and message-sharing domain-specific languages, and implement bidirectional conversion between different formats, specifically: Build syntax templates for JSON and XML, i.e., placeholders for built-in protocol fields; Extract the corresponding protocol rules from the communication message specification library, substitute the rules into the syntax template to replace the placeholders, and generate message instances containing random values, abnormal values, and boundary values ​​within the normal range. The default values ​​of the message header fields in the communication message specification library are the default values ​​for filling the message header, and users can manually modify the fields and field values.

6. The heterogeneous UAV swarm communication message processing method as described in claim 1, characterized in that, Simulate communication scenarios, verify the effectiveness of message transmission, and output optimization suggestions. The specific method is as follows: When performing message reachability testing, the generated multi-format message instance is invoked, and the simulation scenario parameters are configured, including the communication entity identifiers between the UAV and the ground station, the transmission protocol, and the network environment between the UAV and the ground station. Based on user-uploaded usage scenarios, a simulated message sending scenario to a drone or ground base station is performed, and the transmission of the message to the specified drone or ground base station is checked. Combining the user-uploaded usage scenarios, it is determined whether the scenario conforms to the communication protocol rules. The proportion of message types in the test relative to the total number of types in the communication protocol specification library is calculated using the protocols in the library to generate a test report. Specifically: RAG retrieves protocol rules relevant to the current scenario from the communication message specification library, and also retrieves raw data recorded during the test, including transmission success / failure results, details of violations, and a list of message types actually used. LLM receives these retrieved rule data and test data, and first sorts out the information; LLM leverages natural language generation capabilities to transform structured information into a report format that conforms to industry standards, including scenario descriptions, details of test items, type coverage calculation process, and summary conclusions. The final output is a test report that combines data accuracy and readability, supporting prototype system verification.

7. A heterogeneous unmanned aerial vehicle (UAV) swarm communication message processing system based on the method of any one of claims 1-6, characterized in that, It includes a human-computer interaction module, a multimodal parsing module, a message definition module, and a testing module; The human-computer interaction module is used to upload the communication protocol document and scene description of the UAV and transmit them to the multimodal parsing module; The multimodal parsing module is used to automatically identify and process communication protocol documents, and generate knowledge graphs and communication message specification libraries; The input of the message definition module is connected to the output of the multimodal parsing module. The message definition module generates a structured definition according to the scenario and converts it into a multi-format message instance. The input of the test module is connected to the output of the message definition module. The test module is used to simulate communication scenarios, verify the effectiveness of message transmission, and output optimization suggestions.

Images