An unmanned aerial vehicle anti-deception capability improvement method based on LLM multi-modal data generation

Abstract

The application relates to the technical field of deceptive data generation, in particular to a method for improving the anti-deception capability of a UAV based on LLM multi-modal data generation, wherein the method comprises the following steps: collecting multi-modal original data of a target UAV performing a flight task, and performing time synchronization and preprocessing on the multi-modal original data to obtain corresponding multi-modal feature data, and encapsulating the multi-modal feature data into unified structured text description information; a corresponding standardized Prompt template is constructed and input into a large language fine-tuning model to output multi-modal deceptive data in a target format; the multi-modal deceptive data is subjected to consistency verification to construct a deceptive data set, and then based on a feedback dynamic optimization mechanism, the deceptive data set is updated for subsequent use by the UAV to enhance anti-deception training or performance evaluation. The application can effectively expand the attack scene coverage range, reduce the cost of artificially designed samples, and the generated deceptive samples are closer to the real attack logic.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deceptive data generation, in particular to a method for improving the anti-deception ability of a UAV based on LLM multi-modal data generation. BACKGROUND

[0002] In recent years, UAV technology has been widely used in fields such as logistics distribution, geographic mapping, and security inspection. Its working environment is becoming increasingly complex, and it is facing more serious electromagnetic attack and data deception threats. UAV systems highly rely on multi-modal data such as GPS (Global Positioning System), IMU (Inertial Measurement Unit) inertial navigation, and visual sensors for positioning and navigation. Attackers can implement deception attacks by means such as falsifying GPS signals, interfering with IMU data, or tampering with visual information, causing the UAV to deviate from the planned route, fail in its mission, or even cause safety accidents.

[0003] Traditional methods for training anti-deception UAVs mainly rely on manually designed attack samples or virtual attacks generated based on simulation platforms. However, manually designed samples are limited by human experience and are difficult to cover complex electromagnetic environments and diverse attack patterns. Simulation platforms usually use idealized models and are difficult to realistically reproduce the complex characteristics of attack signals in dynamic environments, such as satellite signal multipath effects and IMU sensor temperature drift. The above limitations result in significant differences between the training data generated by traditional methods and real attack scenarios, making the trained UAV system still face a high risk of deception in actual application.

[0004] In recent years, the rapid development of artificial intelligence technology, especially the outstanding ability of large language models (LLM) in complex scene understanding and content generation. Currently, existing technologies can fine-tune large language models to enable them to understand the characteristics of UAV multi-modal data such as GPS trajectory, IMU dynamic changes, and visual scene information, which can achieve automated generation of attack samples and scenario-based adaptation.

[0005] However, the application of existing large language models is still limited to single modal or open-loop generation mode, lacking real-time feedback and dynamic optimization of attack effects, making it difficult to continuously improve the anti-deception ability of UAVs, which needs to be solved urgently. SUMMARY

[0006] The application provides a method for improving the anti-deception ability of a UAV based on LLM multi-modal data generation, to solve the problem that the application of the existing large language model is limited to a single mode or an open-loop generation mode, lacks real-time feedback and dynamic optimization of attack effects, and is difficult to continuously improve the anti-deception ability of the UAV.

[0007] The first aspect embodiment of the application provides a method for improving the anti-deception ability of a UAV based on LLM multi-modal data generation, comprising the following steps: collecting multi-modal original data in the process of a target UAV performing a preset flight task, and performing time synchronization and preprocessing operations on the multi-modal original data to obtain corresponding multi-modal feature data, and encapsulating the multi-modal feature data into unified structured text description information; based on the text description information and a preset attack type, a corresponding standardized Prompt template is constructed, and a pre-constructed large language model is fine-tuned to generate a corresponding large language fine-tuned model; the standardized Prompt template is input into the large language fine-tuned model to output multi-modal deception data in a target format, wherein the multi-modal deception data includes a fake GPS track point column, a disturbed IMU acceleration trend, and a fake image description.

[0008] Optionally, in an embodiment of the present application, after inputting the standardized Prompt template into the large language fine-tuning model to output the multi-modal deception data in the target format, it further includes: obtaining the actual flight trajectory corresponding to the target UAV, and calculating the average offset index between the fake GPS trajectory point column in the multi-modal deception data and the actual flight trajectory, and based on the average offset index, determining the target fake GPS trajectory point column that meets the preset speed change constraint detection requirement; obtaining each inertial data frame corresponding to the target UAV, and performing filtering and alignment operations on each inertial data frame to obtain the corresponding filtered inertial data frame; calculating the Euclidean difference value module length corresponding to each inertial data frame according to the filtered inertial data frame, and calculating the average disturbance amount of all inertial data frames using the Euclidean difference value module length, to perform disturbance consistency verification operation on the disturbance IMU acceleration trend based on the average disturbance amount and the preset maximum tolerance threshold, to generate the target disturbance IMU acceleration trend that meets the preset disturbance consistency verification requirement; performing saliency occlusion extraction operation on the camouflage image description to generate the corresponding camouflage mask area, and calculating the camouflage area and the total image area corresponding to the camouflage mask area and the camouflage image description respectively; based on the camouflage area and the total image area, calculate the corresponding occlusion ratio, and according to the occlusion ratio, perform preset image camouflage rationality verification operation on the camouflage image description to obtain the target camouflage image description that meets the image camouflage rationality verification requirement; based on the target camouflage image description, the target disturbance IMU acceleration trend and the target fake GPS trajectory point column, construct the corresponding deception data set; according to the deception data set and the preset dynamic optimization mechanism, optimize the standardized Prompt template to generate multi-modal deception samples, and through the multi-modal deception samples, perform enhanced anti-deception training and / or performance evaluation operation on the target UAV to obtain the corresponding training data and / or evaluation results, to optimize the performance of the target UAV in executing the preset flight task based on the training data and / or the evaluation results.

[0009] Optionally, in an embodiment of the present application, the step of optimizing the standardized Prompt template according to the spoof data set and a preset dynamic optimization mechanism to generate multi-modal spoof samples, and performing enhanced anti-spoofing training and / or performance evaluation operations on the target UAV through the multi-modal spoof samples comprises: detecting whether the target UAV successfully implements a spoofing attack using the spoof data set in real time, wherein when it is detected that the target UAV successfully implements a spoofing attack, recording multi-modal sensor data and state information of the target UAV when the spoofing attack is successfully implemented to generate attack feedback data according to the multi-modal sensor data and the state information; dynamically adjusting the standardized Prompt template based on the attack feedback data, and generating multi-modal spoof samples according to the dynamically adjusted standardized Prompt template to update the spoof data set through the multi-modal spoof samples; performing enhanced anti-spoofing training and / or performance evaluation operations on the target UAV based on a preset anti-spoofing training strategy, a performance evaluation strategy, and the updated spoof data set.

[0010] Optionally, in an embodiment of the present application, the step of collecting multi-modal original data of the target UAV during execution of a preset flight task and performing time synchronization and preprocessing operations on the multi-modal original data to obtain corresponding multi-modal feature data comprises: obtaining multi-modal original data collected by the target UAV during execution of the preset flight task, wherein the multi-modal original data comprises GPS position information, IMU inertial measurement data, and an onboard camera image sequence; performing millisecond-level alignment operations on the GPS position information, the IMU inertial measurement data, and the onboard camera image sequence in the multi-modal original data to obtain GPS position alignment information, IMU inertial measurement alignment data, and an onboard camera image alignment sequence; performing low-pass filtering processing on the IMU inertial measurement alignment data to obtain corresponding filtered data, and extracting a speed change feature corresponding to the GPS position alignment information and a semantic feature corresponding to the onboard camera image alignment sequence, and performing dimension reduction on the semantic feature to generate corresponding reduced semantic features; constructing the multi-modal feature data based on the filtered data, the speed change feature, and the reduced semantic features.

[0011] The second aspect embodiment of the application provides a device for improving the anti-deception capability of a UAV based on LLM multi-modal data generation, comprising: a preprocessing module configured to collect multi-modal original data of a target UAV during the execution of a preset flight task, and perform time synchronization and preprocessing operations on the multi-modal original data to obtain corresponding multi-modal feature data, and encapsulate the multi-modal feature data into unified structured text description information; a fine-tuning module configured to construct a corresponding standardized Prompt template based on the text description information and a preset attack type, and fine-tune a pre-constructed large language model to generate a corresponding large language fine-tuned model; and an inference module configured to input the standardized Prompt template into the large language fine-tuned model to output multi-modal deception data in a target format, wherein the multi-modal deception data includes a fake GPS track point column, a disturbed IMU acceleration trend, and a fake image description

[0012] Optionally, in an embodiment of the present application, further comprising: a speed change constraint module configured to obtain an actual flight trajectory corresponding to the target UAV after inputting the standardized Prompt template into the large language fine-tuning model to output the multi-modal deception data in the target format, calculate an average offset index between a fake GPS trajectory point column in the multi-modal deception data and the actual flight trajectory, and determine a target fake GPS trajectory point column in the fake GPS trajectory point column that meets a preset speed change constraint detection requirement based on the average offset index; a filtering module configured to obtain each inertial data frame corresponding to the target UAV, and perform filtering and alignment operations on each inertial data frame to obtain a corresponding filtered inertial data frame; a disturbance consistency verification module configured to calculate a Euclidean difference value module length corresponding to each inertial data frame according to the filtered inertial data frame, calculate an average disturbance amount of all inertial data frames using the Euclidean difference value module length, and perform a disturbance consistency verification operation on the disturbance IMU acceleration trend based on the average disturbance amount and a preset maximum tolerance threshold to generate a target disturbance IMU acceleration trend that meets a preset disturbance consistency verification requirement; a calculation module configured to perform a saliency occlusion extraction operation on the camouflage image description to generate a corresponding camouflage mask region, and calculate a camouflage area and a total image area corresponding to the camouflage mask region and the camouflage image description, respectively; a camouflage rationality verification module configured to calculate a corresponding occlusion ratio based on the camouflage area and the total image area, and perform a preset image camouflage rationality verification operation on the camouflage image description according to the occlusion ratio to obtain a target camouflage image description that meets an image camouflage rationality verification requirement; a construction module configured to construct a corresponding deception data set based on the target camouflage image description, the target disturbance IMU acceleration trend, and the target fake GPS trajectory point column; and an optimization module configured to optimize the standardized Prompt template according to the deception data set and a preset dynamic optimization mechanism to generate a multi-modal deception sample, perform an enhanced anti-deception training and / or performance evaluation operation on the target UAV through the multi-modal deception sample to obtain corresponding training data and / or evaluation results, and optimize the performance of the target UAV in performing the preset flight task based on the training data and / or the evaluation results.

[0013] Optionally, in an embodiment of the present application, the optimization module comprises: a detection unit configured to detect in real time whether the target UAV successfully implements a deception attack using the deception data set, wherein, when it is detected that the target UAV successfully implements a deception attack, the multi-modal sensor data and the state information of the target UAV at the time of successful implementation of the deception attack are recorded to generate attack feedback data based on the multi-modal sensor data and the state information; an updating unit configured to dynamically adjust the standardized Prompt template based on the attack feedback data, and generate multi-modal deception samples according to the dynamically adjusted standardized Prompt template, so as to update the deception data set by using the multi-modal deception samples; and a performance evaluation unit configured to perform enhanced anti-deception training and / or performance evaluation operation on the target UAV based on a preset anti-deception training strategy, a performance evaluation strategy, and the updated deception data set.

[0014] Optionally, in an embodiment of the present application, the preprocessing module comprises: an acquisition unit configured to acquire multi-modal original data collected by the target UAV during execution of the preset flight task, wherein the multi-modal original data comprises GPS position information, IMU inertial measurement data, and an onboard camera image sequence; an alignment unit configured to perform millisecond-level alignment operation on the GPS position information, the IMU inertial measurement data, and the onboard camera image sequence in the multi-modal original data to obtain GPS position alignment information, IMU inertial measurement alignment data, and an onboard camera image alignment sequence; a dimension reduction unit configured to perform low-pass filtering processing on the IMU inertial measurement alignment data to obtain corresponding filtered data, extract a speed change feature corresponding to the GPS position alignment information, and extract a semantic feature corresponding to the onboard camera image alignment sequence, and perform dimension reduction on the semantic feature to generate corresponding dimension-reduced semantic feature; and an establishment unit configured to construct the multi-modal feature data based on the filtered data, the speed change feature, and the dimension-reduced semantic feature.

[0015] An electronic device is provided in a third aspect of embodiments of the present application, and comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the program to implement the method for improving anti-deception capability of a UAV based on LLM multi-modal data generation as described in the above embodiments.

[0016] A computer readable storage medium is provided in a fourth aspect of embodiments of the present application, and the computer readable storage medium stores a computer program, and the program is executed by a processor to implement the method for improving anti-deception capability of a UAV based on LLM multi-modal data generation as described above.

[0017] The fifth aspect of the embodiment of the present application provides a computer program product, including a computer program, which is executed to implement the above-mentioned method for improving the anti-deception capability of a UAV based on LLM multi-modal data generation.

[0018] Therefore, the embodiments of the present application have the following beneficial effects:

[0019] The embodiments of the present application can collect multi-modal original data of a target UAV performing a flight task process, and perform time synchronization and preprocessing on the multi-modal original data to obtain corresponding multi-modal feature data, and encapsulate the multi-modal feature data into unified structured text description information; construct a corresponding standardized Prompt template, and input the standardized Prompt template into a large language fine-tuning model to output multi-modal deception data in a target format; perform consistency verification on the multi-modal deception data to construct a deception data set, and then update the deception data set based on a feedback dynamic optimization mechanism, so as to be used by the UAV for subsequent anti-deception training or performance evaluation. The present application can effectively expand the attack scene coverage, reduce the cost of artificially designed samples, and the generated deception samples are closer to the real attack logic. Therefore, the problems of the existing large language model, such as the application being limited to a single mode or an open-loop generation mode, lacking real-time feedback and dynamic optimization of attack effect, and being difficult to continuously improve the anti-deception capability of the UAV, are solved.

[0020] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0021] The above-mentioned and / or additional aspects and advantages of the present application will become apparent and will be easily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, in which:

[0022] Figure 1 A flowchart of a method for improving the anti-deception capability of a UAV based on LLM multi-modal data generation according to an embodiment of the present application is provided.

[0023] Figure 2 An execution logic schematic diagram of a method for improving the anti-deception capability of a UAV based on LLM multi-modal data generation according to an embodiment of the present application is provided.

[0024] Figure 3 An example diagram of a device for improving the anti-deception capability of a UAV based on LLM multi-modal data generation according to an embodiment of the present application is provided.

[0025] Figure 4 A structural schematic diagram of an electronic device according to an embodiment of the present application is provided.

[0026] 10. A device for improving the anti-deception capability of a UAV based on LLM multi-modal data generation; 100, a preprocessing module, 200, a fine-tuning module, 300, an inference module; 401, a memory, 402, a processor, 403, a communication interface. DETAILED DESCRIPTION

[0027] Embodiments of the present application are described below in detail, examples of which are shown in the drawings, wherein the same or similar reference signs represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and are intended to explain the present application, and cannot be understood as a limitation of the present application.

[0028] A method for improving the anti-deception capability of a UAV based on LLM multi-modal data generation is described below with reference to the drawings. In view of the problems mentioned in the above background art, the present application provides a method for improving the anti-deception capability of a UAV based on LLM multi-modal data generation. In this method, multi-modal raw data of a target UAV performing a flight task is collected, and time synchronization and preprocessing are performed on the multi-modal raw data to obtain corresponding multi-modal feature data, and the multi-modal feature data is packaged into unified structured text description information. A corresponding standardized Prompt template is constructed and input into a large language fine-tuning model to output multi-modal deception data in a target format. The multi-modal deception data is subjected to consistency test to construct a deception data set, and then based on a feedback dynamic optimization mechanism, the deception data set is updated for subsequent use by the UAV to enhance anti-deception training or performance evaluation. The present application can effectively expand the attack scene coverage, reduce the cost of manually designed samples, and the generated deception samples are closer to the real attack logic. Thus, the problems of the existing large language model, such as the application being limited to single mode or open loop generation mode, lack of real-time feedback and dynamic optimization of attack effect, and difficulty in continuously improving the anti-deception capability of the UAV, are solved.

[0029] Specifically, Figure 1 A flowchart of a method for improving the anti-deception capability of a UAV based on LLM multi-modal data generation is provided by the embodiments of the present application.

[0030] As Figure 1 shown, the method for improving the anti-deception capability of a UAV based on LLM multi-modal data generation includes the following steps:

[0031] In step S101, multi-modal raw data during execution of a preset flight task by a target UAV is collected, and time synchronization and preprocessing operations are performed on the multi-modal raw data to obtain corresponding multi-modal feature data, and the multi-modal feature data is packaged into unified structured text description information.

[0032] The embodiment of the application can first acquire multi-modal original data in a normal flight task of a UAV, and millisecond-level alignment and standardization preprocessing are performed on the multi-modal data, so that corresponding multi-modal feature data is obtained.

[0033] Then, the embodiment of the application can abstract the multi-modal feature data obtained after the above processing into unified structured text description information, thereby providing reliable data basis for subsequent generation of multi-modal spoofing data.

[0034] Optionally, in an embodiment of the application, multi-modal original data in a process in which a target UAV executes a preset flight task is acquired, and time synchronization and preprocessing operations are performed on the multi-modal original data, so that corresponding multi-modal feature data is obtained, comprising: acquiring multi-modal original data collected by the target UAV in a process in which the target UAV executes a preset flight task, wherein the multi-modal original data comprises GPS position information, IMU inertial measurement data and an image sequence collected by an onboard camera; performing millisecond-level alignment operations on the GPS position information, the IMU inertial measurement data and the image sequence collected by the onboard camera in the multi-modal original data, so as to obtain GPS position alignment information, IMU inertial measurement alignment data and an image alignment sequence of the onboard camera; performing low-pass filtering processing on the IMU inertial measurement alignment data, so as to obtain corresponding filtered data, and extracting a speed change feature corresponding to the GPS position alignment information and a semantic feature corresponding to the image alignment sequence of the onboard camera, and performing dimension reduction on the semantic feature, so as to generate corresponding reduced dimension semantic feature; and constructing the multi-modal feature data based on the filtered data, the speed change feature and the reduced dimension semantic feature.

[0035] Specifically, the embodiment of the application can first acquire multi-modal original data in a normal flight task of a UAV, comprising GPS position information, IMU inertial measurement data and an image sequence collected by an onboard camera; secondly, the embodiment of the application can perform millisecond-level alignment on the multi-modal data by using a unified time synchronization mechanism (such as timestamp synchronization based on a ROS system), so as to ensure consistency of subsequent fusion processing.

[0036] Further, the embodiment of the application can use standardization preprocessing technology to perform first-order difference encoding on GPS trajectory data to extract a speed change feature, perform Butterworth low-pass filtering on IMU data to reduce noise, and extract high-level semantic features from image data by using a lightweight convolutional neural network (such as MobileNetV2), and combine principal component analysis for dimension reduction, so that corresponding multi-modal feature data is obtained.

[0037] Then, the embodiment of the application can abstract the multi-modal feature data obtained after the above processing into unified structured text description information, that is, encapsulate all processed features into structured natural language text description, thereby providing reliable data basis for subsequent generation of multi-modal spoofing data.

[0038] In step S102, based on the text description information and the preset attack type, a corresponding standardized Prompt template is constructed, and a pre-constructed large language model is fine-tuned to generate a corresponding large language fine-tuned model.

[0039] In step S103, the standardized Prompt template is input into the large language fine-tuned model to output multi-modal deception data in a target format, wherein the multi-modal deception data includes a fake GPS track point column, a disturbed IMU acceleration trend, and a fake image description.

[0040] Further, the embodiments of the present application also need to design a standardized Prompt template for a set attack type (such as GPS position deception, IMU inertia disturbance, or visual camouflage), to clearly describe the current environment state, the expected deception target, and the constraint condition of the generated data. The Prompt text is organized in a standard format, including environment summary, attack target description, and output format requirement, so as to ensure that the LLM can accurately understand the task background and generation instruction;

[0041] Secondly, the embodiments of the present application can select a large language model LLM based on LLaMA or Mistral architecture, and use LoRA (Low-Rank Adaptation) technology to fine-tune the LLM to quickly adapt to the ability to generate targeted deception samples, and control the inference temperature (such as temperature=0.5) to balance the generation quality and diversity.

[0042] After that, the embodiments of the present application can input the standardized Prompt template into the fine-tuned LLM (i.e. large language fine-tuned model) to infer and generate multi-modal deception data, which includes fake GPS track point column, disturbed IMU acceleration trend, and fake image description. The embodiments of the present application can use a unified JSON structure (i.e. target format) to format the multi-modal deception data for output, thereby providing reliable data support for subsequent analysis and verification of the deception data.

[0043] Optionally, in an embodiment of the present application, after inputting the standardized Prompt template into the large language fine-tuning model to output the multi-modal deception data in the target format, it further includes: obtaining the actual flight trajectory corresponding to the target UAV, and calculating the average offset index between the fake GPS trajectory point column in the multi-modal deception data and the actual flight trajectory, and based on the average offset index, determining the target fake GPS trajectory point column that meets the preset speed change constraint detection requirement in the fake GPS trajectory point column; obtaining each inertial data frame corresponding to the target UAV, and performing filtering and alignment operations on each inertial data frame to obtain the corresponding filtered inertial data frame; calculating the Euclidean difference value module length corresponding to each inertial data frame according to the filtered inertial data frame, and calculating the average disturbance of all inertial data frames using the Euclidean difference value module length, to perform disturbance consistency verification operation on the disturbed IMU acceleration trend based on the average disturbance and the preset maximum tolerance threshold, to generate the target disturbed IMU acceleration trend that meets the preset disturbance consistency verification requirement; performing saliency occlusion extraction operation on the fake image description to generate the corresponding fake mask area, and calculating the fake area and the total image area corresponding to the fake mask area and the fake image description respectively; based on the fake area and the total image area, calculate the corresponding occlusion ratio, and perform preset image fake rationality verification operation on the fake image description according to the occlusion ratio to obtain the target fake image description that meets the image fake rationality verification requirement; based on the target fake image description, the target disturbed IMU acceleration trend and the target fake GPS trajectory point column, construct the corresponding deception data set; according to the deception data set and the preset dynamic optimization mechanism, optimize the standardized Prompt template to generate multi-modal deception samples, and perform enhanced anti-deception training and / or performance evaluation operation on the target UAV through the multi-modal deception samples to obtain the corresponding training data and / or evaluation results, to optimize the performance of the target UAV in executing the preset flight task based on the training data and / or evaluation results.

[0044] It should be noted that the embodiments of the present application can design a consistency verification mechanism for multi-modal output to ensure the physical consistency and application reliability of the generated deception samples. The consistency verification mechanism includes speed change constraint detection for GPS trajectory continuity to ensure that the trajectory offset conforms to the physically feasible range; consistency analysis of IMU acceleration change and trajectory offset direction to prevent logical contradictions; environmental rationality verification of image fake description to ensure that the visual disturbance content matches the original flight scene.

[0045] In actual execution process, the consistency verification mechanism described above is used to constrain and evaluate the disturbance of GPS trajectory, IMU inertial information and image data, and only data that passes all consistency verification can be included in the deception data set, as follows:

[0046] (1) GPS track offset verification:

[0047] Regarding trajectory consistency, the embodiments of this application first calculate the average offset index D between the forged trajectory and the real flight trajectory (i.e., the actual flight trajectory). offset As shown in the following formula:

[0048]

[0049] in, ( x i , y i )and( x i ′, y i ′) represent the first and second digits respectively. i The original GPS coordinates and generated GPS coordinates at each time point; N represents the total number of GPS data points contained in the trajectory.

[0050] GPS trajectory data is automatically collected through GPS message recordings of the ROS system, and the original and generated trajectories are paired using timestamp alignment. In the embodiments of this application, the geodetic distance between two points can be calculated using the geopy or haversine libraries in Python, and the average offset value of each segment can be obtained.

[0051] To ensure that the trajectory disturbance is both deceptive and within the acceptable range of the flight control system, the embodiments of this application require D. offset Falling within the set range [D min D max Within this range, the trajectory disturbance amplitude is ensured to be reasonable, and only forged samples (i.e., target forged GPS trajectory point series) with offsets within this range are retained.

[0052] (2) IMU acceleration perturbation consistency check:

[0053] Secondly, embodiments of this application may use the acceleration perturbation rate Raccel to characterize the degree of perturbation, which is used to measure the average difference between generated data and real data in three-axis acceleration, as shown in the following formula:

[0054]

[0055] in Indicates the original IMU sensor at the first i The three-axis acceleration values ​​of the frame; For the first i A frame forged 3D acceleration vector.

[0056] Note that the above data source is the inertial data frame in the IMU log, obtained by ROS (Robot Operating System) subscribing / mavros / imu / data; the data is smoothed by Butterworth low-pass filtering and aligned with the generated sample, and the Euclidean difference modulus is calculated frame by frame. The average value is the disturbance rate; the embodiment of the application requires an average disturbance R accel Less than the maximum tolerance threshold ε accel (typically set to within 0.5g), if it exceeds the set threshold, it is considered as an abnormal sample that does not conform to the physical law of flight and is rejected, thereby obtaining the target disturbance IMU acceleration trend.

[0057] (3) Image camouflage reasonableness test:

[0058] As an implementable way, the embodiment of the application can use YOLOv8 or DeepLabV3+ semantic segmentation model to extract saliency occlusion of the corresponding region of the fake image or image description, form a camouflage mask region, and C cover That is, the ratio of the camouflage area to the total image area (i.e., the occlusion ratio), which is shown as follows:

[0059]

[0060] Where, A cover is the number of mask region pixels (i.e., the area of the camouflage region); A total is the total number of pixels of the whole image (i.e., the total area of the image).

[0061] Therefore, the embodiment of the application can process the image through OpenCV or torchvision.transforms, extract the camouflage region and automatically calculate the occlusion ratio. In order to balance the effectiveness and perceptibility of interference, the embodiment of the application requires the occlusion ratio C cover Control in the range of 10% to 40%, so as to ensure that the camouflage interference is effective and does not destroy the overall image understanding, avoid that the ratio may not be able to induce misjudgment, and the ratio may destroy the overall environmental perception logic.

[0062] In the process of generating deception samples, the embodiment of the application can introduce a dynamic optimization mechanism based on feedback. When the unmanned aerial vehicle system detects that the deception attack is successful (such as task anomaly, trajectory deviation, sensor anomaly), the multi-modal sensor data and system state at the corresponding time can be used as feedback information to record the deception attack characteristics. The feedback information includes attack sample type, deception disturbance mode and environment description, etc.

[0063] In addition, the embodiments of the present application can dynamically adjust the deception sample generation strategy according to the feedback information, including optimizing the Prompt prompt content, strengthening the attack strength, introducing new attack combinations or adjusting the scene complexity, to guide the LLM to generate more diverse and challenging multi-modal deception samples; the improved deception samples generated are updated to the output deception data set after consistency verification, for subsequent use by the unmanned aerial vehicle system for anti-deception capability training and performance verification, thereby improving the performance of the unmanned aerial vehicle in executing the preset flight task.

[0064] Optionally, in an embodiment of the present application, according to the deception data set and the preset dynamic optimization mechanism, the standardized Prompt template is optimized to generate multi-modal deception samples, and the target unmanned aerial vehicle is subjected to enhanced anti-deception training and / or performance evaluation operation through the multi-modal deception samples, including: detecting whether the target unmanned aerial vehicle successfully implements deception attack using the deception data set in real time, wherein when it is detected that the target unmanned aerial vehicle successfully implements deception attack, the multi-modal sensor data and the state information of the target unmanned aerial vehicle at the time of successful implementation of deception attack are recorded to generate attack feedback data according to the multi-modal sensor data and the state information; based on the attack feedback data, the standardized Prompt template is dynamically adjusted, and multi-modal deception samples are generated according to the dynamically adjusted standardized Prompt template, so as to update the deception data set through the multi-modal deception samples; based on the preset anti-deception training strategy, performance evaluation strategy and updated deception data set, the target unmanned aerial vehicle is subjected to enhanced anti-deception training and / or performance evaluation operation.

[0065] It should be noted that during the flight or simulation of the unmanned aerial vehicle, if it is detected that the deception attack is successful (such as trajectory deviation, task anomaly, sensor state anomaly), the multi-modal sensor data and system state information at the corresponding time are recorded to form attack feedback data; the embodiments of the present application can dynamically adjust the standardized Prompt design according to the attack feedback data to strengthen the attack strength, combine attack types or increase the environmental complexity, so as to guide the LLM to generate more diverse and challenging multi-modal deception samples.

[0066] After that, the improved deception samples that pass the consistency test can be updated to the output deception data set for subsequent use by the unmanned aerial vehicle system for anti-deception capability training and performance verification.

[0067] As a way that can be implemented, in order to further demonstrate the significant superiority in anti-deception training, the embodiments of the present application can theoretically explain and verify the effectiveness of the multi-modal deception samples generated and the positive effect on the training effect from the angles of mathematical modeling and quantifiable indicators, as follows:

[0068] 1. Entropy analysis of sample diversity improvement:

[0069] In the specific implementation process, the distribution of the traditional artificial constructed data can be set as P human (x), the distribution of the multi-modal spoofing sample generated by the embodiment of the application is P LLM (x), and from the perspective of information entropy, the following can be defined:

[0070]

[0071]

[0072] wherein, and respectively represent the information entropy of the traditional artificial constructed spoofing sample and the information entropy of the multi-modal spoofing sample generated by the large language model; P LLM (x) represents the probability distribution of the multi-modal spoofing sample generated by the large language model in the feature space; P human (x) represents the probability distribution of the traditional artificial constructed spoofing sample in the feature space.

[0073] It should be noted that the embodiment of the application can compare the information entropy H(P LLM )>H(P human ) of the two kinds of samples through the discrete feature space, and reflect that the diversity of the LLM generated sample in the feature dimension space is more abundant. In actual application, the embodiment of the application can estimate the entropy distribution by using the bucket method through the discrete division of the generated sample in the feature dimension.

[0074] 2, sample distribution fitting reality index:

[0075] The embodiment of the application can project the generated multi-modal spoofing sample and the real sample into a unified high-dimensional feature space (such as an image embedding or trajectory embedding space), and then compare their mean and covariance. The mean and covariance of the generated sample in the embedding space are denoted as (μ g ,Σ g ), the distribution of the real attack sample is (μ r ,Σ r ), and the Frechet Inception Distance (FID, image quality index generated by the generated model) can be calculated:

[0076]

[0077] wherein, represents the degree of similarity between the generated data and the real data distribution, and the smaller the FID value, the closer the generated data is to the distribution of the real data; represents the average feature vector of the real attack sample in the high-dimensional feature space; represents the average feature vector of the LLM generated spoofing sample in the high-dimensional feature space; (·) denotes the trace of the covariance matrix; denotes the covariance matrix of each dimension of the feature space of the real sample; denotes the covariance matrix of each dimension of the feature space of the generated sample.

[0078] Therefore, the embodiments of the present application can dynamically reduce the FID value by optimizing the Prompt and feedback mechanism, so that the generated sample is closer to the real attack data distribution.

[0079] 3. Training performance of anti-spoofing robustness improvement:

[0080] It can be understood that performance verification can be performed by constructing an adversarial sample test set, training with / without LLM samples under the same model architecture and hyperparameters, and comparing the accuracy improvement.

[0081] In the model training phase, the embodiments of the present application can introduce LLM generated samples, and assume that the adversarial attack accuracy of the model after training is Acc baseline , and the accuracy of the model without LLM generated samples is Acc LLM , then the performance improvement metric is:

[0082] ΔAcc = Acc LLM - Acc baseline

[0083] wherein, denotes the adversarial attack detection accuracy of the model; Acc LLM denotes the adversarial attack detection accuracy of the model after introducing LLM generated samples for training; Acc baseline denotes the adversarial attack detection accuracy of the baseline model.

[0084] Therefore, in the training set containing LLM samples, the embodiments of the present application can determine the enhancement of model robustness according to the detection rate improvement data of the unmanned aerial vehicle system for complex deception scenarios such as camouflage images and trajectory drift.

[0085] It can be understood that the embodiments of the present application can effectively expand the coverage of attack scenarios, generate deception samples closer to the real attack logic, reduce the cost of artificially designed samples, and be well applied to unmanned aerial vehicle system anti-deception training set enhancement, deception attack script generation in simulation environment and other scenarios.

[0086] The following describes the execution logic of an unmanned aerial vehicle anti-deception ability improvement method based on LLM multi-modal data generation of the present application in conjunction with the accompanying drawings.

[0087] Figure 2 FIG. 1 is a schematic diagram of the execution logic of an unmanned aerial vehicle anti-deception ability improvement method based on LLM multi-modal data generation of the present application. As shown in FIG. 1, the execution logic of the method includes the following steps: Figure 2As shown, the method for improving the anti-deception ability of the unmanned aerial vehicle based on the LLM multi-modal data generation of the present application is executed as follows:

[0088] S201: Collect multi-modal original data (GPS position information, IMU inertial measurement data, and image sequences collected by the on-board camera);

[0089] S202: Based on the preset differential, filtering and dimension reduction strategy, time synchronization and preprocessing are performed on the multi-modal original data to obtain multi-modal feature data;

[0090] S203: Encapsulate the multi-modal feature data into unified structured text description information;

[0091] S204: For the expected attack type, a corresponding standardized Prompt template is constructed;

[0092] S205: Input the standardized Prompt template into the fine-tuned large language model to infer and generate multi-modal deception data;

[0093] S206: Output multi-modal deception data in a unified JSON structure;

[0094] S207: Whether the multi-modal deception data passes the consistency test, if it passes the consistency test, go to S208, otherwise go to S204;

[0095] S208: Use the multi-modal deception data that passes the consistency test to construct a deception sample data set;

[0096] S209: Based on the deception sample data set, detect whether there is an attack success record feedback during the execution of the task of the unmanned aerial vehicle, if there is an attack success record feedback, go to S2010, otherwise end the execution process of the method for improving the anti-deception ability of the unmanned aerial vehicle based on the LLM multi-modal data generation of the present application;

[0097] S2010: Based on the attack success record feedback, perform subsequent anti-deception ability training and performance verification of the unmanned aerial vehicle system.

[0098] According to the unmanned aerial vehicle anti-deception ability improving method based on LLM multi-modal data generation provided in the embodiment of the application, multi-modal original data in the process of a target unmanned aerial vehicle performing a flight task is collected, time synchronization and preprocessing are performed on the multi-modal original data to obtain corresponding multi-modal feature data, and the multi-modal feature data is packaged into unified structured text description information; a corresponding standardized Prompt template is constructed, and the Prompt template is input into a large language fine-tuning model to output multi-modal deception data in a target format; consistency verification is performed on the multi-modal deception data to construct a deception data set, and then based on a feedback dynamic optimization mechanism, the deception data set is updated for subsequent use by the unmanned aerial vehicle to enhance anti-deception training or performance evaluation. The application can effectively expand the attack scene coverage range, reduce the cost of artificially designed samples, and the generated deception samples are closer to the real attack logic.

[0099] Secondly, a kind of unmanned aerial vehicle anti-deception ability improving device based on LLM multi-modal data generation according to the embodiment of the application is described with reference to the drawings.

[0100] Figure 3 It is a kind of unmanned aerial vehicle anti-deception ability improving device based on LLM multi-modal data generation of the embodiment of the application.

[0101] As Figure 3 The unmanned aerial vehicle anti-deception ability improving device 10 based on LLM multi-modal data generation includes a preprocessing module 100, a fine-tuning module 200 and an inference module 300.

[0102] The preprocessing module 100 is configured to collect multi-modal original data in the process of a target unmanned aerial vehicle performing a preset flight task, perform time synchronization and preprocessing on the multi-modal original data to obtain corresponding multi-modal feature data, and package the multi-modal feature data into unified structured text description information.

[0103] The fine-tuning module 200 is configured to construct a corresponding standardized Prompt template based on the text description information and a preset attack type, fine-tune a pre-constructed large language model to generate a corresponding large language fine-tuning model.

[0104] The inference module 300 is configured to input the standardized Prompt template into the large language fine-tuning model to output multi-modal deception data in a target format, wherein the multi-modal deception data includes a fake GPS track point column, a disturbed IMU acceleration trend and a fake image description.

[0105] Optionally, in an embodiment of the present application, the unmanned aerial vehicle anti-deception capability improving device 10 based on LLM multi-modal data generation of the embodiment of the present application further comprises a speed change constraint module, a filtering module, a disturbance consistency verification module, a calculation module, a camouflage rationality verification module, a construction module and an optimization module.

[0106] The speed change constraint module is configured to obtain an actual flight trajectory corresponding to the target unmanned aerial vehicle after inputting the standardized Prompt template into the large language fine-tuning model to output multi-modal deception data in a target format, calculate an average offset index between a fake GPS trajectory point column in the multi-modal deception data and the actual flight trajectory, and determine a target fake GPS trajectory point column in the fake GPS trajectory point column that meets a preset speed change constraint detection requirement based on the average offset index.

[0107] The filtering module is configured to obtain each inertial data frame corresponding to the target unmanned aerial vehicle, and perform filtering and alignment operations on each inertial data frame to obtain a corresponding filtered inertial data frame.

[0108] The disturbance consistency verification module is configured to calculate a Euclidean difference modulus corresponding to each inertial data frame based on the filtered inertial data frame, calculate an average disturbance amount of all inertial data frames using the Euclidean difference modulus, and perform a disturbance consistency verification operation on the disturbance IMU acceleration trend based on the average disturbance amount and a preset maximum tolerance threshold to generate a target disturbance IMU acceleration trend that meets a preset disturbance consistency verification requirement.

[0109] The calculation module is configured to perform a saliency occlusion extraction operation on the camouflage image description to generate a corresponding camouflage mask region, and calculate a camouflage region area and a total image area corresponding to the camouflage mask region and the camouflage image description, respectively.

[0110] The camouflage rationality verification module is configured to calculate a corresponding occlusion ratio based on the camouflage region area and the total image area, and perform a preset image camouflage rationality verification operation on the camouflage image description according to the occlusion ratio to obtain a target camouflage image description that meets an image camouflage rationality verification requirement.

[0111] The construction module is configured to construct a corresponding deception data set based on the target camouflage image description, the target disturbance IMU acceleration trend and the target fake GPS trajectory point column.

[0112] The optimization module is configured to optimize the standardized Prompt template according to the spoofing data set and a preset dynamic optimization mechanism, to generate a multi-modal spoofing sample, and to perform enhanced anti-spoofing training and / or performance evaluation operations on the target UAV through the multi-modal spoofing sample, to obtain corresponding training data and / or evaluation results, and to optimize the performance of the target UAV in performing the preset flight task based on the training data and / or the evaluation results.

[0113] Optionally, in an embodiment of the present application, the optimization module comprises a detection unit, an updating unit, and a performance evaluation unit.

[0114] The detection unit is configured to detect in real time whether the target UAV successfully implements a spoofing attack using the spoofing data set, and to record multi-modal sensor data and state information of the target UAV when the spoofing attack is successfully implemented, to generate attack feedback data according to the multi-modal sensor data and the state information.

[0115] The updating unit is configured to dynamically adjust the standardized Prompt template based on the attack feedback data, and to generate a multi-modal spoofing sample according to the dynamically adjusted standardized Prompt template, to update the spoofing data set through the multi-modal spoofing sample.

[0116] The performance evaluation unit is configured to perform enhanced anti-spoofing training and / or performance evaluation operations on the target UAV based on a preset anti-spoofing training strategy, a performance evaluation strategy, and the updated spoofing data set.

[0117] Optionally, in an embodiment of the present application, the preprocessing module 100 comprises an acquisition unit, an alignment unit, a dimension reduction unit, and an establishment unit.

[0118] The acquisition unit is configured to acquire multi-modal original data collected by the target UAV during the execution of the preset flight task, wherein the multi-modal original data comprises GPS position information, IMU inertial measurement data, and an onboard camera image sequence.

[0119] The alignment unit is configured to perform millisecond-level alignment operations on the GPS position information, the IMU inertial measurement data, and the onboard camera image sequence in the multi-modal original data, to obtain GPS position alignment information, IMU inertial measurement alignment data, and an onboard camera image alignment sequence.

[0120] The dimension reduction unit is configured to perform low-pass filtering processing on the IMU inertial measurement alignment data to obtain corresponding filtered data, to extract a speed change feature corresponding to the GPS position alignment information and a semantic feature corresponding to the onboard camera image alignment sequence, and to perform dimension reduction on the semantic feature to generate corresponding dimension-reduced semantic features.

[0121] The establishing unit is configured to construct multi-modal feature data based on the filtered data, the speed change feature, and the reduced dimension semantic feature.

[0122] It should be noted that the foregoing description of the method for improving the anti-deception capability of the unmanned aerial vehicle based on the LLM multi-modal data generation also applies to the device for improving the anti-deception capability of the unmanned aerial vehicle based on the LLM multi-modal data generation, and details are not repeated here.

[0123] The device for improving the anti-deception capability of the unmanned aerial vehicle based on the LLM multi-modal data generation according to the embodiments of the present application comprises a preprocessing module 100 configured to collect multi-modal original data in a process in which a target unmanned aerial vehicle performs a preset flight task, and perform time synchronization and preprocessing operations on the multi-modal original data to obtain corresponding multi-modal feature data, and encapsulate the multi-modal feature data into text description information in a unified structured form; a fine-tuning module 200 configured to construct a standardized Prompt template based on the text description information and a preset attack type, and fine-tune a pre-constructed large language model to generate a corresponding large language fine-tuned model; and an inference module 300 configured to input the standardized Prompt template into the large language fine-tuned model to output multi-modal deception data in a target format, wherein the multi-modal deception data comprises a fake GPS track point column, a disturbed IMU acceleration trend, and a disguised image description. The present application can effectively expand the attack scene coverage range, reduce the cost of artificially designed samples, and generate deception samples that are closer to real attack logic.

[0124] Figure 4 The electronic device provided by the embodiments of the present application is shown in the structural schematic diagram. The electronic device can comprise:

[0125] The memory 401, the processor 402, and the computer program stored in the memory 401 and executable on the processor 402.

[0126] The processor 402 implements the method for improving the anti-deception capability of the unmanned aerial vehicle based on the LLM multi-modal data generation provided in the above embodiments when executing the program.

[0127] Further, the electronic device further comprises:

[0128] The communication interface 403 is configured to communicate between the memory 401 and the processor 402.

[0129] The memory 401 is configured to store the computer program executable on the processor 402.

[0130] The memory 401 can contain a high-speed RAM memory, and can also include a non-volatile memory, such as at least one disk memory.

[0131] If the memory 401, the processor 402 and the communication interface 403 are implemented independently, the communication interface 403, the memory 401 and the processor 402 can be connected to each other through a bus and complete communication between each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For convenience of representation, Figure 4 In the figure, only one thick line is used to represent the bus, but it does not mean that there is only one bus or only one type of bus.

[0132] Optionally, in a specific implementation, if the memory 401, the processor 402 and the communication interface 403 are integrated on a chip, the memory 401, the processor 402 and the communication interface 403 can complete communication between each other through an internal interface.

[0133] The processor 402 can be a Central Processing Unit (CPU), or an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement one or more embodiments of the present application.

[0134] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the method for improving anti-deception ability of a UAV based on LLM multi-modal data generation.

[0135] The embodiment of the present application also provides a computer program product, which includes a computer program, and the computer program is executed to implement the method for improving anti-deception ability of a UAV based on LLM multi-modal data generation.

[0136] In the description of the application, reference to "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" means that a particular feature, structure, material, or characteristic being described is included in at least one embodiment or example of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment or example. Furthermore, the described specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples. In addition, the usage of "N" means at least two, for example, two, three or the like, unless explicitly stated otherwise.

[0137] Furthermore, the terms "first", "second", or the like, are used merely as a designation of certain elements or features of the application, and do not imply or connote relative importance or a specific order of precedence. Thus, features defined with "first", "second", etc. can include at least one of the features, either explicitly or implicitly.

[0138] Any process or method descriptions or blocks in flow charts or otherwise described herein represent embodiments of modules, segments, or portions of code which include one or more executable instructions for implementing specific logic functions or steps, and alternate implementations are possible. In some embodiments, the processes or methods described in flow charts or otherwise described herein are not necessarily performed in the order shown or discussed, including, for example, performing or depending from other operations or stages, in parallel, in reverse order, or in other orders.

[0139] The logic and / or steps represented in the flowcharts and / or described herein, for example, can be considered as a sequence of executable instructions stored in a computer readable medium, which can be executed by an instruction execution system, apparatus or device, such as a computer-based system, a processor-based system, or other system that can fetch the instructions from the instruction execution system, apparatus or device and execute the instructions, or a combination of the above. For the purposes of this specification, a "computer readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus or device. The computer readable medium can be a computer readable storage medium or a computer readable signal medium. The computer readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a propagation medium. The computer readable signal medium can include, but is not limited to, a computer readable medium that facilitates transfer of the program from one place to another. A specific example of a computer readable medium is a non-transitory computer-readable storage medium. A specific example of a computer readable signal medium is a source or destination of the computer readable medium. Another specific example of a computer readable signal medium is a computer readable signal travelling through space. Thus, a computer readable medium can take many forms of hardware to carry out the program for use by or in connection with the instruction execution system, apparatus or device.

[0140] It should be understood that aspects of the application can be implemented in hardware, software, firmware or a combination thereof. In the above embodiments, the N steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented in hardware and in another embodiment, the hardware can be implemented with any or a combination of the following technologies, which are all well known in the art: a discrete logic circuit(s) having logic gates for implementing logic functions upon an application of data signals, an application specific integrated circuit having appropriate combinational logic gates, a programmable gate array(s) (PGA), a field programmable gate array (FPGA), etc.

[0141] Those of skill in the art would understand that the steps carried out in the above-mentioned embodiments can be implemented by a program instructing the relevant hardware to complete all or part of the steps, and the program can be stored in a computer readable storage medium. When the program is executed, it includes one of the steps of the method embodiments or a combination thereof.

[0142] In addition, each of the functional units in the various embodiments of the present application can be integrated in one processing module, or each of the units can be physically present separately, or two or more units can be integrated in one module. The integrated module can be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer readable storage medium.

[0143] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. A method for improving the anti-spoofing capability of unmanned aerial vehicles (UAVs) based on LLM multimodal data generation, characterized in that, Includes the following steps: Collect multimodal raw data during the execution of a preset flight mission by the target UAV, and perform time synchronization and preprocessing operations on the multimodal raw data to obtain corresponding multimodal feature data, and encapsulate the multimodal feature data into unified structured text description information; Based on the text description information and the preset attack type, a corresponding standardized Prompt template is constructed, and the pre-constructed large language model is fine-tuned to generate a corresponding large language fine-tuning model. The standardized Prompt template is input into the large language fine-tuning model to output multimodal spoofing data in the target format, wherein the multimodal spoofing data includes fake GPS trajectory point series, perturbed IMU acceleration trends and spoofed image descriptions; The method further includes, after inputting the standardized Prompt template into the large language fine-tuning model to output the multimodal spoofing data in the target format, the following steps: The actual flight trajectory corresponding to the target UAV is obtained, and the average offset index between the fake GPS trajectory point series in the multimodal deception data and the actual flight trajectory is calculated. Based on the average offset index, the target fake GPS trajectory point series that meets the preset speed change constraint detection requirements is determined. Each inertial data frame corresponding to the target UAV is acquired, and each inertial data frame is filtered and aligned to obtain the corresponding filtered inertial data frame. The Euclidean difference magnitude corresponding to each inertial data frame is calculated based on the filtered inertial data frame, and the average disturbance amount of all inertial data frames is calculated using the Euclidean difference magnitude. Based on the average disturbance amount and the preset maximum tolerance threshold, a disturbance consistency check operation is performed on the disturbance IMU acceleration trend to generate a target disturbance IMU acceleration trend that meets the preset disturbance consistency check requirements. A saliency occlusion extraction operation is performed on the camouflaged image description to generate a corresponding camouflage mask region, and the area of ​​the camouflage mask region and the camouflage region corresponding to the camouflage image description and the total image area are calculated respectively. Based on the camouflage area and the total image area, the corresponding occlusion ratio is calculated, and a preset image camouflage rationality check operation is performed on the camouflage image description according to the occlusion ratio, so as to obtain a target camouflage image description that meets the requirements of the image camouflage rationality check. Based on the target camouflage image description, the target perturbation IMU acceleration trend, and the target fake GPS trajectory point sequence, a corresponding deception dataset is constructed; Based on the deception dataset and the preset dynamic optimization mechanism, the standardized Prompt template is optimized to generate multimodal deception samples. The target UAV is then subjected to enhanced anti-deception training and / or performance evaluation operations using the multimodal deception samples to obtain corresponding training data and / or evaluation results. Based on the training data and / or the evaluation results, the performance of the target UAV in performing the preset flight mission is optimized.

2. The method for improving the anti-spoofing capability of UAVs based on LLM multimodal data generation according to claim 1, characterized in that, The step of optimizing the standardized Prompt template based on the deception dataset and a preset dynamic optimization mechanism to generate multimodal deception samples, and then using the multimodal deception samples to perform enhanced anti-deception training and / or performance evaluation on the target UAV, includes: Real-time detection of whether the target drone successfully carries out a deception attack using the deception dataset, wherein, when the target drone is detected to have successfully carried out a deception attack, multimodal sensor data and the state information of the target drone at the time of successful deception attack are recorded, so as to generate attack feedback data based on the multimodal sensor data and the state information; Based on the attack feedback data, the standardized Prompt template is dynamically adjusted, and multimodal spoofing samples are generated according to the dynamically adjusted standardized Prompt template, so as to update the spoofing dataset through the multimodal spoofing samples; Based on the preset anti-spoofing training strategy, performance evaluation strategy, and updated deception dataset, the target UAV is subjected to enhanced anti-spoofing training and / or performance evaluation operations.

3. The method for improving the anti-spoofing capability of UAVs based on LLM multimodal data generation according to claim 1, characterized in that, The process involves collecting multimodal raw data from the target UAV during its pre-defined flight mission, and performing time synchronization and preprocessing operations on the multimodal raw data to obtain corresponding multimodal feature data, including: Acquire multimodal raw data collected by the target UAV during the execution of the preset flight mission, wherein the multimodal raw data includes GPS location information, IMU inertial measurement data and airborne camera image sequences; A millisecond-level alignment operation is performed on the GPS location information, the IMU inertial measurement data, and the airborne camera image sequence in the multimodal raw data to obtain GPS location alignment information, IMU inertial measurement alignment data, and airborne camera image alignment sequence; The IMU inertial measurement alignment data is subjected to low-pass filtering to obtain the corresponding filtered data. The velocity change features corresponding to the GPS position alignment information and the semantic features corresponding to the airborne camera image alignment sequence are extracted. The semantic features are then dimensionality-reduced to generate the corresponding dimensionality-reduced semantic features. The multimodal feature data is constructed based on the filtered data, the velocity change features, and the dimensionality reduction semantic features.

4. A device for enhancing the anti-spoofing capability of unmanned aerial vehicles (UAVs) based on LLM multimodal data generation, characterized in that, include: The preprocessing module is used to collect multimodal raw data during the execution of a preset flight mission by the target UAV, and to perform time synchronization and preprocessing operations on the multimodal raw data to obtain corresponding multimodal feature data, and to encapsulate the multimodal feature data into unified structured text description information. The fine-tuning module is used to construct a corresponding standardized Prompt template based on the text description information and the preset attack type, and to fine-tune the pre-constructed large language model to generate a corresponding large language fine-tuning model. The inference module is used to input the standardized Prompt template into the large language fine-tuning model to output multimodal spoofing data in the target format, wherein the multimodal spoofing data includes fake GPS trajectory point series, perturbed IMU acceleration trends and spoofed image descriptions; The device for enhancing the anti-spoofing capability of drones based on LLM multimodal data also includes: The speed change constraint module is used to obtain the actual flight trajectory of the target UAV after inputting the standardized Prompt template into the large language fine-tuning model to output the multimodal deception data in the target format, and to calculate the average offset index between the fake GPS trajectory point series in the multimodal deception data and the actual flight trajectory, and based on the average offset index, to determine the target fake GPS trajectory point series that meets the preset speed change constraint detection requirements. The filtering module is used to acquire each inertial data frame corresponding to the target UAV, and to perform filtering and alignment operations on each inertial data frame to obtain the corresponding filtered inertial data frame. The disturbance consistency verification module is used to calculate the Euclidean difference magnitude corresponding to each inertial data frame based on the filtered inertial data frame, and to calculate the average disturbance amount of all inertial data frames using the Euclidean difference magnitude. Based on the average disturbance amount and a preset maximum tolerance threshold, the module performs a disturbance consistency verification operation on the disturbance IMU acceleration trend to generate a target disturbance IMU acceleration trend that meets the preset disturbance consistency verification requirements. The calculation module is used to perform a significant occlusion extraction operation on the camouflage image description to generate a corresponding camouflage mask region, and to calculate the area of ​​the camouflage mask region and the camouflage area corresponding to the camouflage image description and the total image area, respectively. The camouflage rationality verification module is used to calculate the corresponding occlusion ratio based on the camouflage area and the total image area, and perform a preset image camouflage rationality verification operation on the camouflage image description according to the occlusion ratio, so as to obtain a target camouflage image description that meets the image camouflage rationality verification requirements. The construction module is used to construct a corresponding deception dataset based on the target camouflage image description, the target perturbation IMU acceleration trend, and the target fake GPS trajectory point series; An optimization module is used to optimize the standardized Prompt template according to the deception dataset and a preset dynamic optimization mechanism to generate multimodal deception samples, and to perform enhanced anti-deception training and / or performance evaluation on the target UAV through the multimodal deception samples to obtain corresponding training data and / or evaluation results, so as to optimize the performance of the target UAV in performing the preset flight mission based on the training data and / or the evaluation results.

5. The device for enhancing the anti-spoofing capability of unmanned aerial vehicles based on LLM multimodal data generation according to claim 4, characterized in that, The optimization module includes: The detection unit is used to detect in real time whether the target drone has successfully carried out a deception attack using the deception dataset. When the target drone is detected to have successfully carried out a deception attack, the multimodal sensor data and the state information of the target drone at the time of the successful deception attack are recorded, so as to generate attack feedback data based on the multimodal sensor data and the state information. The update unit is used to dynamically adjust the standardized Prompt template based on the attack feedback data, and generate multimodal deception samples according to the dynamically adjusted standardized Prompt template, so as to update the deception dataset through the multimodal deception samples. The performance evaluation unit is used to perform enhanced anti-spoofing training and / or performance evaluation operations on the target UAV based on a preset anti-spoofing training strategy, a performance evaluation strategy, and an updated deception dataset.

6. An electronic device, characterized in that, include: The device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement a method for improving the anti-spoofing capability of a drone based on LLM multimodal data generation as described in any one of claims 1-3.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement a method for improving the anti-spoofing capability of unmanned aerial vehicles based on LLM multimodal data generation as described in any one of claims 1-3.

8. A computer program product, comprising a computer program, characterized in that, The computer program is executed by a processor to implement a method for improving the anti-spoofing capability of unmanned aerial vehicles based on LLM multimodal data generation as described in any one of claims 1-3.

Images