Counter-reconnaissance unmanned aerial vehicle control system based on covert communication technology and deep reinforcement learning

Abstract

The invention discloses an anti-reconnaissance unmanned aerial vehicle control system based on a covert communication technology and deep reinforcement learning. The anti-reconnaissance unmanned aerial vehicle control system comprises a covert communication module and a deep reinforcement learning module in a flight control system, the covert communication module is used for a communication process of returning reconnaissance information to the ground control station by using the unmanned aerial vehicle as a sender; and the deep reinforcement learning module is used for optimizing the flight route of the unmanned aerial vehicle to enable the unmanned aerial vehicle to arrive at the destination, and starting the covert communication module to covert and return the shot image. According to the method, the communication performance and the optimization of the three-dimensional path are greatly improved on the basis of the prior art, the problems of interception by a monitor, monitoring eavesdropping and the like in the communication process of the unmanned aerial vehicle and the ground control station can be solved, the method can adapt to the three-dimensional path planning problem in a complex environment, and the method is suitable for the reconnaissance task of an anti-reconnaissance unmanned aerial vehicle.

Description

Technical Field

[0001] The present invention belongs to the field of wireless communication technology and unmanned aerial vehicle path planning, and specifically relates to an anti-reconnaissance unmanned aerial vehicle control system based on covert communication technology and deep reinforcement learning. Background Art

[0002] With the rapid development of drone technology, the security defense and privacy protection capabilities of drone communication networks have become increasingly important. They are increasingly widely used in military and civilian fields. In the military field, drones are often used for reconnaissance, surveillance, target tracking and other tasks. However, drones are vulnerable to detection and interference from enemy reconnaissance systems when performing tasks. Therefore, the concealment of communication between drones and ground control stations and how to plan an optimal path that can effectively avoid enemy reconnaissance in complex environments are particularly important.

[0003] In recent years, covert communication technology has developed rapidly. Covert communication technology uses a variety of means to achieve low probability of detection or hide secret information, thereby hiding communication behavior or process. However, at present, covert communication networks still face challenges in line-of-sight link characteristics and high-performance detection, and need to consider the overlap between adaptability to complex drone environments and covert communication scenario requirements.

[0004] Existing UAV path planning methods mainly focus on the static features of the environment, such as terrain, obstacles, etc., but do not take into account the dynamically changing enemy reconnaissance activities. In addition, traditional path planning methods also have shortcomings in terms of concealment and real-time performance.

[0005] Since wireless communication is transmitted to the receiving device in the form of electromagnetic waves, there is no clear physical boundary, which makes the transmitted secret information vulnerable to external attacks and eavesdropping. Especially in medium and long distance transmission, the longer the signal is to be transmitted, the greater the transmission power must be, and the easier it is to be detected by the monitor. The application of drone error correction coding covert communication technology provides a potential solution to improve the security and reliability of covert communication. In wireless communication, steganography is a technology that conceals the existence of communication behavior. During the transmission of the carrier, the interceptor cannot perceive the existence of secret information, thereby achieving the purpose of concealing the transmission of secret information.

[0006] Among the existing technical solutions, there are methods such as artificial potential field method, genetic algorithm, ant colony algorithm, etc., which can plan paths for two-dimensional and three-dimensional environments based on different principles, and fully consider the safety of the planned paths, but still have defects that are difficult to avoid. Intelligent algorithms represented by genetic algorithms and ant colony algorithms have environmental adaptability and system robustness. However, these algorithms require a lot of searching and iterations, resulting in low planning efficiency. The artificial potential field method has good real-time performance due to its simple calculation. If gravity and repulsion are the same at certain points, it is easy to fall into a local optimal solution. If the target is close to an obstacle, the larger repulsion will make it difficult for the UAV to approach the target point.

[0007] As the research on reinforcement learning algorithms continues to deepen, the processing capability of deep reinforcement learning algorithms in big data becomes more prominent, and complex nonlinear functions can be formed based on the input data information. Through function graph fitting, deep reinforcement learning algorithms can be obtained. Compared with reinforcement learning algorithms, this algorithm can be applied to high-dimensional spaces to achieve the purpose of high-speed data processing. At this time, the intelligent agent will also have the ability to simulate and achieve the optimal behavior strategy under complex environmental factors. Summary of the invention

[0008] In view of the concealment of communication information during UAV reconnaissance and the three-dimensional path planning problem in complex environments, the present invention proposes an anti-reconnaissance UAV control system based on covert communication technology and deep reinforcement learning, and proposes a solution that combines error correction coding concealment technology with the construction of a deep reinforcement learning algorithm with a recurrent neural network to plan the three-dimensional path problem in a complex environment. The solution has greatly improved the communication performance and the optimization of the three-dimensional path on the existing basis, can solve the problems of being intercepted, monitored and eavesdropped by the monitor during the communication between the UAV and the ground control station, can adapt to the three-dimensional path planning problem in complex environments, and is suitable for the reconnaissance missions of anti-reconnaissance UAVs.

[0009] To achieve the above object, the present invention provides the following solutions:

[0010] The anti-reconnaissance UAV control system based on covert communication technology and deep reinforcement learning includes the following steps: a covert communication module and a deep reinforcement learning module in the flight control system;

[0011] The covert communication module is used for the communication process of using the UAV as a transmitter to transmit reconnaissance information back to the ground control station;

[0012] The deep reinforcement learning module is used to optimize the flight route of the drone so that the drone reaches the destination, and enable the covert communication module to covertly transmit the captured images back.

[0013] Preferably, the covert communication module comprises: a secret information embedding unit, a data transmission unit and a receiving-end decoding unit;

[0014] The secret information embedding unit is used to encrypt the secret information to be hidden, and then embed the encrypted secret information into the error correction coded data stream;

[0015] The data transmission unit is used to encode the sequence containing secret information, the encoded data is modulated by the signal source to a carrier frequency with orthogonal characteristics, and finally the modulated signal is transmitted in the form of electromagnetic waves;

[0016] The receiving end decoding unit is used to receive the signal from the ground control station, demodulate the signal, extract the information and obtain the secret information through error correction decoding.

[0017] Preferably, encrypting the secret information to be hidden and then embedding the encrypted secret information into the error correction coded data stream comprises:

[0018] According to the two parties in covert communication, a key K is agreed upon through a secret channel to generate a random control selection sequence and a synchronization sequence for communication;

[0019] The sender uses random control to select a sequence, first embeds the synchronization sequence in the error correction code, and then performs error correction coding on the secret information and embeds it into the error correction code to obtain the implicit information.

[0020] Preferably, receiving the signal using the ground control station, demodulating the signal, extracting the information, and obtaining the secret information through error correction decoding includes:

[0021] The ground control station receives the signal, demodulates it and uses the key K to generate a random control selection sequence;

[0022] A random control selection sequence is used to detect whether the error correction code contains a synchronization sequence. If the synchronization sequence is detected, the information is extracted and the secret information is obtained through error correction decoding.

[0023] Preferably, the deep reinforcement learning module includes: a data collection unit, a data preprocessing unit, a data enhancement unit and a data storage unit;

[0024] The data collection unit is used to obtain the data of the laser radar ranging sensor and the posture sensor, wherein the laser radar ranging sensor data is: the laser radar ranging sensor constructs a local three-dimensional map environment centered on itself in real time to obtain information about obstacles within a preset distance; the posture sensor data is: the posture sensor obtains and transmits the position and posture of the drone in real time;

[0025] The data preprocessing unit is used to perform feature extraction and normalization extraction on the acquired laser radar ranging sensor data and posture sensor data, wherein the feature extraction includes: extracting 3D features from the point cloud, using the concept of local features to encode the shape of small blocks around the preset key points;

[0026] The data enhancement unit is used to generate additional data through the simulation platform, and rotate, scale, and shear the pre-processed data to simulate different flight conditions; at the same time, random noise is added to the data;

[0027] The data storage unit is used to store the enhanced data in a database.

[0028] Preferably, generating additional data through the simulation platform includes: building and training a deep reinforcement learning model on the ElegantRL open source platform, and using the ANOTC ground station V5 host computer to monitor the real-time flight data of the drone in real time.

[0029] Preferably, a deep reinforcement learning model is built and trained on the ElegantRL open source platform, and the ANOTC ground station V5 host computer is used to monitor the real-time flight data of the drone in real time, including:

[0030] Model the environment and obtain effective local information from the environment. Modeling the environment includes: using the real-time data of the lidar sensor, using the traditional grid method to divide the 3D space, and then creating a plane containing the starting point of the drone and the target end point. By comparing the height of the plane with the obstacle, the obstacles below the plane are ignored. The effective local information includes: the height of the obstacle in the horizontal position, the height of the drone relative to the ground, the drone's own position information, and the relative position of the current point to the target point.

[0031] Construct a historical trajectory information and model the historical trajectory information as an observable Markov decision process; the observable Markov decision process is represented as (S, A, P, R, Ω, O, γ), where S represents the real state S in the environment t A represents the set of all available actions of the drone, P represents the probability distribution of moving to other states after executing the action, P(S t+1 |S t ,a t )→(0,1), R is the reward for executing the action in state and transferring to state, Ω is the set of observations, O is the local environment information obtained by the sensor, O(O t |S t ,a t ) indicates S t Execute action a in state t The conditional observation probability distribution of , γ discount factor;

[0032] A strategy is obtained by interactive learning of historical trajectory information and the environment. The state value function is defined as the expected sum of discounted rewards under the strategy.

[0033] According to the current strategy, take action a in state S to obtain the expected discounted reward, and then continuously iterate and update through the Bellman formula to obtain the state at the next moment and take action.

[0034] The present invention also provides an anti-reconnaissance UAV control method based on covert communication technology and deep reinforcement learning, comprising the following steps:

[0035] The communication process of using drones as transmitters to transmit reconnaissance information back to ground control stations;

[0036] The flight route of the UAV is optimized to enable the UAV to reach the destination, and the covert communication module is enabled to covertly transmit the captured images back.

[0037] The present invention also provides an anti-reconnaissance UAV based on covert communication technology and deep reinforcement learning, and the UAV applies the anti-reconnaissance UAV control method based on covert communication technology and deep reinforcement learning.

[0038] The present invention also provides a storage medium, on which a computer program is stored, and when the computer program is running, the anti-reconnaissance drone control method based on covert communication technology and deep reinforcement learning is executed.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] (1) Through the UAV error correction coding covert communication technology, the UAV can improve the reliability and security of secret information transmission during the communication process with the ground control station. It enhances the communication security between the UAV and the ground control station and ensures the stability of the command and control link.

[0041] (2) Using deep reinforcement learning algorithms, drones can interact with the environment in real time and make more reasonable path planning decisions when faced with complex dynamic environments, thereby reducing the risk of being detected by enemy reconnaissance systems.

[0042] (3) Based on the RNN module, the temporal feature information of the sample sequence is extracted, which fundamentally solves the memory problem of the sequence samples. Through a method with the ability to remember historical trajectories, the drone can effectively avoid falling into the "selection trap" and thus more accurately estimate the real environment.

[0043] (4) During the information encryption process, images, text, audio, etc. are used as carrier data, and the carrier contains random components, which further ensures the security of secret information. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0045] Figure 1 A schematic diagram of a wireless covert communication model according to an embodiment of the present invention;

[0046] Figure 2 A schematic diagram of a covert communication model based on error correction coding according to an embodiment of the present invention;

[0047] Figure 3 A schematic diagram of a wireless covert communication and trajectory planning model according to an embodiment of the present invention;

[0048] Figure 4 A schematic diagram of a network architecture based on CNN and RNN according to an embodiment of the present invention;

[0049] Figure 5 A schematic diagram of the working process of a drone according to an embodiment of the present invention;

[0050] Figure 6 Schematic diagram of a drone according to an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0053] Embodiment 1

[0054] like Figure 1-Figure 4 As shown, the present invention provides an anti-reconnaissance UAV control system based on covert communication technology and deep reinforcement learning, comprising the following steps: a covert communication module and a deep reinforcement learning module in a flight control system;

[0055] The covert communication module is used for the communication process of using the UAV as the transmitter to transmit reconnaissance information back to the ground control station;

[0056] The deep reinforcement learning module is used to optimize the flight route of the drone so that the drone reaches the destination, and enable the covert communication module to covertly transmit the captured images back.

[0057] In this embodiment, the covert communication module includes: a secret information embedding unit, a data transmission unit and a receiving-end decoding unit;

[0058] The secret information embedding unit is used to encrypt the secret information to be hidden, and then embed the encrypted secret information into the error correction coded data stream;

[0059] The data transmission unit is used to encode the sequence containing secret information (the process of converting digital signals into analog signals in wireless communication). The encoded data is modulated by the signal source onto a carrier frequency with orthogonal characteristics. This process is also called "signal modulation". Modulation refers to the process of combining information signals (such as audio, video or data) with carrier signals so that the information can be transmitted via radio waves, cables or other transmission media. The main purpose of modulation is to convert low-frequency signals (such as baseband signals) into high-frequency signals suitable for transmission on a specific communication channel. The carrier signal is a high-frequency oscillating signal, usually a sine wave, used to carry information signals. The frequency, amplitude and phase of the carrier can be modulated to carry information. The purpose of modulation is to allow data to be transmitted farther and more stably in the channel. Finally, the modulated signal is emitted in the form of electromagnetic waves.

[0060] The receiving end decoding unit is used to receive the signal from the ground control station, demodulate the signal, extract the information, and obtain the secret information through error correction decoding.

[0061] The method of encrypting the secret information to be hidden and then embedding the encrypted secret information into the error correction coded data stream includes:

[0062] The two parties in covert communication agree on a key K through a secret channel to generate a random control selection sequence and a synchronization sequence for communication; the secret channel is an information hiding technology that allows the sender to hide information in the return image, making it difficult to detect or detect. Unlike encryption, the secret channel does not protect information from being understood (i.e. confidentiality), but protects information from being detected (i.e. concealment).

[0063] The sender uses random control to select a sequence, first embeds a synchronization sequence in the error correction code, and then performs error correction coding on the secret information and embeds it into the error correction code to obtain implicit information. The implicit information is the fused information sequence after the encoded digital information and the embedded secret information sequence.

[0064] Among them, the ground control station receives the signal, demodulates the signal, extracts the information, and obtains the secret information through error correction decoding, including:

[0065] The ground control station receives the signal, demodulates it and uses the key K to generate a random control selection sequence;

[0066] The random control selection sequence is used to detect whether the error correction code contains a synchronization sequence. If the synchronization sequence is detected, the information is extracted and the secret information is obtained through error correction decoding. The random control selection sequence is used to detect whether the error correction code contains a synchronization sequence, including: using a random number generator to generate a series of random binary data, which will be used as test input.

[0067] The following is a detailed process of secret information steganography: Assume that the error correction code used in the communication system is a binary linear block code (n, k, t), where n is the code length, k is the information bit length, and t is the error correction capability, all in bits. The binary code sequence after error correction coding is recorded as m = (m 1 ,m 2 ,…,m k ,…). The length of the secret information is N bits.

[0068] Issuer:

[0069] Step 1: Encrypt the information to be transmitted and then perform forward error correction coding. The error correction coding selected is consistent with the error correction coding of the communication system. This ensures the transmission reliability of the secret channel. It is denoted as c = (c 1 ,c 2 ,…,c N ), and the key K generates a random control selection sequence a=(a 1 ,a 2 ,…,a k ,…), the binary synchronization sequence is recorded as s=(s 1 ,s 2 ,…,s N ).

[0070] Step 2: Embed synchronization sequence s, when a i ∈A, s j Embedded into the i-th position of sequence m, that is, m i =s j ,j=1,2,…,t. A is secretly agreed upon by both parties and embedded in sequence c. i ∈A, c j Embedded into the i-th position of sequence m, that is, m i =c j ,j=1,2,…,N。

[0071] Recipient:

[0072] Step 1: The key K generates a random control selection sequence a=(a 1 ,a2 ,…,a i ,…), the binary synchronization sequence is recorded as s=(s 1 ,s 2 ,…,s t ), then detect the synchronization sequence s, when a i ∈m i , compare sequence b and synchronization sequence s, if b and s are basically consistent, then extract sequence c, when a i ∈A, let c i =m i , and get the sequence c.

[0073] Step 2: Perform error correction decoding and cryptographic decoding on sequence c.

[0074] The steganographic system model based on error correction coding is that error correction coding can be used as an effective carrier of steganography because of its inherent redundancy and randomness. Since error correction coding itself contains a large amount of redundant information and has a certain degree of randomness, it is difficult to detect the existence of secret information through traditional steganalysis technology. Therefore, during the signal transmission process, the monitor cannot perceive the existence of secret information, thereby concealing the existence of secret information.

[0075] In this embodiment, the deep reinforcement learning module in the flight control system mainly includes data collection, data preprocessing, data enhancement and data storage processes. The processed data is used as input and input into the modeled deep reinforcement learning model for training. It is also verified in different environments to evaluate its performance. This process is called model training and verification.

[0076] The deep reinforcement learning module includes: data collection unit, data preprocessing unit, data enhancement unit and data storage unit;

[0077] The data collection unit is used to obtain the data of the laser radar ranging sensor and the posture sensor, wherein the laser radar ranging sensor data is: the laser radar ranging sensor is used to construct a local three-dimensional map environment centered on itself in real time, and obtain the information of obstacles within a preset distance; the posture sensor data is: the posture sensor is used to obtain and transmit the position and posture of the drone in real time; wherein, the laser radar ranging sensor is used to construct a local three-dimensional map environment centered on itself in real time, including: the 3DSLAM technology based on the laser point cloud uses a three-dimensional laser sensor to obtain three-dimensional space point data, estimates the posture through scanning matching between adjacent point cloud frames, and establishes a complete point cloud map. The information of nearby obstacles includes: the laser radar's recognition of obstacles involves the processing of single-frame multi-line laser radar acquisition data, which is a three-dimensional point cloud map. The purpose of obstacle recognition is to determine the size, distance and other information of obstacles within a certain range around the drone to avoid collisions.

[0078] The data preprocessing unit is used to extract features and normalize the acquired LiDAR ranging sensor data and posture sensor data, wherein the feature extraction and normalization extraction of the acquired LiDAR ranging sensor data and posture sensor data include: the raw data obtained from the LiDAR device is usually in LAS or ASCII format, and the data can be read using the laspy library in Python, and the data is converted and corrected to ensure that the data accurately corresponds to the ground coordinate system. Then the data is denoised and filtered: outliers are removed: outliers are removed using the statistical method of the 3σ rule. Filtering: the data is smoothed using a filter, and the method used in this solution is Gaussian filtering to reduce high-frequency noise in the data and remove impulse noise. Feature extraction includes: extracting 3D features from the point cloud, using the concept of local features to encode the shape of small blocks around specific key points, which helps avoid storing many points while facilitating the matching process.

[0079] The data enhancement unit is used to generate additional data through the simulation platform and rotate, scale, and shear the pre-processed data to simulate different flight conditions; at the same time, random noise is added to the data;

[0080] The data storage unit is used to store the enhanced data in a database.

[0081] Specifically, real-time construction of the local environment requires a large amount of environmental data, and the data collection sources include lidar ranging sensors and posture sensors. The lidar ranging sensor is used to build a local three-dimensional map environment centered on itself in real time to obtain information about nearby obstacles. The posture sensor obtains and transmits the position and posture of the drone in real time. The advantage of the lidar ranging sensor and posture sensor module is that they maintain relative independence, that is, no manual settings are required, and environmental data is automatically collected and processed. The output can be selected on demand. The specific parameters selected in this solution include raw data such as speed, position, and deflection angle.

[0082] The raw data contains interference signals such as outliers and noise, so the raw information needs to be preprocessed to further ensure the quality of the acquired data. It is necessary to extract features useful for flight control from the raw data output by the sensor and scale the data to a certain range to speed up learning and improve model stability. This process is called data feature extraction and data normalization.

[0083] Generate additional data through the simulation platform, and rotate, scale, shear and other transformations on the existing data to simulate different flight conditions. At the same time, add random noise to the data to improve the robustness of the model. Store the processed data in the database for subsequent training and analysis. In order to ensure the storage security and portability of the data, it should be stored in a suitable format. This module uses CSV storage for subsequent reading and processing.

[0084] Use the data loading tool DataLoader provided by the deep learning framework PyTorch to load the prepared dataset into the model for training.

[0085] The above is the process of data collection, preprocessing, enhancement and storage. The stored data is used as input and input into the model for training. Next is the path planning algorithm module based on deep reinforcement learning, which is expressed as a partially observable Markov decision process.

[0086] In this embodiment, generating additional data through the simulation platform includes: building and training a deep reinforcement learning model on the ElegantRL open source platform, and using the ANOTC ground station V5 host computer to monitor the real-time flight data of the drone in real time.

[0087] In this embodiment, a deep reinforcement learning model is built and trained on the ElegantRL open source platform, and the ANOTC ground station V5 host computer is used to monitor the real-time flight data of the drone in real time, including:

[0088] Model the environment and obtain effective local information from the environment. Modeling the environment includes: using the real-time data of the lidar sensor, using the traditional grid method to divide the 3D space, and then creating a plane containing the starting point of the drone and the target end point. By comparing the height of the plane with the obstacle, the obstacles below the plane are ignored. The effective local information includes: the height of the obstacle in the horizontal position, the height of the drone relative to the ground, the drone's own position information, and the relative position of the current point to the target point.

[0089] Construct a historical trajectory information and model the historical trajectory information as an observable Markov decision process; the observable Markov decision process is represented as (S, A, P, R, Ω, O, γ), where S represents the real state S in the environment t A represents the set of all available actions of the drone, P represents the probability distribution of moving to other states after executing the action, P(S t+1 |S t ,a t )→(0,1), R is the reward for executing the action in state and transferring to state, Ω is the set of observations, O is the local environment information obtained by the sensor, O(O t |St ,a t ) indicates S t Execute action a in state t The conditional observation probability distribution of , γ discount factor;

[0090] A strategy is obtained by interactive learning of historical trajectory information and the environment. The state value function is defined as the expected sum of discounted rewards under the strategy.

[0091] According to the current strategy, take action a in state S to obtain the expected discounted reward, and then continuously iterate and update through the Bellman formula to obtain the state at the next moment and take action.

[0092] Specifically, the environment is modeled first. The effective local information that can be obtained from the environment includes key data such as the height of the obstacle in the horizontal position, the height of the drone relative to the ground, the drone's own position information, and the relative position from the current point to the target point. Considering the maneuvering restrictions of the aircraft, such as the minimum turning radius and the maximum climbing angle, it is difficult for the drone to detect the dangerous area between two obstacles. If the drone takes direct flight or action, the probability of obstacle avoidance failure may be very high. If the drone chooses to turn left, flight safety can be guaranteed. Therefore, the stronger the horizontal maneuverability, the greater the probability of the drone escaping safely. Actions include horizontal actions and vertical actions. Horizontal actions include turning left 45° / 63°, flying straight, and turning right 45° / 63°. Vertical actions include climbing 45°, flying straight, and descending 45°. There are 11 actions in the drone's action space.

[0093] When the drone encounters an "inverted V-shaped" obstacle, in order to prevent it from falling into a selection loop, consider constructing a historical trajectory information and modeling it as an observable Markov decision process. The observable Markov decision process can be expressed as (S, A, P, R, Ω, O, γ), where S represents the real state S in the environment t A represents the set of all available actions of the drone, P represents the probability distribution of moving to other states after executing the action, P(S t+1 |S t ,a t )→(0,1). R is the reward for executing an action in state and transferring to state, Ω is the set of observations, O is the local environment information obtained by the sensor, O(O t |S t ,a t ) indicates S t Execute action a in state t The conditional observation probability distribution of , γ discount factor.

[0094] The reinforcement learning algorithm obtains a strategy by learning from the environment through interaction. The state value function is defined as the expected sum of discounted rewards under this strategy. According to the current strategy, take action a in state S to obtain the expected discounted reward, and then continuously iterate and update through the Bellman formula to obtain the state at the next moment and take action.

[0095] The network structure includes three neural networks. The input of the model is a historical trajectory sequence. The CNN module is used to extract spatial feature information. The function of the CNN module is to extract low-dimensional and effective spatial features. The RNN module is constructed to extract temporal feature information. Its network properties fundamentally solve the memory problem of sequence samples. The hidden layer not only receives the features of the CNN module, but also receives the hidden layer state of the previous moment. Finally, the estimated state-action value (the action and state to be made at the next moment obtained through strategy learning) is predicted through the fully connected neural network. The above is the detailed process of network construction, initialization and operation of the algorithm module.

[0096] Due to the correlation of sample sequences, the neural network has difficulty in convergence. A fixed Q-target network and experience replay mechanism are used to speed up the convergence of the algorithm.

[0097] The samples generated by the interaction between the drone and the environment are stored in the memory pool, and the network randomly takes a certain number of sample sequences from the memory pool for training.

[0098] The above is the entire operation process based on the deep reinforcement learning algorithm module.

[0099] Embodiment 2

[0100] The present invention also provides an anti-reconnaissance UAV control method based on covert communication technology and deep reinforcement learning, comprising the following steps:

[0101] The communication process of using drones as transmitters to transmit reconnaissance information back to ground control stations;

[0102] The flight route of the UAV is optimized to enable the UAV to reach the destination, and the covert communication module is enabled to covertly transmit the captured images back.

[0103] Embodiment 3

[0104] like Figure 5 , Figure 6 As shown, the present invention also provides an anti-reconnaissance UAV based on covert communication technology and deep reinforcement learning, and the UAV applies the anti-reconnaissance UAV control method based on covert communication technology and deep reinforcement learning.

[0105] Embodiment 4

[0106] The present invention also provides a storage medium, on which a computer program is stored, and when the computer program is running, the anti-reconnaissance drone control method based on covert communication technology and deep reinforcement learning is executed.

[0107] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute the dialogue method of any of the above embodiments.

[0108] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0109] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the multimodal multi-person open dialogue method of any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0110] The communication module of the present invention adopts a covert communication technology based on error correction coding. The essence of error correction coding is to add a certain number of supervisory code elements in the information code, which satisfy a certain constraint relationship. When an error occurs during transmission, this constraint relationship is used to detect and correct the error. Information is interfered with by various aspects during transmission. Error correction coding is a technology to overcome these interferences and increase transmission reliability, and its purpose is to improve the error detection and correction capabilities of the encoded signal as much as possible. This covert communication technology based on error correction coding can ensure the security of the system under the condition of complex channel noise interference; at the same time, in the case of random errors in the channel, the embedding of secret information is low, which can further reduce the probability of erroneous transmission of secret information.

[0111] In the flight control system, the deep reinforcement learning algorithm module learns the path planning strategy suitable for the current environment by interacting with the environment without manually setting complex rules. A new action selection strategy is proposed by combining the current reward R value and Q value, which solves the problem of inaccurate prediction of neural networks in the early stage of training. An adaptive sampling mechanism is proposed to build two memory pools according to the average reward value of each round. Samples are drawn from important memory pools according to the frequency of task failure. This method improves the learning efficiency and convergence stability of the algorithm at a lower computational cost.

[0112] Traditional Q-learning uses tables to record learned state-action values, which limits the algorithm's ability to solve large-scale planning state space problems. Therefore, we use neural networks to approximate state-action values. Neural networks can theoretically fit any nonlinear model with enough training data and appropriate learning parameters.

[0113] In summary, anti-reconnaissance drones based on covert communication technology and deep reinforcement learning algorithms provide a more covert, intelligent and efficient means of communication and reconnaissance in the modern battlefield.

[0114] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. An anti-reconnaissance UAV control system based on covert communication technology and deep reinforcement learning, characterized in that: include: Deep reinforcement learning modules in covert communication modules and flight control systems; The covert communication module is used for the communication process of using the UAV as a transmitter to transmit reconnaissance information back to the ground control station; The deep reinforcement learning module is used to optimize the flight route of the drone so that the drone reaches the destination, and enable the covert communication module to covertly transmit the captured images back.

2. According to claim 1, the anti-reconnaissance UAV control system based on covert communication technology and deep reinforcement learning is characterized in that: The covert communication module includes: a secret information embedding unit, a data transmission unit and a receiving end decoding unit; The secret information embedding unit is used to encrypt the secret information to be hidden, and then embed the encrypted secret information into the error correction coded data stream; The data transmission unit is used to encode the sequence containing secret information, the encoded data is modulated by the signal source to a carrier frequency with orthogonal characteristics, and finally the modulated signal is transmitted in the form of electromagnetic waves; The receiving end decoding unit is used to receive the signal from the ground control station, demodulate the signal, extract the information and obtain the secret information through error correction decoding.

3. The anti-reconnaissance UAV control system based on covert communication technology and deep reinforcement learning according to claim 2 is characterized in that: Encrypting the secret information to be hidden and then embedding the encrypted secret information into the error correction coded data stream includes: According to the two parties in covert communication, a key K is agreed upon through a secret channel to generate a random control selection sequence and a synchronization sequence for communication; The sender uses random control to select a sequence, first embeds the synchronization sequence in the error correction code, and then performs error correction coding on the secret information and embeds it into the error correction code to obtain the implicit information.

4. The anti-reconnaissance UAV control system based on covert communication technology and deep reinforcement learning according to claim 2 is characterized in that: The ground control station receives the signal, demodulates the signal, extracts the information, and decodes it through error correction to obtain the secret information including: The ground control station receives the signal, demodulates it and uses the key K to generate a random control selection sequence; A random control selection sequence is used to detect whether the error correction code contains a synchronization sequence. If the synchronization sequence is detected, the information is extracted and the secret information is obtained through error correction decoding.

5. The anti-reconnaissance UAV control system based on covert communication technology and deep reinforcement learning according to claim 1 is characterized in that: The deep reinforcement learning module includes: a data collection unit, a data preprocessing unit, a data enhancement unit and a data storage unit; The data collection unit is used to obtain the data of the laser radar ranging sensor and the posture sensor, wherein the laser radar ranging sensor data is: the laser radar ranging sensor constructs a local three-dimensional map environment centered on itself in real time to obtain information about obstacles within a preset distance; the posture sensor data is: the posture sensor obtains and transmits the position and posture of the drone in real time; The data preprocessing unit is used to perform feature extraction and normalization extraction on the acquired laser radar ranging sensor data and posture sensor data, wherein the feature extraction includes: extracting 3D features from the point cloud, using the concept of local features to encode the shape of small blocks around the preset key points; The data enhancement unit is used to generate additional data through the simulation platform, and rotate, scale, and shear the pre-processed data to simulate different flight conditions; at the same time, random noise is added to the data; The data storage unit is used to store the enhanced data in a database.

6. The anti-reconnaissance UAV control system based on covert communication technology and deep reinforcement learning according to claim 5 is characterized in that: Generating additional data through the simulation platform includes: building and training a deep reinforcement learning model on the ElegantRL open source platform, and using the ANOTC ground station V5 host computer to monitor the real-time flight data of the drone.

7. The anti-reconnaissance UAV control system based on covert communication technology and deep reinforcement learning according to claim 6 is characterized in that: The deep reinforcement learning model was built and trained on the ElegantRL open source platform, and the real-time flight data of the drone was monitored in real time using the ANOTC ground station V5 host computer, including: Model the environment and obtain effective local information from the environment. Modeling the environment includes: using the real-time data of the lidar sensor, using the traditional grid method to divide the 3D space, and then creating a plane containing the starting point of the drone and the target end point. By comparing the height of the plane with the obstacle, the obstacles below the plane are ignored. The effective local information includes: the height of the obstacle in the horizontal position, the height of the drone relative to the ground, the drone's own position information, and the relative position of the current point to the target point. Construct a historical trajectory information and model the historical trajectory information as an observable Markov decision process; the observable Markov decision process is represented as (S, A, P, R, Ω, O, γ), where S represents the real state S in the environment t A represents the set of all available actions of the drone, P represents the probability distribution of moving to other states after executing the action, P(S t+1 |S t ,a t )→(0,1), R is the reward for executing the action in state and transferring to state, Ω is the set of observations, O is the local environment information obtained by the sensor, O(O t |S t ,a t ) indicates S t Execute action a in state t The conditional observation probability distribution of , γ discount factor; A strategy is obtained by interactive learning of historical trajectory information and the environment. The state value function is defined as the expected sum of discounted rewards under the strategy. According to the current strategy, take action a in state S to obtain the expected discounted reward, and then continuously iterate and update through the Bellman formula to obtain the state at the next moment and take action. 8.An anti-reconnaissance UAV control method based on covert communication technology and deep reinforcement learning, characterized in that: The following steps are involved: The communication process of using drones as transmitters to transmit reconnaissance information back to ground control stations; The flight route of the UAV is optimized to enable the UAV to reach the destination, and the covert communication module is enabled to covertly transmit the captured images back. 9.An anti-reconnaissance drone based on covert communication technology and deep reinforcement learning, characterized in that: The drone applies the anti-reconnaissance drone control method based on covert communication technology and deep reinforcement learning as described in claim 8.

10. A storage medium, characterized in that: The storage medium stores a computer program, which, when running, executes the anti-reconnaissance drone control method based on covert communication technology and deep reinforcement learning as described in claim 8.

Images