UAV swarm communication quality optimization system and method based on deep reinforcement learning

Abstract

A system and method for optimizing the communication quality of drone swarms based on deep reinforcement learning, belonging to the field of electrical data processing technology. The system includes a central control device and a drone swarm, and any drone in the drone swarm includes a sending control strategy network, a sending device, a receiving control strategy network, and a receiving device; the central control device sends errors to the two drones according to the parameters of the sending devices of the two drones in the drone swarm that are communicating, the parameters of the receiving devices, the sending action, the sending state, the receiving action, the receiving state, and the reward; the two drones modify the parameters of the sending control strategy network, the sending device, the receiving control strategy network, and the receiving device according to the received errors. The present invention greatly improves the communication quality by error-correcting the parameters of the sending device and the receiving device.

Description

Technical Field

[0001] The present invention relates to a system and method for optimizing the communication quality of an unmanned aerial vehicle swarm based on deep reinforcement learning, and belongs to the technical field of digital communications. Background Art

[0002] For example, a Chinese invention application with publication number CN114598440A discloses an OFDM transmission method and system based on two-dimensional chaotic mapping constellation encryption. In the Chinese invention application, when sending information, the bit information to be transmitted is subjected to serial-to-parallel conversion, IFFT conversion and parallel-to-serial conversion; when receiving information, the received information is subjected to serial-to-parallel conversion, FFT conversion and parallel-to-serial conversion. However, the system parameters of the prior art are artificially selected, and its communication quality is low. Summary of the invention

[0003] In order to overcome the shortcomings of the prior art, the purpose of the present invention is to provide a drone swarm communication quality optimization system and method based on deep reinforcement learning, which greatly improves the communication quality through deep reinforcement learning reasonable parameters.

[0004] In order to overcome the shortcomings of the prior art, the present invention provides a UAV swarm communication quality optimization system based on deep reinforcement learning, which is characterized by comprising a central control device and a UAV swarm, wherein any UAV in the UAV swarm includes a sending control strategy network. Transmitter, receiving control strategy network and the receiving device, where θ1 is the parameter of the sending control strategy network, For sending status, is the sending action, θ2 is the parameter of the receiving control strategy network, In receiving state, The central control device receives the action according to the drone U in the drone swarm that is communicating. k and UAV j The parameters of the sending device, the parameters of the receiving device, the sending action, the sending status, the receiving action, the receiving status and the reward are given to the two drones; the two drones modify the parameters of the sending control strategy network, the sending device, the receiving control strategy network and the receiving device according to the received error.

[0005] To achieve the above-mentioned object of the invention, the present invention also provides a method for optimizing the communication quality of drone swarm based on deep reinforcement learning, which comprises the following steps:

[0006] Step 1: The central control device detects the two drones U in the drone swarm that are communicating. k and U jThe parameters of the sending device, the parameters of the receiving device, the sending action, the sending status, the receiving action, the receiving status and the reward are given to the two drones for sending errors;

[0007] Step 2: The two UAVs modify the parameters of the sending control strategy network, the sending device, the receiving control strategy network and the receiving device according to the received error.

[0008] Compared with the prior art, the UAV swarm communication quality optimization system and method based on deep reinforcement learning provided by the present invention has the following beneficial effects:

[0009] 1. It uses reinforcement learning to reasonably select the parameters of the sending device and the receiving device, greatly improving the communication quality; 2. It simplifies the communication system and reduces costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a block diagram of the composition of the UAV swarm communication quality optimization system based on deep reinforcement learning provided by the present invention;

[0011] Figure 2 It is a block diagram of the composition of the OFDM symbol generator of the signal transmission device provided by the present invention;

[0012] Figure 3 The present invention is a block diagram of the bit stream recovery device of the receiving device. DETAILED DESCRIPTION

[0013] It should be noted that the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. The advantages and features of the present invention and methods for achieving the advantages and features will become clear with reference to the accompanying drawings and the embodiments described in detail below.

[0014] However, the present invention is not limited to the embodiments disclosed below, and can be implemented in a variety of different forms. This embodiment is only used to make the disclosure of the present invention more complete and to fully inform the scope of the invention to ordinary technicians in the technical field to which the present invention belongs. The present invention is only defined by the scope of protection claimed in the invention.

[0015] Although "first", "second", etc. are used to describe various elements, components and / or parts, these elements, components and / or parts are not limited by these terms. These terms are only used to distinguish one element, constituent element or part from other elements, constituent elements or parts. Therefore, it is obvious that within the technical spirit of the present disclosure, the first element, first element or first part mentioned below may also be the second element, second element or second part, and the terms used in this specification are only used to describe the embodiments and are not intended to limit the present disclosure.

[0016] In this specification, as long as there is no special mention in the sentence, the singular also includes the plural. The use of "comprising" and / or "consisting of..." in the specification does not exclude the existence or addition of one or more other structural elements, steps, actions and / or elements of the mentioned structural elements, steps, actions and / or elements.

[0017] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those of ordinary skill in the art to which the invention belongs. In addition, terms defined in commonly used dictionaries shall not be interpreted ideally or excessively unless otherwise specifically defined.

[0018] In addition, the same reference numerals are given to the same or similar structures, and duplicate descriptions are omitted.

[0019] First embodiment

[0020] Figure 1 is a block diagram of a drone swarm communication quality optimization system based on deep reinforcement learning provided by the first embodiment of the present invention. Figure 1 As shown, the UAV swarm communication quality optimization system based on deep reinforcement learning provided by the first embodiment of the present invention includes a central control device and a UAV swarm, and any UAV in the UAV swarm includes a sending control strategy network Transmitter, receiving control strategy network and the receiving device, where θ1 is the parameter of the sending control strategy network, For sending status, is the sending action, θ2 is the parameter of the receiving control strategy network, In receiving state, The central control device receives the action according to the drone U in the drone swarm that is communicating. k and UAV j The parameters of the sending device, the parameters of the receiving device, the sending action, the sending status, the receiving action, the receiving status and the reward are given to the two drones; the two drones modify the parameters of the sending control strategy network, the sending device, the receiving control strategy network and the receiving device according to the received error.

[0021] In the first embodiment, the central control device includes a first reinforcement learning module, and the first reinforcement learning module includes a first value network First target value network and the first error calculator, where w 1now is the parameter of the first value network, is the parameter of the first target value network, and They are the UAVs for communication k The sending action and sending state of the sending device at that time t, and They are the UAVs for communication j The receiving action and receiving state of the receiving device at that time t; the process of training the first reinforcement learning module includes:

[0022] S1-01: The first value network makes a prediction to obtain the first prediction value

[0023] S1-02: UAV k The sending device makes a prediction to obtain a predicted sending action In the formula, For UAV k The sending state at the next time t+1, For UAV k Parameters of the sending device;

[0024] S1-03: UAV j The receiving device makes a prediction to obtain a predicted receiving action

[0025] In the formula, For UAV j The receiving state at the next time t+1, For UAV j Parameters of the receiving device;

[0026] S1-04: The first target value network makes a prediction to obtain the first time variance target value

[0027] In the formula For UAV k The reward received by the sending device, For UAV j The reward obtained by the receiving device; γ1 is the adjustment coefficient;

[0028] S1-05: The first error calculator is based on Calculate the first error δ1 and use the first error δ1 to update the first value network parameter w 1now w 1new :

[0029] In the formula, α1 is the adjustment coefficient, is the gradient function of w1;

[0030] S01-06: The first value network parameter uses the updated first value network parameter w 1new Update the first target value network parameters for

[0031]

[0032] S01-07: The central control device sends the first error δ1 to the drones U k and UAV j .

[0033] In the first embodiment, the drone U k and UAV j Receive δ1 sent by the central control device and implement the following process:

[0034] S1-08: Update the drone U through the following method k Send the parameters of the control strategy network for

[0035]

[0036] In the formula, for The gradient function; β1 is the adjustment coefficient;

[0037] S1-09: Update the drone U through the following method j Receive the parameters of the control strategy network for

[0038]

[0039] In the formula, for The gradient function, β2 is the adjustment coefficient;

[0040] S1-10: UAV k Send control strategy network utilization Update UAV k Parameters of the sending device for

[0041]

[0042] S1-11: UAV j The receiving control strategy network utilizes Update UAV jParameters of the receiving device for

[0043]

[0044] In the first embodiment, the central control device further includes a second reinforcement learning module, and the second reinforcement learning module includes a second value network Second target value network and the second error calculator, where w 2now is the parameter of the second value network, is the parameter of the second target value network; and They are the UAVs for communication k The receiving action and receiving state of the receiving device at that time t, and They are the UAVs for communication j The sending action and sending state of the sending device at that time t; the process of training the second reinforcement learning module includes:

[0045] S2-01: The second value network makes a prediction to obtain the second prediction value

[0046] S2-02: UAV j The sending device makes a prediction to obtain a predicted sending action

[0047] In the formula, For UAV j The sending state at the next time t+1, For UAV j Parameters of the sending device;

[0048] S2-03: UAV k The receiving device makes a prediction to obtain a predicted receiving action In the formula, For UAV k The receiving state at the next time t+1, For UAV k Parameters of the receiving device;

[0049] S2-04: The second target value network makes a prediction to obtain the second time variance target value

[0050] In the formula, For UAV k The receiving device receives the reward, For UAV j The reward obtained by the sending device; γ2 is the adjustment coefficient;

[0051] S2-05: The second error calculator is based on Calculate the second error δ2 and use the second error δ2 to update the second value network parameter w 2now w 2new :

[0052] In the formula, α2 is the adjustment coefficient, is the gradient function of w2;

[0053] S2-06: The second value network parameter uses the updated second value network parameter w 2new Update the second target value network parameters for

[0054]

[0055] S02-07: The central control device sends the second error δ2 to the drones U k and UAV j .

[0056] In the first embodiment, the drone U k and UAV j Receive δ2 sent by the central control device and implement the following process:

[0057] S2-08: Update the drone U through the following method k The parameters of the receiving control strategy network for

[0058] In the formula, for Gradient function; β3 is the adjustment coefficient

[0059] S2-09: Update the drone U through the following method j Send the parameters of the control strategy network for

[0060]

[0061] In the formula, for The gradient function; β4 is the adjustment coefficient;

[0062] S2-10: UAV k The receiving control strategy network utilizes Update UAV k Parameters of the receiving device for

[0063]

[0064] S2-11: UAV j Send control strategy network utilization Update UAV j Parameters of the sending device for

[0065]

[0066] In the first embodiment, the drone U k Send a letter, and the drone U j When receiving a message, the parameters that need to be optimized are the UAV k Number of subcarriers in the transmitter Carrier frequency of the transmitter UAV j The mixer frequency in the receiving device

[0067] Specifically,

[0068]

[0069] In the formula, For N k The gradient function of For The gradient function of

[0070]

[0071] In the formula, Yes The gradient function of .

[0072] UAV j Send a letter, and the drone U k When receiving a message, the parameters that need to be optimized are the UAV j Number of subcarriers in the transmitter Carrier frequency of the transmitter UAV K The frequency of the mixer in the receiving device

[0073] Specifically,

[0074]

[0075] In the formula, For N j The gradient function of For The gradient function of

[0076]

[0077] In the formula, Yes The gradient function of .

[0078] The first embodiment of the present invention reasonably selects parameters of the sending device and the receiving device through reinforcement learning, thereby greatly improving the communication quality.

[0079] Figure 2 : is a block diagram of the OFDM symbol generator of the transmitting device provided by the present invention, such as Figure 2 As shown, the signaling device includes an OFDM symbol generator, and the OFDM symbol generator includes a first data grouping module, an IFFT module and a first loop module, wherein the first data grouping module is based on N new The binary code stream to be sent is divided into groups in units, and provided to the IFFT module in units of one group. The IFFT module performs the following operations:

[0080]

[0081] In the formula, the first cycle module uses Make the IFFT module perform Operations are performed to obtain the serial output OFDM symbol sequence In the first embodiment, the signal transmitting device further includes a DAC converter and a modulation device, wherein the DAC converter converts the OFDM symbol sequence {X(p)} into an analog signal, and the modulation device modulates the analog signal converted from the OFDM symbol sequence {X(p)} to a carrier frequency of Generate a high-frequency modulated wave based on the high-frequency signal.

[0082] In the first embodiment, the receiving device further includes a mixer, a demodulator and an ADC converter. The mixer combines the received high-frequency signal with the local oscillator frequency. Mixing is performed to obtain an intermediate frequency signal, the demodulator is used to demodulate the intermediate frequency signal to obtain an analog signal converted from the OFDM symbol sequence {X(p)} sent by the transmitting device, and the ADC converter is used to perform analog-to-digital conversion on the analog signal converted from the demodulated OFDM symbol sequence {X(p)} to obtain the OFDM symbol sequence {X(p)} sent by the transmitting device.

[0083] Figure 3 FIG. 1 is a block diagram of a bit stream restorer of a receiving device provided by a first embodiment of the present invention. Figure 3 As shown, the receiving device also includes a bit stream restorer, which includes a second data grouping module, an FFT module and a second loop module, wherein the second data grouping module is The OFDM symbol sequence {X(p)} is grouped into units, and provided to the FFT module in units of one group. The FFT module performs the following operations:

[0084]

[0085] In the formula, the second cycle module uses Make the IFFT module perform Operations are performed to obtain the serial output bit sequence

[0086] The first embodiment of the present invention simplifies the communication system and reduces the cost through the above technical solution.

[0087] Second embodiment

[0088] The second embodiment of the present invention only scans the contents that are different from those of the first embodiment, and the same contents will not be described repeatedly.

[0089] The second embodiment of the present invention provides a method for optimizing the communication quality of a drone swarm based on deep reinforcement learning, which comprises the following steps:

[0090] Step 1: The central control device detects the two drones U in the drone swarm that are communicating. k and U j The parameters of the sending device, the parameters of the receiving device, the sending action, the sending status, the receiving action, the receiving status and the reward are given to the two drones for sending errors;

[0091] Step 2: The two UAVs modify the parameters of the sending control strategy network, the sending device, the receiving control strategy network and the receiving device according to the received error.

[0092] The beneficial effects of the second embodiment of the present invention are the same as those of the first embodiment, and will not be described again here.

[0093] Third embodiment

[0094] The third embodiment of the present invention provides a computer program product, which encodes the method involved in the first embodiment, the OFDM symbol generator and the bit stream restorer in the first embodiment, and the method of the second embodiment into program codes that can be called by a processor and stored in a computer storage medium using a computer program language.

[0095] The beneficial effects of the third embodiment of the present invention are the same as those of the first embodiment, and will not be described again here.

[0096] Fourth embodiment

[0097] The fourth embodiment of the present invention also provides a computer storage medium for storing computer program code, which can be called and executed by a processor to implement the method involved in the first embodiment, the OFDM symbol generator and bit stream restorer in the first embodiment, and the method of the second embodiment.

[0098] The beneficial effects of the second embodiment of the present invention are the same as those of the first embodiment, and will not be described again here.

[0099] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A UAV swarm communication quality optimization system based on deep reinforcement learning, characterized in that: It includes a central control device and a drone swarm. Each drone in the drone swarm includes a control strategy network. Transmitter, receiving control strategy network and the receiving device, where θ1 is the parameter of the sending control strategy network, For sending status, is the sending action, θ2 is the parameter of the receiving control strategy network, In receiving state, The central control device receives the action according to the drone U in the drone swarm that is communicating. k and UAV j The parameters of the transmitting device, the parameters of the receiving device, the sending action, the sending state, the receiving action, the receiving state and the reward are given to the two drones for sending errors; the two drones modify the parameters of the sending control strategy network, the transmitting device, the receiving control strategy network and the receiving device according to the received errors; The central control device includes a first reinforcement learning module, and the first reinforcement learning module includes a first value network First target value network and the first error calculator, where w 1now is the parameter of the first value network, is the parameter of the first target value network, and They are the UAVs for communication k The sending action and sending state of the sending device at that time t, and They are the UAVs for communication j The receiving action and receiving state of the receiving device at that time t; the process of training the first reinforcement learning module includes: S1-01: The first value network makes a prediction to obtain the first prediction value S1-02: UAV k The sending device makes a prediction to obtain a predicted sending action In the formula, For UAV k The sending state at the next time t+1, For UAV k Parameters of the sending device; S1-03: UAV j The receiving device makes a prediction to obtain a predicted receiving action In the formula, For UAV j The receiving state at the next time t+1, For UAV j Parameters of the receiving device; S1-04: The first target value network makes a prediction to obtain the first time variance target value In the formula, For UAV k The reward received by the sending device, For UAV j The reward obtained by the receiving device; γ1 is the adjustment coefficient; S1-05: The first error calculator is based on Calculate the first error δ1 and use the first error δ1 to update the first value network parameter w 1now w 1new : In the formula, α1 is the adjustment coefficient, is the gradient function of w1; S01-06: The first value network parameter uses the updated first value network parameter w 1new Update the first target value network parameters for S01-07: The central control device sends the first error δ1 to the drones U k and UAV j .

2. The UAV swarm communication quality optimization system based on deep reinforcement learning according to claim 1 is characterized in that: UAV k and UAV j Receive δ1 sent by the central control device and implement the following process: S1-08: Update the drone U through the following method k Send the parameters of the control strategy network for In the formula, for The gradient function of ; β1 is the adjustment coefficient; S1-09: Update the drone U through the following method j Receive the parameters of the control strategy network for In the formula, for The gradient function, β2 is the adjustment coefficient; S1-10: UAV k Send control strategy network utilization Update UAV k Parameters of the sending device for S1-11: UAV j The receiving control strategy network utilizes Update UAV j Parameters of the receiving device for 3. The UAV swarm communication quality optimization system based on deep reinforcement learning according to claim 2 is characterized in that: The central control device also includes a second reinforcement learning module, which includes a second value network Second target value network and the second error calculator, where w 2now is the parameter of the second value network, is the parameter of the second target value network; and They are the UAVs for communication k The receiving action and receiving state of the receiving device at that time t, and They are the UAVs for communication j The sending action and sending state of the sending device at that time t; The process of training the second reinforcement learning module includes: S2-01: The second value network makes a prediction to obtain the second prediction value S2-02: UAV j The sending device makes a prediction to obtain a predicted sending action In the formula, For UAV j The sending state at the next time t+1, For UAV j Parameters of the sending device; S2-03: UAV k The receiving device makes a prediction to obtain a predicted receiving action In the formula, For UAV k The receiving state at the next time t+1, For UAV k Parameters of the receiving device; S2-04: The second target value network makes a prediction to obtain the second time variance target value In the formula, For UAV k The receiving device receives the reward, For UAV j The reward obtained by the sending device; γ2 is the adjustment coefficient; S2-05: The second error calculator is based on Calculate the second error δ2 and use the second error δ2 to update the second value network parameter w 2now w 2new : In the formula, α2 is the adjustment coefficient, is the gradient function of w2; S2-06: The second value network parameter uses the updated second value network parameter w 2new Update the second target value network parameters for S02-07: The central control device sends the second error δ2 to the drones U k and UAV j .

4. The UAV swarm communication quality optimization system based on deep reinforcement learning according to claim 3 is characterized in that: UAV k and UAV j The second bit error difference δ2 sent by the central control device is received, and the following process is implemented: S2-08: Update the drone U through the following method k The parameters of the receiving control strategy network for In the formula, for The gradient function; β3 is the adjustment coefficient; S2-09: Update the drone U through the following method j Send the parameters of the control strategy network for In the formula, for The gradient function; β4 is the adjustment coefficient; S2-10: UAV k The receiving control strategy network utilizes Update UAV k Parameters of the receiving device for S2-11: UAV j The sending control strategy network utilizes Update UAV j Parameters of the sending device for 5. A method for optimizing the communication quality of drone swarms based on deep reinforcement learning, characterized in that: The steps include: Step 1: The central control device detects the drones communicating in the drone swarm. k and UAV j The parameters of the sending device, the parameters of the receiving device, the sending action, the sending state, the receiving action, the receiving state and the reward are given to the two drones for sending errors; Step 2: Drone U k and UAV j Modify the parameters of the sending control strategy network, the sending device, the receiving control strategy network and the receiving device according to the received error; wherein the central control device includes a first reinforcement learning module, and the first reinforcement learning module includes a first value network First target value network and the first error calculator, where w 1now is the parameter of the first value network, is the parameter of the first target value network, and They are the UAVs for communication k The sending action and sending state of the sending device at that time t, and They are the UAVs for communication j The receiving action and receiving state of the receiving device at that time t; the process of training the first reinforcement learning module includes: S1-01: The first value network makes a prediction to obtain the first prediction value S1-02: UAV k The sending device makes a prediction to obtain a predicted sending action In the formula, For UAV k The sending state at the next time t+1, For UAV k Parameters of the sending device; S1-03: UAV j The receiving device makes a prediction to obtain a predicted receiving action In the formula, For UAV j The receiving state at the next time t+1, For UAV j Parameters of the receiving device; S1-04: The first target value network makes a prediction to obtain the first time variance target value In the formula, For UAV k The reward received by the sending device, For UAV j The reward obtained by the receiving device; γ1 is the adjustment coefficient; S1-05: The first error calculator is based on Calculate the first error δ1 and use the first error δ1 to update the first value network parameter w 1now w 1new : In the formula, α1 is the adjustment coefficient, is the gradient function of w1; S01-06: The first value network parameter uses the updated first value network parameter w 1new Update the first target value network parameters for S01-07: The central control device sends the first error δ1 to the drones U k and UAV j .

6. The method for optimizing the communication quality of drone swarm based on deep reinforcement learning according to claim 5 is characterized in that: UAV k and UAV j Receive δ1 sent by the central control device and implement the following process: S1-08: Update the drone U through the following method k Send the parameters of the control strategy network for In the formula, for The gradient function of ; β1 is the adjustment coefficient; S1-09: Update the drone U through the following method j Receive the parameters of the control strategy network for In the formula, for The gradient function, β2 is the adjustment coefficient; S1-10: UAV k Send control strategy network utilization Update UAV k Parameters of the sending device for S1-11: UAV j The receiving control strategy network utilizes Update UAV j Parameters of the receiving device for 7. The method for optimizing the communication quality of drone swarm based on deep reinforcement learning according to claim 6 is characterized in that: The central control device also includes a second reinforcement learning module, which includes a second value network Second target value network and the second error calculator, where w 2now is the parameter of the second value network, is the parameter of the second target value network; and They are the UAVs for communication k The receiving action and receiving state of the receiving device at that time t, and They are the UAVs for communication j The sending action and sending state of the sending device at that time t; The process of training the second reinforcement learning module includes: S2-01: The second value network makes a prediction to obtain the second prediction value S2-02: UAV j The sending device makes a prediction to obtain a predicted sending action In the formula, For UAV j The sending state at the next time t+1, For UAV j Parameters of the sending device; S2-03: UAV k The receiving device makes a prediction to obtain a predicted receiving action In the formula, For UAV k The receiving state at the next time t+1, For UAV k Parameters of the receiving device; S2-04: The second target value network makes a prediction to obtain the second time variance target value In the formula, For UAV k The receiving device receives the reward, For UAV j The reward obtained by the sending device; γ2 is the adjustment coefficient; S2-05: The second error calculator is based on Calculate the second error δ2 and use the second error δ2 to update the second value network parameter w 2now w 2new : In the formula, α2 is the adjustment coefficient, is the gradient function of w2; S2-06: The second value network parameter uses the updated second value network parameter w 2new Update the second target value network parameters for S02-07: The central control device sends the second error δ2 to the drones U k and UAV j .

8. The method for optimizing the communication quality of drone swarm based on deep reinforcement learning according to claim 7 is characterized in that: UAV k and UAV j The second error δ2 sent by the central control device is received, and the following process is implemented: S2-08: Update the drone U through the following method k The parameters of the receiving control strategy network for In the formula, for Gradient function; β3 is the adjustment coefficient S2-09: Update the drone U through the following method j Send the parameters of the control strategy network for In the formula, for The gradient function; β4 is the adjustment coefficient; S2-10: UAV k The receiving control strategy network utilizes Update UAV k Parameters of the receiving device for S2-11: UAV j Send control strategy network utilization Update UAV j Parameters of the sending device for

Images