A fleet cooperation data collection method based on reinforcement learning

Abstract

The application discloses a kind of based on reinforcement learning's machine group cooperation data acquisition method, comprising the following steps: S1, constructs environment mathematical model, constructs the mathematical model of cooperation data acquisition in mobile crowd sensing scene and can be solved and can be analyzed, environment mathematical model includes: scene parameter construction, data value classification and cooperation data acquisition;S2, establish perception model, perception model is used in unmanned aerial vehicle mobile crowd data acquisition process cooperation perception target position, relative velocity and target quantity information, and perception model includes: distance perception and relative velocity perception.The based on reinforcement learning's machine group cooperation data acquisition method disclosed in the application combines multi-agent reinforcement learning and deep learning, utilizes deep neural network to solve the representation problem in wide state action space, and provides intelligent decision-making ability by the trial-and-error mechanism of reinforcement learning, enhances the ability of efficient collection of heterogeneous data in a generalized manner in multiple dimensions.

Description

Technical Field

[0001] This invention belongs to the field of cluster collaborative data acquisition technology, specifically relating to a cluster collaborative data acquisition method based on reinforcement learning. Background Technology

[0002] With the continuous development of the Internet of Things, mobile computing technology and virtual digital technology, virtual digital technologies such as big data and cloud computing are gradually being applied to various fields of people's daily production and life. Data acquisition, as the foundation of virtual digital technology, mainly refers to the process of collecting non-electrical or electrical signals from sensors or other analog devices and digital measured units. Digital acquisition provides a data foundation for virtual digital technology to identify and detect the real state, and plays a vital role in the field of virtual digital technology.

[0003] Mobile crowdsensing is a method of effectively collecting data using ubiquitous mobile devices, mainly used to improve the efficiency and accuracy of data collection. Among them, drones, as a highly mobile and ubiquitous mobile device, have the advantages of high mobility, high flexibility and wide data collection coverage when drone swarms are used for data collection. At the same time, data collection is also a basic task in mobile crowdsensing.

[0004] Existing data acquisition methods typically optimize the number of objects to be collected, neglecting the heterogeneity of these objects and failing to consider the collaborative relationships between mobile swarm intelligence sensors, as well as energy efficiency and collision avoidance issues that arise during data acquisition. This results in low collaborative data acquisition efficiency among multiple mobile swarm intelligence sensors in complex scenarios.

[0005] The information disclosed in this background section is intended only to enhance the understanding of the overall background of the invention and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0006] The purpose of this invention is to provide a data acquisition method based on reinforcement learning for collaborative machine swarms, which can perform efficient data acquisition in complex, multi-constraint environments.

[0007] To achieve the above objectives, the technical solution provided by a specific embodiment of the present invention is as follows:

[0008] A data acquisition method for swarm collaboration based on reinforcement learning includes the following steps:

[0009] S1. Construct an environmental mathematical model, and construct a solvable and analyzable mathematical model for collaborative data collection in a mobile crowd sensing scenario. The environmental mathematical model includes: scenario parameter construction, data value classification, and collaborative data collection.

[0010] S2. Establish a perception model, which is used to collaboratively perceive target position, relative speed and target quantity information during the data collection process of UAV mobile swarm. The perception model includes: distance perception and relative speed perception.

[0011] S3. Reinforcement learning, wherein the reinforcement learning method transforms the data quality-driven optimal data acquisition problem into a Markov decision process;

[0012] S4. Algorithm parameter learning and updating: The learning and updating of algorithm parameters is used to prioritize the collection of high-value data in complex multi-constraint environments.

[0013] In one or more embodiments of the present invention, the scene parameter construction consists of a drone swarm U@{u=1,2,LU}, a given target area M@{m=1,2,LM}, obstacles B@{b=1,2,L b,LB}, and a data value set W@{w1,w2,…}, wherein the data values ​​of all points of interest in the environmental mathematical model in S1 are not equal.

[0014] In one or more embodiments of the present invention, the movement direction of the UAV swarm in S2 is: u θ ∈[0, 2π), the UAV swarm moves within a given target area and is guided to a new location for data collection according to a strategy generated by reinforcement learning.

[0015] In one or more embodiments of the present invention, the perception model uses multiple perception lines to simulate the process of UAV airborne radar or sensors perceiving the target's azimuth, distance, and position. The perception model calculates the target's azimuth based on the positional relationship between the target and the perception lines, thereby measuring the distance and relative speed between the target and the UAV moving group.

[0016] In one or more embodiments of the present invention, the equation for calculating the number of sensing lines is: N L =π / (arctan(r) m / R pl The sensing line includes: L = {l1, l2, ..., l...} NL Arbitrary sensing line and initial sensing line l1.

[0017] In one or more embodiments of the present invention, the distance sensing includes: and This indicates the location of the drones, and the spacing between the drones is:

[0018] Δx(t)=x m (t)-y m (t), Δy(t)=y m (t)-yu (t),

[0019] The distance C between the given target area and each sensing line i The defining equation for (l, m) is:

[0020] C i (l, m) = Δx(t)·cosα l +Δy(t)·sinα l .

[0021] In one or more embodiments of the present invention, the equation for determining whether a given target region can be sensed by a sensing line is:

[0022]

[0023] The sensing distance D from the sensing line to the given target area t The defining equation for (l, m) is:

[0024]

[0025] In one or more embodiments of the present invention, the relative velocity sensing includes: and The relative velocity between the sensing line and the data of the given target area, representing the velocity of the UAV swarm and the data of the given target area in a time slot, is defined by the following equation:

[0026] V t (l, m) * )=Δv x (t)·cosα l +Δv y (t)·sinα l

[0027] in,

[0028] In one or more embodiments of the present invention, the algorithm parameter learning and updating method comprises a policy network, a critique network, and a target network. Each of the policy network, critique network, and target network is composed of a DNA structure. The critique network is updated by minimizing a loss function, specifically as follows:

[0029]

[0030] in,

[0031] In one or more embodiments of the present invention, the policy network is updated using gradients, specifically as follows:

[0032]

[0033] Compared with the prior art, the reinforcement learning-based swarm collaborative data acquisition method disclosed in this invention combines multi-agent reinforcement learning and deep learning, and is suitable for UAV collaborative data collection scenarios. It uses deep neural networks to solve the representation problem in a broad state-action space, and provides intelligent decision-making capabilities through the trial-and-error mechanism of reinforcement learning.

[0034] By designing an environment collaboration awareness model, a heterogeneous data collection strategy, and a guided reward function, the ability to efficiently collect heterogeneous data in a generalized manner is enhanced across multiple dimensions. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a functional diagram of a machine swarm cooperative data acquisition method based on reinforcement learning in one embodiment of the present invention;

[0037] Figure 2 This is a neural network structure of a UAV swarm in one embodiment of the present invention;

[0038] Figure 3 This is a functional diagram of a criticism network in one embodiment of the present invention;

[0039] Figure 4 This is a diagram illustrating the actor network functionality in one embodiment of the present invention;

[0040] Figure 5 This is a flowchart of a swarm collaborative data acquisition method for reinforcement learning according to an embodiment of the present invention;

[0041] Figure 6 This is used to update the weights of the participant-critic network in one embodiment of the present invention. Detailed Implementation

[0042] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.

[0043] like Figures 1 to 6 As shown, an embodiment of the present invention provides a swarm cooperative data acquisition method based on reinforcement learning, comprising the following steps:

[0044] S1. Construct an environmental mathematical model. Construct a solvable and analyzable mathematical model for collaborative data collection in a mobile crowd sensing scenario. The environmental mathematical model includes: scenario parameter construction, data value classification, and collaborative data collection.

[0045] The scene parameter construction consists of a drone swarm U@{u=1,2,LU}, a given target area M@{m=1,2,LM}, obstacles B@{b=1,2,L b,LB}, and a data value set W@{w1,w2,…}.

[0046] Specifically, the obstacles are buildings such as skyscrapers and high-voltage towers.

[0047] Furthermore, it cannot be assumed that the data value of all points of interest is equal when constructing environmental mathematical models. For example, in the military field, the information value of combat command centers and communication centers is usually higher than that of other combat units; in the fields of rescue and disaster relief, information collection from areas of high population density takes priority over other targets.

[0048] Meanwhile, collaborative data acquisition is based on a dual-drone collaboration method for data collection.

[0049] S2. Establish a perception model. The perception model is used to collaboratively perceive target position, relative speed and target quantity information during the data collection process of UAV mobile swarms. The perception model includes: distance perception and relative speed perception.

[0050] S3. Reinforcement learning: Reinforcement learning methods transform the optimal data acquisition problem driven by data quality into a Markov decision process.

[0051] S4. Algorithm parameter learning and updating: The learning and updating of algorithm parameters is used to prioritize the collection of high-value data in complex, multi-constrained environments.

[0052] In S2, the direction of movement of the drone swarm is: u θ ∈[0, 2π), the drone swarm moves within a given target area and is guided to a new location for data collection based on a policy generated by reinforcement learning.

[0053] Simultaneously, the perception model employs multiple perception lines to simulate the process of UAV-borne radar or sensors sensing the target's azimuth, distance, and position. Based on the positional relationship between the target and the perception lines, the model calculates the target's azimuth, thereby measuring the distance and relative speed between the target and the UAV swarm. The target's azimuth can be calculated from the positional relationship between the target and the perception lines, and then the distance and relative speed between the target and the UAVs can be measured.

[0054] Specifically, the equation for calculating the number of sensing lines is: N L =π / (arctan(r) m / R pl The sensing lines include: L = {l1, l2, ..., l...} NL Arbitrary sensing line and initial sensing line I1.

[0055] Meanwhile, distance perception is based on: and This indicates the location of the drones, and the spacing between the drones is:

[0056] Δx(t)=x m (t)-y m (t), Δy(t)=y m (t)-y u (t),

[0057] Given the distance C between the target region and each sensing line i The defining equation for (l, m) is:

[0058] C i (l, m) = Δx(t)·cosα l +Δy(t)·sinα l .

[0059] Furthermore, the equation for determining whether a given target area can be sensed by sensing lines is:

[0060]

[0061] The sensing distance D from the sensing line to the given target area t The defining equation for (l, m) is:

[0062]

[0063] Specifically, during data acquisition, a sensing line can sense multiple given target areas, but it will select the given target area with the closest sensing distance to calculate the sensing distance between the sensing line and the given target area. Obviously, if the sensing distance changes in the next time step, the distance sensing information will also change.

[0064] Among them, relative speed perception includes: and The relative velocity between the sensing line and the data of the given target area, representing the velocity of the UAV swarm and the data of the given target area in a time slot, is defined by the following equation:

[0065] V t (l, m) * )=Δv x (t)·cosα l +Δv y (t)·sinα l

[0066] in,

[0067] Specifically, the distance and relative speed information perceived by the drone are used as part of the state space, and the corresponding actions are output through a policy network of reinforcement learning.

[0068] Specifically, we treat the drone data collection process as a Markov decision process, defined as: M =<S,A,K,R,γ> γ>S={S i =(S t1 S t2 S t3 )}.

[0069] The detailed composition of all distance information from all drones to the given target area is as follows:

[0070]

[0071]

[0072]

[0073] Among them, S t，1 This indicates the relative speed information of the drone from time to a given target area; This represents the relative speed information of a drone with data values ​​to a given target area; The distance information from the drone's perception line to the given target area is represented by data values.

[0074] like Figures 3 to 4 As shown, the algorithm's parameter learning and update method consists of a policy network and a critique / target network. The policy network, critique network, and target network all share a DNA structure. The critique network is updated by minimizing the loss function, specifically:

[0075]

[0076] in,

[0077] The policy network uses gradients for updates, specifically:

[0078]

[0079] like Figures 5 to 6 As shown, in this experiment, the reward value is divided into five parts: the first part is the reward for the drone itself, achieved by controlling the drone's smooth movement and avoiding violent fluctuations; the second part is a guiding value that encourages cooperation between drones; the third part is that the reward for the agent differs depending on the data quality. The fourth part is used to guide the drone to approach high-value data; the fifth part is to indicate the reward value that the drone will receive when two drones capture the same data at the same time; the sixth part is to indicate the penalty based on the distance between the drones.

[0080] The data acquisition method for swarm collaboration based on reinforcement learning disclosed in this invention combines multi-agent reinforcement learning and deep learning. It is applicable to UAV collaborative data collection scenarios. It uses deep neural networks to solve the representation problem in a broad state-action space and provides intelligent decision-making capabilities through the trial-and-error mechanism of reinforcement learning. By designing an environmental collaborative perception model, a heterogeneous data collection strategy, and a guided reward function, it enhances the ability to efficiently collect heterogeneous data in a generalized manner in multiple dimensions.

[0081] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0082] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A swarm collaborative data acquisition method based on reinforcement learning, characterized in that, Includes the following steps: S1. Construct an environmental mathematical model, and construct a solvable and analyzable mathematical model for collaborative data collection in a mobile crowd sensing scenario. The environmental mathematical model includes: scenario parameter construction, data value classification, and collaborative data collection. S2. Establish a perception model, which is used to collaboratively perceive target position, relative speed, and target quantity information during the data collection process of the UAV swarm. The perception model includes distance perception and relative speed perception. The movement direction of the UAV swarm in S2 is: The UAV swarm moves within a given target area and is guided to a new location for data collection based on a strategy generated by reinforcement learning. The perception model uses multiple perception lines to simulate the process of UAV-borne radar or sensors sensing the target's azimuth, distance, and position. The perception model calculates the target's azimuth based on the positional relationship between the target and the perception lines, thereby measuring the distance and relative speed between the target and the UAV swarm. The equation for calculating the number of perception lines is: The sensing line includes: arbitrary sensing line and initial sensory line ; S3. Reinforcement learning, wherein the reinforcement learning method transforms the data quality-driven optimal data acquisition problem into a Markov decision process; S4. Algorithm parameter learning and updating: The learning and updating of algorithm parameters is used to prioritize the collection of high-value data in complex, multi-constraint environments. The algorithm parameter learning and updating method consists of a policy network, a critique network, and a target network. Each of the policy network, critique network, and target network is composed of a DNA structure. The critique network is updated by minimizing a loss function, specifically: in, .

2. The data acquisition method for machine swarm collaboration based on reinforcement learning according to claim 1, characterized in that, The scene parameters are constructed by a drone swarm. Given target area ,obstacle and data value set The data values ​​of all points of interest in the environmental mathematical model in S1 are not equal.

3. The data acquisition method for machine swarm collaboration based on reinforcement learning according to claim 1, characterized in that, In the distance perception, the following is used: and This indicates the location of the drones, and the spacing between the drones is: ， The distance between the given target area and each sensing line The defining equation is: 。 4. The reinforcement learning-based swarm cooperative data acquisition method according to claim 3, characterized in that, The equation for determining whether a given target area can be sensed by sensing lines is: ， The sensing distance from the sensing line to the given target area The defining equation is: 。 5. The data acquisition method for machine swarm collaboration based on reinforcement learning according to claim 1, characterized in that, In the relative velocity perception, the following is used: and The relative velocity between the sensing line and the data of the given target area, representing the velocity of the UAV swarm and the data of the given target area in a time slot, is defined by the following equation: in, , .

6. The data acquisition method for machine swarm collaboration based on reinforcement learning according to claim 1, characterized in that, The policy network is updated using gradients, specifically as follows: 。

Images