A crowd-sensing incentive mechanism method based on deep reinforcement learning

Abstract

The present invention relates to a method for a crowd-sensing incentive mechanism based on deep reinforcement learning. The method obtains the location and movement trajectory information of participants, models the participant movement process as a Markov decision process, uses deep reinforcement learning to predict their movement trajectory in the next incentive cycle, predicts the location distribution of participants at the end of the next incentive cycle, and calculates the relative entropy between the predicted location distribution of participants and the target distribution of perception data provided by the data requester, and selects participants within an area greater than the relative entropy threshold for incentives. The present invention avoids incentivizing all participants in the same time period and incentivizing the same participant in all incentive cycles. The reasonable incentive mechanism solves the problem of large differences between the distribution of perception data collected by crowd-sensing participants and the target data distribution provided by the data requester, and low coverage quality. The method can be widely used in the field of mobile crowd-sensing to reduce the cost of incentivizing participants.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing systems or methods specifically suitable for administrative, trade, financial, management, supervisory or forecasting purposes, and in particular to a crowd sensing incentive mechanism method based on deep reinforcement learning in the field of mobile crowd sensing. Background Art

[0002] In recent years, with the rapid growth of the perception and computing capabilities of smart devices (such as mobile phones and tablets) and the development of wireless communication technologies, the methods of information collection and sharing have also been quietly changing. As an emerging model for sensing the environment, collecting data, and providing information services, mobile crowdsensing is a current research hotspot in computer science.

[0003] At present, mobile crowd sensing also has certain limitations. One of them is that the distribution of the perception data collected by participants is quite different from the target data distribution provided by the data requester, resulting in low coverage quality. For example, if a data requester wants to collect air quality data for a city, in order to make the results representative, the locations of the perception data need to be evenly distributed in the city. However, participants are usually concentrated in crowded places such as shopping malls and restaurants, while places such as suburban factories are sparsely populated. An incentive mechanism is usually adopted to encourage participants to move from their original locations by means of rewards, so that the overall location distribution of participants is close to the target location distribution of the perception data to solve this problem. Common incentive strategies include: (1) Reward incentive, based on game theory, to fully mobilize the enthusiasm of participants to participate in the perception task by giving rewards. (2) Game incentive, designing the perception task into a game form, and making participants feel interesting by completing the task to obtain points and rankings. (3) Social incentive, establishing a trust system between participants through existing social networks, and enabling participants to gain more recognition from others in social relationships by completing the perception task.

[0004] However, traditional incentive strategies typically prioritize the static distribution of participants when calculating costs, ignoring the impact of participant mobility on incentive costs. Therefore, incentives are typically applied across the entire sensing process, with a high budget set before incentives are implemented. When the number of participants grows exponentially, costs skyrocket and become difficult to control. Summary of the Invention

[0005] The present invention solves the problem in the existing technology that the distribution of perception data collected by traditional mobile crowd perception participants is significantly different from the target data distribution provided by the data requester, resulting in low coverage quality. The general incentive mechanism only considers the static distribution of participants and ignores the impact of participant mobility on incentive costs, which leads to soaring incentive costs. A crowd perception incentive mechanism based on deep reinforcement learning is proposed, which uses an improved DDPG algorithm to predict the movement trajectory of participants to decide whether to incentivize them, thereby effectively and reasonably reducing incentive costs.

[0006] The technical solution adopted by the present invention is a method for a crowd-sensing incentive mechanism based on deep reinforcement learning, which includes the following steps:

[0007] Step 1: Set the excitation period T;

[0008] Step 2: Obtain the movement trajectory of the participant in the initial state of each excitation period T;

[0009] Step 3: Based on deep reinforcement learning, the participant's movement trajectory in the current incentive cycle is predicted using the participant's movement trajectory in the initial state to obtain the participant's predicted position at the end of the current incentive cycle;

[0010] Step 4: Based on the predicted position of the participants at the end of the current incentive period, obtain the target distribution of the perception data provided by the data requester and select appropriate participants for incentives;

[0011] Step 5: Calculate the relative entropy between the actual position distribution of participants and the target distribution of perception data at the end of the current incentive cycle and the incentive cost required in the current incentive cycle, and adjust the incentive cycle T based on the calculation results.

[0012] Preferably, the step 2 comprises the following steps:

[0013] Step 2.1: Divide the perception area into an a×b grid system according to the size of the perception area;

[0014] Step 2.2: Divide the excitation period T into time intervals of length t;

[0015] Step 2.3: Every time t, obtain the latitude and longitude information of each participant's current location and convert it to the coordinate system of the a×b grid system, represented by the coordinate (i, j), 1≤i≤a, 1≤j≤b;

[0016] Step 2.5: Based on the coordinate information, obtain the movement trajectory of each participant within the excitation period T.

[0017] Preferably, the deep reinforcement learning is an improved DDPG algorithm.

[0018] Preferably, step 3 comprises the following steps:

[0019] Step 3.1: Initialize the Critic and Actor neural networks in the improved DDPG algorithm;

[0020] Step 3.2: Initialize the memory replay buffer;

[0021] Step 3.3: Generate participant movement trajectory experience data and store it in the replay buffer;

[0022] Step 3.4: Randomly take sample data from the memory replay buffer for training;

[0023] Step 3.5: Output the algorithm to predict the participant's movement trajectory in the next incentive cycle;

[0024] Step 3.6: Obtain the predicted position distribution of participants at the end of the current incentive period.

[0025] Preferably, in step 3.1, the neural network in the Critic part is and The neural network in the Actor part is μ(s i |θ μ ), the corresponding Target network is and μ′(s i |θ μ′ ); where s i Represents the participant's last moving state, s i+1 represents the next moving state, a represents the moving direction, θ μ ,θ μ′ Represent the weight parameters of these six networks respectively.

[0026] Preferably, the step 3.3 comprises the following steps:

[0027] Step 3.3.1: Initialize random noise N to perturb the training process;

[0028] Step 3.3.2: Get the participant's current movement state s t ;

[0029] Step 3.3.3: Set s t and noise N into the strategy function to get the next moving direction a t ;

[0030] Step 3.3.4: Execute a t , get reward r t and the next step participant moves to state st+1 ;

[0031] Step 3.3.5: Set (s t , a t , r t , s t+1 ) is stored in the memory replay buffer.

[0032] Preferably, in step 3.3.3, the strategy function is a t =μ(s t |θ μ )+N.

[0033] Preferably, the step 3.4 comprises the following steps:

[0034] Step 3.4.1: Randomly extract data from the memory replay buffer (s i , a i , r i , s i+1 );

[0035] Step 3.4.2: Substitute the parameters of Q1′ and Q2′ into the Bellman equation respectively, process the extracted data, select the result with the smaller value as the training data label value, and obtain the training data label value y i =r i +γQ′(s i+1 , μ′(s i+1 |θ μ′ )|θ Q′ ) min , where r i represents the reward value of the participant to reach a specific moving state, γ represents the learning rate, Q′(s i+1 , μ′(s i+1 |θ μ′ )|θ Q′ ) min Represents the smaller value of the two target network calculation results corresponding to the Critic part;

[0036] Step 3.4.3: Extract training data and propagate the error backward to update the critic network parameters.

[0037] Step 3.4.4: Update the actor network parameters μ by backpropagating the gradient.

[0038] Step 3.4.5: Update the target network parameters of the Critic and Actor parts.

[0039] Preferably, the step 4 comprises the following steps:

[0040] Step 4.1: Obtain the target distribution of the sensing data provided by the data requester;

[0041] Step 4.2: Calculate the relative entropy between the predicted participant position distribution and the target distribution of the perception data;

[0042] Step 4.3: Set the relative entropy threshold and select participants whose relative entropy in the perception area is greater than the threshold area for incentives, otherwise no incentives will be given.

[0043] Preferably, the step 5 comprises the following steps:

[0044] Step 5.1: Count the number of participants incentivized in the current incentive cycle;

[0045] Step 5.2: Calculate the incentive cost to be paid based on the number of participants to be incentivized;

[0046] Step 5.3: Calculate the relative entropy between the actual position distribution of the participants and the target distribution of the perception data at the end of the current incentive period;

[0047] Step 5.4: Adjust the incentive period T according to the incentive cost and relative entropy.

[0048] The present invention provides a crowd perception incentive mechanism method based on deep reinforcement learning. The location and movement trajectory information of the participants are obtained through the GPS sensors of the smart devices carried by the crowd perception participants, the movement process of the participants is modeled as a Markov decision process, and the movement trajectory of the participants in the next incentive cycle is predicted by deep reinforcement learning. The location distribution of the participants at the end of the next incentive cycle is predicted, and then the relative entropy of the predicted location distribution of the participants and the target distribution of the perception data provided by the data requester is calculated, and the participants in the area greater than the relative entropy threshold are selected for incentives.

[0049] This paper improves the traditional deep reinforcement learning (DDPG) algorithm by adopting the idea of ​​Double Q-learning. It splits the original critic network into two and designs two independent networks to calculate different Q values. By selecting the minimum Q value, it suppresses the continuous overestimation of the Q value and reduces the calculation error caused by the overestimation of the Q value.

[0050] The present invention can select appropriate participants for incentives, avoiding the need to incentivize all participants in the same time period or the need to incentivize the same participant in all incentive cycles. It solves the problem of large discrepancies between the distribution of perception data collected by crowd perception participants and the target data distribution provided by the data requester, and the resulting low coverage quality, through a reasonable incentive mechanism. The invention can be widely used in the field of mobile crowd perception to reduce the cost of incentivizing participants. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 Flow chart of the method of the present invention. DETAILED DESCRIPTION

[0052] The present invention is further described in detail below with reference to the embodiments, but the protection scope of the present invention is not limited thereto.

[0053] The present invention relates to a method for a crowd-sensing incentive mechanism based on deep reinforcement learning, the method comprising the following steps:

[0054] Step 1: Set the excitation period T;

[0055] In the present invention, the incentive period T refers to an incentive performed every T time interval. The perception data requester divides the entire time process of mobile crowd intelligence perception into multiple incentive periods, and selects appropriate participants for incentives in each incentive period.

[0056] In the present invention, the incentive period T does not have a clear value range, is not evenly distributed, and can be adjusted based on demand. The demand includes but is not limited to the budget, the total perception time, and the number of participants. The perception data requester needs to consider the actual situation to formulate it; when it is necessary to control costs, the incentive period should be set longer and the frequency should be reduced. If it is necessary to narrow the gap between the participant position distribution and the perception data target distribution as soon as possible, the incentive period should be set shorter and the number of times should be increased; in short, T≥0.

[0057] Step 2: Obtain the movement trajectory of the participant in the initial state of each excitation period T;

[0058] The step 2 comprises the following steps:

[0059] Step 2.1: Divide the perception area into an a×b grid system according to the size of the perception area;

[0060] Step 2.2: Divide the excitation period T into time intervals of length t;

[0061] Step 2.3: Every time t, obtain the latitude and longitude information of each participant's current location and convert it to the coordinate system of the a×b grid system, represented by the coordinate (i, j), 1≤i≤a, 1≤j≤b;

[0062] Step 2.5: Based on the coordinate information, obtain the movement trajectory of each participant within the excitation period T.

[0063] In the present invention, participants refer to people who participate in the collection of perception data.

[0064] In the present invention, the movement trajectory refers to the change in the position coordinates of the participant in the grid system during the previous incentive cycle; the movement trajectory of the participant in the previous incentive cycle is used as the input of the deep reinforcement learning algorithm, which can predict the change in the position coordinates of the participant in the next incentive cycle without incentives, as well as the specific position distribution of the participant at the end of the next incentive cycle; the position is compared with the target distribution of the perception data, and if the difference exceeds the threshold, the corresponding participant needs to be incentivized in the next incentive cycle.

[0065] In the present invention, step 2.1 is to model the irregular perception area into a regular one. The perception area can be specifically quantified into an a×b grid system by the data requester, and the location information and movement trajectory changes of the participants can be accurately represented in the form of coordinates.

[0066] In the present invention, the longitude and latitude information of the participant's location in step 2.3 can be submitted manually by the participant, or can be automatically obtained by the perception data requester through a wireless network.

[0067] Step 3: Based on deep reinforcement learning, the participant's movement trajectory in the current incentive cycle is predicted using the participant's movement trajectory in the initial state to obtain the participant's predicted position at the end of the current incentive cycle;

[0068] The deep reinforcement learning is an improved DDPG algorithm.

[0069] The step 3 comprises the following steps:

[0070] Step 3.1: Initialize the Critic and Actor neural networks in the improved DDPG algorithm;

[0071] In step 3.1, the neural network in the Critic part is and The neural network in the Actor part is μ(s i |θ μ ), the corresponding Target network is and μ′(s i |θ μ′ ); where s i Represents the participant's last moving state, s i+1 represents the next moving state, a represents the moving direction, θ μ ,θ μ′ Represent the weight parameters of these six networks respectively.

[0072] In the present invention, in step 3.1, the movement process of the participant is modeled as a Markov process; the improved deep reinforcement learning DDPG algorithm is used to predict the movement trajectory of the participant, using the Critic and Actor neural networks and the corresponding Target networks of the two parts; since the movement of the participant is a continuous action, the network parameters of Q1, Q2 and μ will be updated in real time after each training, while the network parameters of Q1′, Q2′ and μ′ will not be updated until multiple steps of training have passed, thereby disrupting the correlation between the actions and improving the learning effect.

[0073] In the present invention, in step 3.1, the network of the Critic part in the traditional DDPG algorithm is split into two, and two networks for calculating Q values ​​are defined. This is because the DDPG algorithm is derived from the DQN algorithm, and the DQN algorithm generally has the problem of overestimation, that is, the Q value is calculated too high. Although this can speed up convergence, the error in Q value calculation will also become larger and larger as the number of training times increases. Therefore, the idea of ​​Double Q-learning is adopted, and two networks are designed separately to calculate different Q values. By selecting the smallest Q value, the continuous overestimation of Q value is suppressed. The Q value here is a conventional concept in deep reinforcement learning, which can be understood as the value of the action (movement direction) taken by the participant.

[0074] Step 3.2: Initialize the memory replay buffer;

[0075] In the present invention, in step 3.2, the memory bank refers to a storage structure for storing past learning experiences. It is a memory bank used to store the participant's movement trajectory learning experience. The trained data can be directly stored in the memory bank. Each new learning can be randomly sampled from the replay buffer. This can also disrupt the correlation between continuous action samples and improve the learning effect.

[0076] Step 3.3: Generate participant movement trajectory experience data and store it in the replay buffer;

[0077] The step 3.3 includes the following steps:

[0078] Step 3.3.1: Initialize random noise N to perturb the training process;

[0079] Step 3.3.2: Get the participant's current movement state s t ;

[0080] Step 3.3.3: Set s t and noise N into the strategy function to get the next moving direction a t ;

[0081] In step 3.3.3, the policy function is a t =μ(s t |θ μ )+N.

[0082] Step 3.3.4: Execute a t , get reward r t and the next step participant moves to state s t+1 ;

[0083] Step 3.3.5: Set (s t , a t , r t , s t+1 ) is stored in the memory replay buffer.

[0084] In the present invention, in step 3.3, the strategy function a t =μ(s t |θ μ )+N, a t Adding noise N to the actor movement direction predicted by the actor network μ can increase randomness for the algorithm, so that it can explore as many situations as possible and make learning more efficient. In applications, the noise N can be selected to include but not limited to Gaussian distribution noise and Poisson noise.

[0085] In the present invention, in step 3.3, the empirical data refers to the historical training data, which is the memory bank mechanism of the DDPG algorithm. As an offline learning algorithm, the DDPG algorithm stores past training results in the memory bank. Each time the parameters of the neural network are updated, sample data in the memory bank can be randomly extracted for learning, which can disrupt the correlation of the original continuous actions and make the update of the neural network more efficient. Of course, if it is in the cold start state, there is no need to retrieve data from the memory bank, and the training results can be directly stored.

[0086] In this invention, in step 3.3.2, t is used primarily to distinguish between two processes: decision-making and updating network parameters by sampling from the memory bank. t represents the decision-making process, which predicts the participant's movement direction and stores the result in the memory bank.

[0087] In the present invention, the reward in step 3.3.4 is a conventional concept in the reinforcement learning algorithm. The greater the "reward" a participant obtains by moving to a specific location, the higher the possibility that the participant will move to that location next time. The "reward" rule is set in advance based on the target distribution of the perceived data of the data requester.

[0088] In the present invention, the "movement state" in step 3.3.4 is obtained by observation.

[0089] Step 3.4: Randomly take sample data from the memory replay buffer for training;

[0090] The step 3.4 includes the following steps:

[0091] Step 3.4.1: Randomly extract data from the memory replay buffer (s i , a i , r i , s i+1 );

[0092] Step 3.4.2: Substitute the parameters of Q1′ and Q2′ into the Bellman equation respectively, process the extracted data, select the result with the smaller value as the training data label value, and obtain the training data label value y i =r i +γQ′(s i+1 , μ′(s i+1 |θ μ′ )|θ Q′ ) min , where r i represents the reward value of the participant to reach a specific moving state, γ represents the learning rate, Q′(s i+1 , μ′(s i+1 |θ μ′ )|θ Q′ ) min Represents the smaller value of the two target network calculation results corresponding to the Critic part;

[0093] Step 3.4.3: Extract training data and propagate the error backward to update the critic network parameters.

[0094] Step 3.4.4: Update the actor network parameters μ by backpropagating the gradient.

[0095] Step 3.4.5: Update the target network parameters of the Critic and Actor parts.

[0096] In the present invention, the training set label y in step 3.4.2 i is the Bellman equation, with Q′(s i+1 , μ′(s i+1 |θ μ′ )|θ Q′ ) minThe smaller value of the two target network calculation results corresponding to the Critic part is taken to suppress the continuous overestimation of the Q value; among them, the reward value is preset by the data requester. Participants can obtain specific rewards by moving to a specific location. The reward value may also be negative, indicating that the location where the participant moves is too far from the target distribution of the perception data.

[0097] In the present invention, in step 3.4.3, the mean square error function can be used to update the parameters of the critic network, and the regression loss Where M is the number of sample training data collected, y i Represents the label value of the training data, The estimated value of the participant's moving direction for the current choice is obtained by the critic part network corresponding to the target network that obtains the smaller value.

[0098] In the present invention, in step 3.4.4, the parameters of the Actor network μ can be updated by gradient back propagation, and the gradient of μ The gradient of the first half, Q, is calculated from the Critic network, indicating how the participant should move to obtain a larger Q. The gradient of μ is calculated from the Actor network, indicating how to modify the parameters to make it more likely that the participant will move in that direction. That is, the Actor network μ should modify the parameters in a direction that is more likely to obtain a larger Q.

[0099] In the present invention, in step 3.4, the Target partial network Q′ i and μ′ parameters and θ μ′ The update is done through soft update, that is, not all updates are done at once, but only a little bit at a time, that is, Where τ is the update coefficient, which is selected by the data requester based on actual conditions and is generally a small number such as 0.1 or 0.01.

[0100] Step 3.5: Output the algorithm to predict the participant's movement trajectory in the next incentive cycle;

[0101] In the present invention, this part is to update the target network, that is, to update Q′ and μ′.

[0102] Step 3.6: Obtain the predicted position distribution of participants at the end of the current incentive period.

[0103] Step 4: Based on the predicted position of the participants at the end of the current incentive period, obtain the target distribution of the perception data provided by the data requester and select appropriate participants for incentives;

[0104] The step 4 comprises the following steps:

[0105] Step 4.1: Obtain the target distribution of the sensing data provided by the data requester;

[0106] Step 4.2: Calculate the relative entropy between the predicted participant position distribution and the target distribution of the perception data;

[0107] Step 4.3: Set the relative entropy threshold and select participants whose relative entropy in the perception area is greater than the threshold area for incentives, otherwise no incentives will be given.

[0108] In the present invention, the target distribution of perception data is provided by the data requester, and is the position distribution of the perception data collected by the expected participants within the perception area.

[0109] In this paper, relative entropy is used to measure the difference between the participant location distribution and the target distribution of perceived data. This serves as a criterion for determining whether to incentivize participants. Participants in areas with greater relative entropy are more likely to be incentivized. The goal of incentives is to reduce relative entropy, bringing the participant location distribution as close as possible to the target distribution of perceived data. Relative entropy is the difference between the participant location distribution and the target distribution of perceived data, and its calculation method is well known to those skilled in the art.

[0110] Step 5: Calculate the relative entropy between the actual position distribution of participants and the target distribution of perception data at the end of the current incentive cycle and the incentive cost required in the current incentive cycle, and adjust the incentive cycle T based on the calculation results.

[0111] The step 5 comprises the following steps:

[0112] Step 5.1: Count the number of participants incentivized in the current incentive cycle;

[0113] Step 5.2: Calculate the incentive cost to be paid based on the number of participants to be incentivized;

[0114] Step 5.3: Calculate the relative entropy between the actual position distribution of the participants and the target distribution of the perception data at the end of the current incentive period;

[0115] Step 5.4: Adjust the incentive period T according to the incentive cost and relative entropy.

[0116] In the present invention, the purpose of step 5 is to make the participant position distribution and the target distribution of the perception data fit as much as possible (the relative entropy is as low as possible), and to control the incentive cost. Therefore, the incentive period T needs to be continuously adjusted throughout the perception process to optimize these two indicators. If the relative entropy between the participant position distribution and the target distribution of the perception data is still large after the incentive, then the length of the subsequent incentive period should be shortened and incentives should be performed more frequently to obtain a better fit. If the calculated incentive cost is high, or even exceeds the budget, then the length of the subsequent incentive period should be extended to control the cost.

Claims

1. A method for crowd-sensing incentive mechanism based on deep reinforcement learning, characterized by: The method comprises the following steps: Step 1: Set the excitation period T; Step 2: Obtain the participant's movement trajectory in the initial state of each excitation cycle T, including the following steps: Step 2.1: Divide the perception area into an a×b grid system according to the size of the perception area; Step 2.2: Divide the excitation period T into time intervals of length t; Step 2.3: At every time t, the GPS sensors of the smart devices carried by the crowd-sensing participants are used to obtain the location of each participant. The latitude and longitude information of each participant's current location is obtained and converted into the coordinate system of the a×b grid system, represented by the coordinate (i, j), 1≤i≤a, 1≤j≤b; Step 2.5: Based on the coordinate information, obtain the movement trajectory of each participant within the incentive period T; Step 3: Based on deep reinforcement learning, the participant's movement trajectory in the current incentive cycle is predicted using the participant's movement trajectory in the initial state to obtain the participant's predicted position at the end of the current incentive cycle; Step 4: Based on the predicted position of the participants at the end of the current incentive period, obtain the target distribution of the perception data provided by the data requester and select appropriate participants for incentives; Step 5: Calculate the relative entropy between the actual location distribution of participants and the target distribution of perception data at the end of the current incentive cycle, as well as the incentive cost required in the current incentive cycle. Adjust the incentive cycle T based on the calculation results, including the following steps: Step 5.1: Count the number of participants incentivized in the current incentive cycle; Step 5.2: Calculate the incentive cost to be paid based on the number of participants to be incentivized; Step 5.3: Calculate the relative entropy between the actual position distribution of the participants and the target distribution of the perception data at the end of the current incentive period; Step 5.4: Adjust the incentive period T according to the incentive cost and relative entropy.

2. The method of a crowd-sensing incentive mechanism based on deep reinforcement learning according to claim 1, characterized in that: The deep reinforcement learning is an improved DDPG algorithm.

3. The method of the crowd intelligence perception incentive mechanism based on deep reinforcement learning according to claim 2 is characterized by: The step 3 comprises the following steps: Step 3.1: Initialize the Critic and Actor neural networks in the improved DDPG algorithm; Step 3.2: Initialize the memory replay buffer; Step 3.3: Generate participant movement trajectory experience data and store it in the replay buffer; Step 3.4: Randomly take sample data from the memory replay buffer for training; Step 3.5: Output the DDPG algorithm to predict the participant's movement trajectory in the next incentive cycle; Step 3.6: Obtain the predicted position distribution of participants at the end of the current incentive period.

4. The method of the crowd intelligence perception incentive mechanism based on deep reinforcement learning according to claim 3 is characterized by: In step 3.1, the neural network in the Critic part is and The neural network in the Actor part is μ(s i |θ μ ), the corresponding Target network is and μ′(s i |θ μ′ ); where s i Represents the participant's last moving state, s i+1 represents the next moving state, a represents the moving direction, θ μ ,θ μ′ Represent the weight parameters of these six networks respectively.

5. The method of a crowd-sensing incentive mechanism based on deep reinforcement learning according to claim 3 is characterized by: The step 3.3 includes the following steps: Step 3.3.1: Initialize random noise N to perturb the training process; Step 3.3.2: Get the participant's current movement state s t ; Step 3.3.3: Set s t and noise N into the strategy function to get the next moving direction a t ; Step 3.3.4: Execute a t , get reward r t and the next step participant moves to state s t+1 ; Step 3.3.5: Set (s t ,a t ,r t ,s t+1 ) is stored in the memory replay buffer.

6. The method of a crowd-sensing incentive mechanism based on deep reinforcement learning according to claim 5, characterized in that: In step 3.3.3, the policy function is a t =μ(s t |θ μ )+N.

7. The method of a crowd-sensing incentive mechanism based on deep reinforcement learning according to claim 4 is characterized by: The step 3.4 includes the following steps: Step 3.4.1: Randomly extract data from the memory replay buffer (s i ,a i ,r i ,s i+1 ); Step 3.4.2: Substitute the parameters of Q1′ and Q2′ into the Bellman equation respectively, process the extracted data, select the result with the smaller value as the training data label value, and obtain the training data label value y i =r i +γQ′(s i+1 ,μ′(s i+1 |θ μ′ )|θ Q′ ) min , where r i represents the reward value of the participant to reach a specific moving state, γ represents the learning rate, Q′(s i+1 ,μ′(s i+1 |θ μ′ )|θ Q′ ) min Represents the smaller value of the two target network calculation results corresponding to the Critic part; Step 3.4.3: Extract training data and propagate the error backward to update the critic network parameters. Step 3.4.4: Update the actor network parameters μ by backpropagating the gradient. Step 3.4.5: Update the target network parameters of the Critic and Actor parts.

8. The method of a crowd intelligence perception incentive mechanism based on deep reinforcement learning according to claim 1 is characterized by: The step 4 comprises the following steps: Step 4.1: Obtain the target distribution of the sensing data provided by the data requester; Step 4.2: Calculate the relative entropy between the predicted participant position distribution and the target distribution of the perception data; Step 4.3: Set the relative entropy threshold and select participants whose relative entropy in the perception area is greater than the threshold area for incentives, otherwise no incentives will be given.

Images