A user recruitment method for vehicle crowd-sensing based on deep reinforcement learning

Abstract

The present invention discloses a method for recruiting users for vehicle crowd intelligence perception based on deep reinforcement learning. The specific steps are as follows: constructing a vehicle network scenario under a highway, the constructed scenario including a highway, several vehicles, a communication base station, a cloud center server and a task publisher; establishing a channel model for communication between the vehicle and the base station, a model of the vehicle's cost and reputation, and a utility model of the cloud center server; establishing an optimization problem with the goal of maximizing task coverage; designing the input transition state, reward function and output action of DDPG; using the vehicle's information as the input state, obtaining the output action under various states, and training and updating the neural network according to the reward function. The present invention takes into account the communication between the vehicle and the base station, the vehicle's reputation and cost, and the task budget, has practicality in actual scenarios, and has a high task coverage; defines the input state information of DDPG and designs an effective reward function.

Description

Technical Field

[0001] The present invention relates to the field of vehicle crowd sensing, and in particular to a method for recruiting users for vehicle crowd sensing based on deep reinforcement learning. Technical Background

[0002] Mobile crowd sensing (MCS) has become increasingly popular in recent years, becoming an attractive paradigm for sensing and collecting data. MCS relies on the built-in sensors and communication capabilities of mobile devices, such as smartphones and wearables, to upload sensed data to cloud servers. Unlike traditional sensor networks, MCS systems do not require the deployment of dedicated sensors.

[0003] With the rise of the Internet of Vehicles (IoV), researchers have found that the built-in sensors, communication resources, computing platforms, and mobility of vehicles are more in line with the requirements of MCS for participants, so vehicle crowd sensing (VCS) was born. VCS generally consists of a task requester, a cloud center server, and participating vehicles. The main process of VCS includes: (1) the task requester publishes the task to the cloud center; (2) the cloud center server publishes the tasks it receives to the participating vehicles; (3) the participating vehicles select the tasks they are interested in and submit information such as their location, reputation, and task price (i.e., the vehicle's quotation for the task, and the reward it will receive once the task is completed) to the cloud center; (4) the cloud center server selects a specific vehicle to complete the relevant task based on the information submitted by the vehicle; (5) the vehicle completes the task and uploads the task data; (6) the cloud center server rewards the vehicle.

[0004] However, vehicles have limited and valuable communication and computing resources. To ensure that vehicles complete their missions, the system must provide incentives. The road model currently used in VCS user recruitment technology does not comply with 3GPP regulations and fails to adequately consider factors such as vehicle-to-base station communication, vehicle reputation and costs, and mission budget, resulting in limited practicality. The algorithm for predicting vehicle travel time (Timeliness-Aware Incentive Mechanism for Vehicular Crowdsourcing in Smart Cities) considers both mission budget and vehicle costs, but not vehicle-to-base station communication and vehicle reputation. Furthermore, its road model also fails to comply with 3GPP regulations, resulting in limited practicality. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of the existing technology and propose a new method for recruiting users for vehicle crowd intelligence perception, which comprehensively considers factors such as communication between vehicles and base stations, vehicle reputation and cost, and task budget, maximizes task coverage while having higher cloud center server utility.

[0006] The present invention is achieved through at least one of the following technical solutions.

[0007] A method for recruiting users for vehicle crowd-sensing based on deep reinforcement learning, comprising the following steps:

[0008] S1. Construct a connected vehicle scenario on a highway. The constructed scenario includes a highway, several vehicles, communication base stations, a cloud center server, and a task publisher.

[0009] S2. Establish a channel model for communication between the vehicle and the base station, taking into account the location of the vehicle and the base station, antenna gain, and noise factors; establish a cost and reputation model for the vehicle and a utility model for the cloud center server;

[0010] S3. Build an optimization problem model based on constraints with the goal of maximizing task coverage.

[0011] S4. Based on the optimization problem model and the server utility of the cloud center, a DDPG network structure is constructed. The DDPG network structure includes the input transition state of the DDPG network, the DDPG reward function, and the DDPG output action.

[0012] S5. Based on the DDPG network structure of step 4, the vehicle information is used as the input state to obtain the output actions under various states, and the DDPG network is trained and updated according to the reward function.

[0013] Furthermore, in step S1, a scenario of recruiting users for vehicle crowd sensing on a highway is constructed: the highway is wl kilometers long, and lane n The number of lanes, each lane is lane wide w meters, where the number of vehicles in lane i is N i By Poisson sampling we get:

[0014]

[0015] In the above formula, k represents the number of vehicles obtained by Poisson sampling, and the intensity λ is equal to the length of the highway divided by the average distance between vehicles:

[0016]

[0017] In the above formula, vS represents the speed of the vehicle;

[0018] So the number of vehicles Vehicle i,j It is represented as the jth vehicle in the i-th lane.

[0019] Furthermore, a channel model for communication between the vehicle and the base station is established, taking into account factors such as communication distance, vehicle and base station antenna gains, and noise. The vehicle coordinates are (x1, y1, z1) and the base station coordinates are (x2, y2, z2). Therefore, the communication distance d is obtained by the following formula:

[0020]

[0021] According to the communication distance d, the path loss between the vehicle and the base station is obtained by the following formula and recorded as pI:

[0022] pl=128.1+37.6lgd (4)

[0023] The combined path loss is calculated as follows, denoted as combinedPL:

[0024] combinedPL=-(pl+sD*stdV2I) (5)

[0025] In the above formula, sD is the logarithmic normal distribution of shadow attenuation, and stdV2I is the standard deviation of shadow attenuation;

[0026] The channel gain is calculated by the following formula, denoted as gain:

[0027] gain=combinedPL+vaG+baG-bsN (6)

[0028] In the above formula, vaG is the vehicle antenna gain, baG is the base station antenna gain, and bsN is the base station noise;

[0029] Finally, the signal-to-noise ratio (SNR) between the vehicle and the base station is calculated using the following formula, denoted as SNR:

[0030] snr=vtP+gain-bgN (7)

[0031] In the above formula, vtP is the transmission power of the vehicle and bgN is the background noise.

[0032] Furthermore, a cost, pricing, and reputation model for the vehicle is established. The cost of the vehicle is related to the amount of data it senses and transmits.

[0033] For vehicle v i,j , its perception cost is calculated by the following formula:

[0034] cS i,j =sL i,j *sW i,j *udV i,j *usP i,j(8)

[0035] In the above formula, cS i,j is the perceived cost, sL i,j is the perceived length of the vehicle, sW i,j is the perceived width of the vehicle, udV i,j It is the data perceived by the unit sensing area, usP i,j It is the price cost required to perceive unit data;

[0036] For vehicle v i,j The transmission cost is calculated as follows:

[0037]

[0038] In the above formula, cT i,j is the transmission cost, utP i,j is the price cost required to transmit unit data, snr i,j Is the vehicle v i,j The signal-to-noise ratio of communications with the base station;

[0039] For vehicle v i,j The cost of completing a task is calculated as follows:

[0040] c i,j =α*cS i,j +β*cT i,j (10)

[0041] In the above formula, c i,j Is the vehicle v i,j The cost of completing the task, α and β are weight values, both greater than zero, and α + β = 1;

[0042] Based on the cost of the vehicle, the list price of the vehicle is obtained by the following formula:

[0043] b i,j =ga i,j *c i,j (11)

[0044] In the above formula, b i,j It's the price tag, ga i,j is the proportionality coefficient;

[0045] When a vehicle completes a task, its reputation value re will be updated. min ,R max ] and is divided into H levels, namely,

[0046]

[0047] In the above formula

[0048] where re i is the reputation value of vehicle i, R min is the minimum reputation value, R max is the maximum reputation value, The difference between two adjacent reputation values;

[0049] The reputation update process of a vehicle is modeled as a first-order Markov process. The H levels of the reputation value re correspond to the H transition states of the Markov process, and its state transition matrix is:

[0050]

[0051] In the state transfer matrix, p i,j represents the probability of transitioning from the i-th transition state to the j-th transition state; h represents a total of h reputation values, 1≤i≤h, 1≤j≤h;

[0052] Furthermore, a utility model of the cloud center server is established, which mainly considers the budget of the task and the cost of recruiting vehicles, and the utility of the cloud center server is recorded as u s , then:

[0053]

[0054] In the above formula, b i,j represents the price of the task for the jth vehicle in the i-th lane, lane n is the number of lanes, N i is the number of vehicles in the i-th lane, tB is the budget of the task, which is provided by the task requester when publishing the task to the cloud center, and x i,j It is the action output by the recruitment method, which becomes 0 or 1 after the function f. The f function is as follows:

[0055]

[0056] Furthermore, in step S3, with the goal of maximizing the mission coverage, an optimization problem is established based on the recruited vehicles and the constraints:

[0057]

[0058] c1:re m ≥re th 1≤m≤M

[0059]

[0060] c3:P(N i =k)=e -λ λ k / k! k=0,1,2,…λ>0

[0061] In the above formula, re m is the reputation value of the mth car, M means that a total of M cars are recruited, re th represents the reputation threshold required by the task, λ represents the vehicle spacing, k represents the number of vehicles obtained by Poisson sampling, and x i,j represents the action value of the jth vehicle in the i-th lane, η is the ratio of the road area perceived by all recruited vehicles to the highway road area, lane n is the number of lanes, N i is the number of vehicles in the i-th lane; s i,j Represents vehicle v i,j The perception area, function f(x i,j ) If it is 1, it means vehicle v i,j Recruited, function f(x i,j ) If it is 0, it means that the vehicle v i,j not recruited; Indicates the calculation of the perceived area of ​​lane i, and the repeated perceived area needs to be removed; wl represents the length of the highway, lane w represents the width of the lane; the first constraint c1 indicates that the reputation value of the recruited vehicle cannot be less than the reputation threshold required by the task; the second constraint c2 indicates the price b of all recruited vehicles i,j The sum cannot be greater than the task budget tB; the third constraint c3 indicates that the total number of vehicles in each lane conforms to the Poisson distribution.

[0062] Furthermore, in step S4, the server in the cloud center uses the DDPG algorithm in reinforcement learning to solve the optimization problem given in step S3, and numbers the vehicles in all lanes in ascending order to obtain a one-dimensional vehicle number vector;

[0063] The input state of DDPG is as follows:

[0064]

[0065] The input state matrix s contains the relevant information required by N vehicles, where no i Indicates the number of the i-th vehicle; po i represents the position of the i-th vehicle; sL i represents the perceived length of the i-th vehicle; sW i represents the perceived width of the i-th vehicle; re i represents the reputation of the i-th car; b i Indicates the price of the i-th car;

[0066] The output actions of DDPG are as follows:

[0067]

[0068] Where a represents the output action vector, x i,j represents the action value of the jth vehicle in the i-th lane, -1≤x i,j ≤1;x i,j After the function f, it gets 0 or 1, where 1 means the car is recruited and 0 means it is not;

[0069] The reward function of DDPG is as follows:

[0070]

[0071] In the above formula, r represents the reward function of DDPG, which consists of three parts. The first part is the task coverage rate η obtained by the recruited vehicles. The larger the value of this part, the better. The second part is For successful recruitment of vehicles, there is a function f(x i,j ) is equal to 1, and there is re i,j -re th ≥0, the larger the value of this part, the better; the third part is Indicates the utility of the cloud center server, and the larger the better; lane n is the number of lanes, b i,j is the price, i,j is the reputation value of the jth vehicle in the i-th lane, re th Represents the reputation threshold required by the task; w1, w2, and w3 are the weights of each part and are hyperparameters; the second part represents the satisfaction of the first constraint c1, the third part represents the satisfaction of the second constraint c2, and the third constraint c3 is automatically satisfied when the vehicle is generated.

[0072] Furthermore, in step S5, based on the input state matrix s, reward function r, and output action vector a obtained in step S4, the reinforcement learning algorithm DDPG is selected and the DDPG network structure is constructed.

[0073] Furthermore, the DDPG algorithm includes an environment, an action network (Actor network), a criticism network (Crtic network), a target action network (Target_Actor network), a target criticism network (Target_Critic network), and a buffer pool Buffer. The Actor network includes a convolutional layer and four linear layers, and the Critic network includes one convolutional layer and seven linear layers. The Target_Actor network has the same parameters as the Actor network, and the Target_Critic network has the same parameters as the Ctiric network.

[0074] Furthermore, the training and updating process of the DDPG network is as follows:

[0075] S5.1. Randomly initialize the Actor network μ(s|θμ )’s weight parameter θ μ and Critic network Q(s,a|θ Q )’s weight parameter θ Q , randomly initialize the experience replay pool Buffer;

[0076] S5.2. Initialize the weight parameters of the Target_Actor network μ′ and the Target_Critic network Q′, namely: θ μ′ ←θ μ ,θ Q′ ←θ Q ; where μ′ is the Target_Actor network, θ μ′ is the weight parameter of Target_Actor network, Q′ is the Target_Critic network, θ μ′ Weight parameters of the Target_Critic network.

[0077] S5.3. Initialize the Internet of Vehicles environment to obtain the input state s(t);

[0078] S5.4, Actor network obtains output action a(t) based on input state s(t);

[0079] S5.5. Output action a(t) to act on the constructed vehicle network scenario, obtain reward r(t), and transfer to the next state s(t+1);

[0080] S5.6. Store the current experience [s(t), a(t), r(t), s(t+1)] into the experience replay pool Buffer;

[0081] S5.7. Extract samples from the experience replay pool Buffer to train and update the Actor network and Crtic network;

[0082] S5.8. Periodically update the Target_Actor network and the Target_Critic network.

[0083] The present invention has the following advantages and effects compared to the prior art:

[0084] (1) The present invention is the first to use the highway model specified by 3GPP in the road model in the field of vehicle crowd intelligence perception user recruitment.

[0085] (2) The present invention comprehensively considers the communication between the vehicle and the base station, the reputation and cost of the vehicle and the budget of the task, and has a higher task coverage rate and better cloud center server utility.

[0086] (3) Based on the vehicle crowd-sensing user recruitment scenario, the present invention defines the input state information of DDPG and designs an effective reward function, and the model has good robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] Figure 1 This is a flowchart of a method for recruiting users for vehicle crowd intelligence perception based on deep reinforcement learning disclosed in an embodiment of the present invention;

[0088] Figure 2 Schematic diagram of a highway scenario in which vehicles perceive users through crowd intelligence in an embodiment of the present invention;

[0089] Figure 3 Schematic diagram of the communication distance between a vehicle and a base station in an embodiment of the present invention;

[0090] Figure 4 is a schematic diagram of the DDPG algorithm in an embodiment of the present invention;

[0091] Figure 5 This is a diagram showing the effect of task budget on task coverage when the vehicle speed is fixed in an embodiment of the present invention;

[0092] Figure 6 This is a diagram showing the effect of task budget on the cloud center server when the vehicle speed is fixed in an embodiment of the present invention;

[0093] Figure 7 This is a diagram showing the effect of vehicle speed on task coverage when the task budget is fixed in an embodiment of the present invention;

[0094] Figure 8 This is a diagram showing the effect of vehicle speed on the utility of the cloud center server when the task budget is fixed in an embodiment of the present invention. DETAILED DESCRIPTION

[0095] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the embodiments described are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0096] like Figure 1 As shown, a method for recruiting users for vehicle crowd sensing based on deep reinforcement learning disclosed in this embodiment uses the DDPG algorithm as an optimization algorithm to solve the problem of recruiting users for vehicle crowd sensing on highways. The specific steps include:

[0097] S1. Build a vehicle networking scenario on a highway, such as Figure 2The figure shows a schematic diagram of a highway scenario constructed according to an embodiment of the present invention, including a highway, several vehicles, communication base stations, a cloud center server, and a task publisher. Vehicles possess certain perception, computing, and communication capabilities, and the number of vehicles complies with 3GPP regulations; the cloud center server possesses powerful computing capabilities. The task under consideration is road environment monitoring; the task requester submits the task to the cloud center, which then distributes the task's perception range, budget, and reputation threshold to the vehicles within the crowd-sensing system. To complete the task, a vehicle submits its price, location, and other information to the cloud center. The cloud center selects a specific vehicle to complete the task based on the information submitted by the vehicle. Upon completing the task, the vehicle receives a reward from the cloud center.

[0098] As a preferred embodiment, the road environment detection task needs to cover a 2km long highway with 3 one-way lanes, each lane is 4 meters wide, and there are several vehicles in each lane. i is the number of vehicles in the i-th lane. According to 3GPP regulations, N i Should be obtained by Poisson sampling:

[0099]

[0100] In the above formula, the intensity λ is equal to the length of the highway divided by the average distance between vehicles:

[0101]

[0102] In the above formula, vS represents the speed of the vehicle in kilometers per hour, and 2.5 refers to 2.5 seconds.

[0103] So the total number of vehicles N is equal to Here is the lane n is 3,. Vehicle v i,j It is represented as the jth vehicle in the i-th lane.

[0104] S2. Build a channel model for communication between the vehicle and the base station, taking into account factors such as the location of the vehicle and base station, antenna gain, and noise. Build models for the vehicle's cost and reputation, as well as a model for the cloud center's server benefits.

[0105] S2.1. Establish a channel model for communication between the vehicle and the base station, taking into account factors such as communication distance, vehicle and base station antenna gain, and noise. Figure 3 As shown, the base station is 35 meters vertically from the roadside and 35 meters high, located in the middle of the mission road. The vehicle is 1.5 meters high, and its position changes over time. Let the vehicle coordinates be (x1, y1, z1) and the base station coordinates be (x2, y2, z2). Therefore, the communication distance d (in kilometers) can be calculated using the following formula:

[0106]

[0107] According to the communication distance d, the path loss (denoted as pI) between the vehicle and the base station can be calculated:

[0108] pl=128.1+37.6lgd (4)

[0109] In addition to path loss, shadow attenuation also needs to be considered:

[0110] combinedPL=-(pl+sD*stdV2I) (5)

[0111] In the above formula, combinedPL is the combined path loss, sD is the logarithmic normal distribution obeyed by shadow fading, stdV2I is the standard deviation of shadow fading, and the value of stdV2I is 8 dB in this embodiment.

[0112] The channel gain is calculated by the following formula, denoted as gain:

[0113] gain=combinedPL+vaG+baG-bsN (6)

[0114] In the above formula, vaG is the vehicle antenna gain, baG is the base station antenna gain, and bsN is the base station noise. In this embodiment, the value of vaG is 3dBi, the value of baG is 8dBi, and the value of bdN is 5dB.

[0115] Finally, the signal-to-noise ratio (SNR) between the vehicle and the base station is calculated using the following formula, denoted as SNR:

[0116] snr=vtP+gain-bgN (7)

[0117] In the above formula, vtP is the vehicle's transmit power, and bgN is the background noise. The value of vtP is 23dBm, and the value of bgN is -114dBm. Obviously, the signal-to-noise ratio (snr) changes as the vehicle moves.

[0118] S2.2. Establish a cost, pricing, and reputation model for vehicles. The cost of a vehicle is related to the size of the data it senses and transmits.

[0119] For vehicle v i,j , its perception cost can be calculated by the following formula:

[0120] cS i,j =sL i,j *sW i,j *udV i,j *usP i,j (8)

[0121] In the above formula, cS i,jis the perceived cost, sL i,j is the perceived length of the vehicle, sW i,j is the perceived width of the vehicle, udV i,j It is the data perceived by the unit sensing area, usP i,j Is the price cost required to perceive unit data. i,j ∈[60,100], sW i，j The value is 4m, udV i,j The value of usP is 10MB. i,j ∈[1,5].

[0122] For vehicle v i,j , its transmission cost can be calculated by the following formula:

[0123]

[0124] In the above formula, cT i,j is the transmission cost, utP i,j is the price cost required to transmit unit data, snr i,j Is the vehicle v i,j The signal-to-noise ratio for communication with the base station. usP i,j ∈[1,5]. Obviously, snr i,j The larger the value of, the better the channel environment is and the lower the transmission cost of the vehicle is.

[0125] For vehicle v i,j , its cost can be calculated by the following formula:

[0126] c i,j =α*cS i,j +β*cT i,j (10)

[0127] In the above formula, c i,j Is the vehicle v i,j The cost of completing the task, α and β are weight values, both greater than zero, and α + β = 1. In this embodiment, α = 0.7.

[0128] Based on the cost of the vehicle, the list price of the vehicle can be obtained by the following formula:

[0129] b i,j =ga i,j *c i,j (11)

[0130] In the above formula, b i,j It's the price tag, ga i,j is the proportional coefficient, because vehicles are rational and will not complete the task at a price lower than the cost, so ga i,jIs greater than or equal to 1; and because the crowd intelligence perception user recruitment system is an adjustment mechanism, an excessively high price will prevent the vehicle from being selected, so ga i,j There is an upper bound. In this embodiment, ga i,j ≤2.

[0131] When a vehicle completes a mission, its reputation value re will be updated. In this recruitment method, re is in [R min ,R max ] and is divided into H levels. That is,

[0132]

[0133] In the above formula

[0134] The reputation update process of a vehicle is modeled as a first-order Markov process. The H levels of the reputation value re correspond to the H transition states of the Markov process, and its state transition matrix is:

[0135]

[0136] In the state transfer matrix, p i,j represents the probability of transitioning from the i-th transition state to the j-th transition state. In this embodiment, R min is 10, R max is 28, and H is 7. So:

[0137] re i ∈{10,13,16,19,22,25,28}

[0138]

[0139] S2.3. Establish a benefit model for the cloud center server, mainly considering the mission budget and the cost of recruiting vehicles. The benefit of the cloud center server is recorded as u s , then:

[0140]

[0141] In the above formula, tB is the budget of the task, which is provided by the task requester when publishing the task to the cloud center, and x i,j is the action output by the recruit method, x i,j The output of the Actor network is obtained. The activation function of the output layer of the network is tanh. The tanh function is as follows:

[0142]

[0143] So -1≤x ij ≤1,x ijAfter the function f, it becomes 0 or 1. The f function is as follows:

[0144]

[0145] S3. With the goal of maximizing task coverage, combined with constraints, establish an optimization problem.

[0146]

[0147] c1:re k ≥re th 1≤k≤M

[0148]

[0149] c3:P(N i =k)=e -λ λ k / k! k=0,1,2,…λ>0

[0150] In the above formula, η is the coverage of the task, which represents the ratio of the road area perceived by all recruited vehicles to the highway road area; s i,j Represents vehicle v i,j The sensing area of ​​the vehicle is a rectangle in this embodiment; f(x i,j ) If it is 1, it means vehicle v i，j Recruited, f(x i,j ) If it is 0, it means that the vehicle v i,j not recruited; Indicates calculating the perceived area of ​​lane i. It is necessary to remove duplicate perceived areas. This can be achieved by a simple interval merging algorithm. n represents the number of lanes, which is 3 in this embodiment; wl represents the length of the highway to be sensed, which is 2 km in this embodiment; lane w represents the lane width, which is 4 meters in this example. Constraint c1 states that the reputation value of the recruited vehicles cannot be less than the reputation threshold required by the task (this threshold is provided by the task requester when publishing the task to the cloud center); constraint c2 states that the sum of the bid prices of all recruited vehicles cannot be greater than the task budget; constraint c3 states that the total number of vehicles in each lane follows a Poisson distribution.

[0151] S4. Based on the optimization problem model and the cloud center server utility, design the DDPG input transition, DDPG reward function, and DDPG output action.

[0152] To solve the optimization problem in step S3, the cloud center server uses the DDPG algorithm from reinforcement learning. The cloud center collects information about vehicles within the system and obtains the input state for the DDPG algorithm. To obtain a two-dimensional input state matrix, the vehicles in all lanes are numbered in ascending order, resulting in a one-dimensional vehicle number vector. Assuming there are 30 vehicles in lane 1, 35 vehicles in lane 2, and 30 vehicles in lane 3, then vehicles numbered 1-30 are in lane 1, vehicles numbered 31-65 are in lane 2, and vehicles numbered 66-95 are in lane 3.

[0153] The vehicle number, vehicle position, vehicle perceived length, vehicle perceived width, vehicle reputation, and the price given by the vehicle together constitute the input state.

[0154] The input state of DDPG is as follows:

[0155]

[0156] The input state matrix s contains the relevant information required by N vehicles. i Indicates the number of the i-th vehicle; po i represents the position of the i-th vehicle; sL i represents the perceived length of the i-th vehicle; sW i represents the perceived width of the i-th vehicle; re i represents the reputation of the i-th car; b i Indicates that the price of the i-th car is given.

[0157] The output actions of DDPG are as follows:

[0158]

[0159] Where a represents the output action vector, x i,j represents the action value of the jth vehicle in the i-th lane, and -1≤x i,j ≤1;x i,j After the function f, it gets 0 or 1, where 1 means the car is recruited and 0 means it is not. It should be noted that a is a one-dimensional matrix during neural network training. It is written in this form for ease of understanding.

[0160] The reward function of DDPG is as follows:

[0161]

[0162] In the above formula, r represents the DDPG reward function, which consists of three parts. The first part is the mission coverage rate η obtained by the recruited vehicles. The larger the value of this part, the better. The second part is For successfully recruited vehicles, there is f(x i,j ) is equal to 1, and there is rei,j -re th ≥0, the larger the value of this part, the better; the third part is Indicates the server utility of the cloud center, and the larger the better. w1, w2, and w3 are the weights of each part and are hyperparameters. In addition, the second part indicates that constraint 1 is satisfied, the third part indicates that constraint 2 is satisfied, and constraint 3 is automatically satisfied when the vehicle is generated. In this embodiment, lane n is 3, w1 is 1000, w2 is 10, w3 is 100, re th is 20.

[0163] S5. Design the DDPG network structure according to step 4, use the vehicle information as the input state, obtain the output action under various states, and train and update the neural network according to the reward function to obtain the final task coverage and server utility of the cloud center.

[0164] Based on the input state matrix s, reward function r, output action vector a, etc. obtained in step S4, select the reinforcement learning algorithm DDPG and design the network structure.

[0165] As another preferred embodiment, Figure 4 As shown in the figure, the DDPG algorithm includes an environment, an actor network, a critic network, a target actor network, a target critic network, and a buffer. The actor network consists of one convolutional layer and four linear layers, and the critic network consists of one convolutional layer and seven linear layers. The target actor network has the same parameters as the actor network, and the target critic network has the same parameters as the critic network.

[0166] The training and updating process of the DDPG neural network is as follows:

[0167] S5.1. Randomly initialize the weight parameters θμ of the Actor network μ(s|θμ) and the Critic network Q(s, a|θ Q ) weight parameter θQ, and randomly initialize the experience replay pool Buffer.

[0168] S5.2. Initialize the weight parameters of the Target_Actor network μ′ and the Target_Critic network Q′, namely: θ μ′ ←θ μ ,θ Q′ ←θ Q .

[0169] S5.3. Initialize the Internet of Vehicles environment to obtain the input state s(t).

[0170] S5.4. The action network obtains the output action a(t) based on the input state s(t) and the action selection strategy.

[0171] S5.5. Output action a(t) acts on the environment, obtains reward r(t), and transfers to the next state s(t+1).

[0172] S5.6. Store the current experience [s(t), a(t), r(t), s(t+1)] into the experience replay pool Buffer.

[0173] S5.7. Extract samples from the experience replay pool Buffer to train and update the Actor network and Crtic network. The number of samples extracted is bs.

[0174] Crtic network update process: Input s(t) and a(t) in the sample into the Crtic network Q(s, a|θ Q ), we get the Q(s) of the Crtic network t , a t |θ Q ) value, input the sample s(t+1) into the Target_Actor network μ′, and obtain the corresponding action μ′(s t+1 |θ μ′ ), and then s(t+1) and μ′(s i+1 |θ μ′ ) is input to the Target_Critic network Q′, the Target_Critic network Q′(s i+1 , μ′(s i+1 |θ μ′ )|θ Q′ )value.

[0175] y t =r t +γQ′(s t+1 , μ′(s t+1 |θ μ′ )|θ Q′ ) (20)

[0176]

[0177] In the above formula, γ is the reward discount, and the Crtic network can be updated by minimizing the L value.

[0178] Actor Network Updates:

[0179]

[0180] The core idea is to perform gradient descent on the Crtic network and the Actor network.

[0181] S5.8. Periodically update the Target_Actor network and Target_Critic network. Update the parameters of the Target_Actor network and Target_Critic network every C training cycles.

[0182] θ Q′ ←τθ Q +(1-τ)θ Q′ (twenty three)

[0183] θ μ′ ←τθ μ +(1-τ)θ μ′ (twenty four)

[0184] In this embodiment, C is 4 and τ is 0.01.

[0185] After the DDPG network training is completed, the task coverage and the utility of the cloud center server can be calculated based on the output actions. The specific simulation parameters of this embodiment are shown in Table 1.

[0186] Table 1 Simulation parameters are shown in the following table

[0187]

[0188]

[0189] Figure 5 This is the effect of the task budget on the task coverage when the vehicle speed is fixed in the embodiment of the present invention. When the speed is 70km / h, it can be seen that as the budget increases, the task coverage of the three algorithms in the figure increases, but the algorithm BCMCRA of the embodiment of the present invention has the best effect (BCMCRA is the method proposed by the present invention); compared with the algorithm WSP, it is still based on the greedy idea and does not make good use of the vehicle group intelligence to perceive all the information of the user recruitment system. When the budget is sufficient, the task coverage of BCMCRA is about 17% higher than that of WSP; BANDOM is a random algorithm. When the budget is sufficient, the task coverage of BCMCRA is about 77% higher than that of WSP. Because the number of vehicles on the road is limited, as the task budget continues to increase, the task coverage will grow to an upper bound, Figure 5 This is also well proven.

[0190] Figure 6This is the impact of the task budget on the cloud center server utility when the vehicle speed is fixed in the embodiment of the present invention. The speed is 70km / h. First of all, the effect of BCMCRA is better than the comparison algorithm WSP. When the budget is 350, the server utility of BCMCRA is about 2.7 times that of WSP. However, BCMCRA did not exceed the random algorithm. This is because the random algorithm recruits very few vehicles, so the task coverage of the random algorithm is very low, so most of the task budget remains in the cloud center and becomes the utility of the cloud center, but this does not meet the task requirements (first of all, the task coverage is optimized). At the same time, it can be found that when the task budget increases to a certain extent, the server utility of the cloud center is linearly related to the task budget. This is because the increased task budget at this time cannot recruit more vehicles, so it all becomes the utility of the server. Therefore, compared with the control algorithm, BCMCRA is still excellent.

[0191] Figure 7 This is the effect of vehicle speed on mission coverage when the mission budget is fixed in this embodiment of the present invention. For a budget of 400, it can be seen that as speed increases, the mission coverage of all three algorithms decreases. This is because, when the budget is sufficient, the number of vehicles decreases as speed increases, and the number of vehicles available for recruitment also decreases accordingly. However, BCMCRA still outperforms the other two algorithms, with BCMCRA's mission coverage being approximately 0.1 higher than WSP and 0.27 higher than RANDOM.

[0192] Figure 8 This is the effect of vehicle speed on the cloud center server utility when the task budget is fixed in the embodiment of the present invention. The budget is 400. It can be seen that as the speed increases, the cloud center server utility increases. The method BCMCRA proposed in the present invention is better than the comparison algorithm WSP, but does not exceed the random algorithm. This is because the random algorithm recruits very few vehicles, so the task coverage of the random algorithm is very low, so most of the task budget remains in the cloud center and becomes the utility of the cloud center, but this does not meet the task requirements (first of all, the task coverage is optimized). Therefore, compared to the control algorithm, BCMCRA can have better server utility while the task coverage is not lower than that of the control algorithm.

[0193] The above embodiments are preferred embodiments of the present invention, but the implementation methods of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications made without departing from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A method for recruiting users for vehicle crowd intelligence perception based on deep reinforcement learning, characterized in that: The user recruitment method comprises the following steps: S1. Construct a connected vehicle scenario on a highway. The constructed scenario includes a highway, several vehicles, communication base stations, a cloud center server, and a task publisher. S2. Establish a channel model for communication between the vehicle and the base station, taking into account the location of the vehicle and the base station, antenna gain, and noise factors; establish a cost and reputation model for the vehicle and a utility model for the cloud center server; S3. Build an optimization problem model based on constraints with the goal of maximizing task coverage. S4. Based on the optimization problem model and the server utility of the cloud center, a DDPG network structure is constructed. The DDPG network structure includes the input transition state of the DDPG network, the DDPG reward function, and the DDPG output action. S5. Based on the DDPG network structure of step 4, the vehicle information is used as the input state to obtain the output actions under various states, and the DDPG network is trained and updated according to the reward function.

2. The method for recruiting users for vehicle crowd intelligence perception based on deep reinforcement learning according to claim 1 is characterized in that: In step S1, a scenario of recruiting users for vehicle crowd sensing on a highway is constructed: the highway is wl kilometers long, and lane n The number of lanes, each lane is lane wide w meters, where the number of vehicles in lane i is N i By Poisson sampling we get: In the above formula, k represents the number of vehicles obtained by Poisson sampling, and λ represents the ratio of the highway length to the distance between vehicles: In the above formula, vS represents the speed of the vehicle; So the number of vehicles Vehicle i,j It is represented as the jth vehicle in the i-th lane.

3. The method for recruiting users for vehicle crowd intelligence perception based on deep reinforcement learning according to claim 1 is characterized in that: A channel model for communication between the vehicle and the base station is established, taking into account the communication distance, vehicle and base station antenna gains, and noise factors; the vehicle coordinates are (x1, y1, z1), and the base station coordinates are (x2, y2, z2), so the communication distance d is obtained by the following formula: According to the communication distance d, the path loss between the vehicle and the base station is obtained by the following formula and recorded as pI: pl=128.1+37.6lgd (4) The combined path loss is calculated as follows, denoted as combinedPL: combinedPL=-(pl+sD*stdV2I) (5) In the above formula, sD is the logarithmic normal distribution of shadow attenuation, and stdV2I is the standard deviation of shadow attenuation; The channel gain is calculated by the following formula, denoted as gain: gain=combinedPL+vaG+baG-bsN (6) In the above formula, vaG is the vehicle antenna gain, baG is the base station antenna gain, and bsN is the base station noise; Finally, the signal-to-noise ratio (SNR) between the vehicle and the base station is calculated using the following formula: snr=vtP+gain-bgN (7) In the above formula, vtP is the transmission power of the vehicle and bgN is the background noise.

4. The method for recruiting users for vehicle crowd intelligence perception based on deep reinforcement learning according to claim 1 is characterized in that: Establish a cost, pricing, and reputation model for the vehicle, where the cost of the vehicle is related to the amount of data it senses and transmits; For vehicle v i,j , its perception cost is calculated by the following formula: cS i,j =sL i,j *sW i,j *udV i,j *usP i,j (8) In the above formula, cS i,j is the perceived cost, sL i,j is the perceived length of the vehicle, sW i,j is the perceived width of the vehicle, udV i,j It is the data perceived by the unit sensing area, usP i,j It is the price cost required to perceive unit data; For vehicle v i,j The transmission cost is calculated as follows: In the above formula, cT i,j is the transmission cost, utP i,j is the price cost required to transmit unit data, snr i,j Is the vehicle v i,j The signal-to-noise ratio of communications with the base station; For vehicle v i,j The cost of completing a task is calculated as follows: c i,j =α*cS i,j +β*cT i,j (10) In the above formula, c i,j Is the vehicle v i,j The cost of completing the task, α and β are weight values, both greater than zero, and α + β = 1; Based on the cost of the vehicle, the list price of the vehicle is obtained by the following formula: b i,j =ga i,j *c i,j (11) In the above formula, b i,j It's the price tag, ga i,j is the proportionality coefficient; When a vehicle completes a task, its reputation value re will be updated. min ,R max ] and is divided into H levels, namely, In the above formula where re i is the reputation value of vehicle i, R min is the minimum reputation value, R max is the maximum reputation value, The difference between two adjacent reputation values; The reputation update process of a vehicle is modeled as a first-order Markov process. The H levels of the reputation value re correspond to the H transition states of the Markov process, and its state transition matrix is: In the state transfer matrix, p i,j represents the probability of transitioning from the i-th transition state to the j-th transition state; h represents a total of h reputation values, 1≤i≤h, 1≤j≤h.

5. The method for recruiting users for vehicle crowd intelligence perception based on deep reinforcement learning according to claim 1 is characterized in that: Establish a utility model for the cloud center server, mainly considering the budget of the task and the cost of recruiting vehicles, and record the utility of the cloud center server as u s , then: In the above formula, b i,j represents the price of the task for the jth vehicle in the i-th lane, lane n is the number of lanes, N i is the number of vehicles in the i-th lane, tB is the budget of the task, which is provided by the task requester when publishing the task to the cloud center, and x i,j It is the action output by the recruitment method, which becomes 0 or 1 after the function f. The f function is as follows:

6. The method for recruiting users for vehicle crowd intelligence perception based on deep reinforcement learning according to claim 2 is characterized in that: In step S3, with the goal of maximizing the mission coverage, an optimization problem is established based on the recruited vehicles and the constraints: c1:re m ≥re th 1≤m≤M c2: c3:P(N i =k)=e -λ λ k / k!k=0,1,2,…λ>0 In the above formula, re m is the reputation value of the mth car, M means that a total of M cars are recruited, re th represents the reputation threshold required by the task, λ represents the ratio of the highway length to the distance between the vehicles, k represents the number of vehicles obtained by Poisson sampling, and x i,j represents the action value of the jth vehicle in the i-th lane, η is the ratio of the road area perceived by all recruited vehicles to the highway road area, lane n is the number of lanes, N i is the number of vehicles in the i-th lane; s i,j Represents vehicle v i,j The perception area, function f(x i,j ) If it is 1, it means vehicle v i,j Recruited, function f(x i,j ) If it is 0, it means that the vehicle v i,j not recruited; Indicates the calculation of the perceived area of ​​lane i, and the repeated perceived area needs to be removed; wl represents the length of the highway, lane w Represents the width of the lane; The first constraint c1 indicates that the reputation value of the recruited vehicle cannot be less than the reputation threshold required by the task; The second constraint c2 represents the price b of all recruited vehicles i,j The sum cannot be greater than the task budget tB; the third constraint c3 indicates that the total number of vehicles in each lane conforms to the Poisson distribution.

7. The method for recruiting users for vehicle crowd intelligence perception based on deep reinforcement learning according to claim 6 is characterized in that: In step S4, the server in the cloud center uses the DDPG algorithm in reinforcement learning to solve the optimization problem given in step S3, and numbers the vehicles in all lanes in ascending order to obtain a one-dimensional vehicle number vector; The input state of DDPG is as follows: The input state matrix s contains the relevant information required by N vehicles, where no i Indicates the number of the i-th vehicle; po i represents the position of the i-th vehicle; sL i represents the perceived length of the i-th vehicle; sW i represents the perceived width of the i-th vehicle; re i represents the reputation of the i-th car; b i Indicates the price of the i-th car; The output actions of DDPG are as follows: Where a represents the output action vector, x i,j represents the action value of the jth vehicle in the i-th lane, -1≤x i,j ≤1;x i,j After the function f, it gets 0 or 1, where 1 means the car is recruited and 0 means it is not; The reward function of DDPG is as follows: In the above formula, r represents the reward function of DDPG, which consists of three parts. The first part is the task coverage rate η obtained by the recruited vehicles. The larger the value of this part, the better. The second part is For successful recruitment of vehicles, there is a function f(x i,j ) is equal to 1, and there is re i,j -re th ≥0, the larger the value of this part, the better; the third part is Indicates the utility of the cloud center server, and the larger the better; lane n is the number of lanes, b i,j is the price, i,j is the reputation value of the jth vehicle in the i-th lane, re th represents the reputation threshold required by the task; w1, w2, and w3 are the weights of each part and are hyperparameters; The third constraint c3 is automatically satisfied when the vehicle is generated.

8. The method for recruiting users for vehicle crowd intelligence perception based on deep reinforcement learning according to any one of claims 1 to 7, characterized in that: In step S5, based on the input state matrix s, reward function r, and output action vector a obtained in step S4, the reinforcement learning algorithm DDPG is selected and the DDPG network structure is constructed.

9. The method for recruiting users for vehicle crowd intelligence perception based on deep reinforcement learning according to claim 8 is characterized in that: The DDPG algorithm includes an environment, an action network (Actor network), a criticism network (Crtic network), a target action network (Target_Actor network), a target criticism network (Target_Critic network), and a buffer pool. The Actor network includes a convolutional layer and four linear layers, and the Critic network includes one convolutional layer and seven linear layers. The Target_Actor network has the same parameters as the Actor network, and the Target_Critic network has the same parameters as the Ctiric network.

10. The method for recruiting users for vehicle crowd intelligence perception based on deep reinforcement learning according to claim 9 is characterized in that: The training and updating process of the DDPG network is as follows: S5.

1. Randomly initialize the Actor network μ(s|θ μ )’s weight parameter θ μ and Critic network Q(s,a|θ Q )’s weight parameter θ Q , randomly initialize the experience replay pool Buffer; s represents the input state, a represents the output action; S5.

2. Initialize Target_Actor network μ ′ Target_Critic network Q ′ The weight parameter is: θ μ′ ←θ μ ,θ Q′ ←θ Q ; where μ′ is the Target_Actor network, θ μ′ is the weight parameter of Target_Actor network, Q′ is the Target_Critic network, θ Q′ is the weight parameter of the Target_Critic network; S5.

3. Initialize the Internet of Vehicles environment to obtain the input state s(t); S5.4, Actor network obtains output action a(t) based on input state s(t); S5.

5. Output action a(t) to act on the constructed vehicle network scenario, obtain reward r(t), and transfer to the next state s(t+1); S5.

6. Store the current experience [s(t), a(t), r(t), s(t+1)] into the experience replay pool Buffer; S5.

7. Extract samples from the experience replay pool Buffer to train and update the Actor network and Crtic network; S5.

8. Periodically update the Target_Actor network and the Target_Critic network.

Images