Automatic driving Internet of Vehicles task unloading method based on width reinforcement learning

Abstract

The invention discloses an automatic driving Internet of Vehicles task unloading method based on width reinforcement learning, and the method specifically comprises the steps: obtaining data through a VISSIM simulator, taking the data as the input of a width reinforcement learning environment, and providing a basis for the subsequent decision; width reinforcement learning based on dynamic pricing is used for decision making, the number of vehicles capable of being unloaded is determined, and therefore the uncertainty in the unloading process is effectively dealt with. In the task division stage, nearby vehicles suitable for unloading are found out through a type-2 fuzzy system, tasks are ingeniously distributed to the appropriate vehicles, and the task unloading efficiency and accuracy are improved; and reward solution is carried out by applying width reinforcement learning based on dynamic pricing again, so that better reward is achieved, and the utilization rate of resources is further improved. The invention aims to solve the task unloading problem in the automatic driving Internet of Vehicles and realize intelligent and efficient resource management.

Description

Technical Field

[0001] The present invention relates to the field of vehicle networking technology, and in particular to a method for unloading autonomous driving vehicle networking tasks based on width reinforcement learning. Background Art

[0002] The Internet of Vehicle (IoV) is a cloud-based vehicle operation information platform, and its ecosystem includes intelligent transportation systems, logistics, emergency rescue, and other aspects. Autonomous driving is the core service of the development of the Internet of Vehicles, and is also the core application of the development of automobile intelligence and networking. It is committed to organically linking "people-vehicle-road-cloud" through the Internet of Vehicles. With the rapid development of autonomous driving technology and the continuous promotion of the application of connected autonomous driving in real-world scenarios, the Internet of Vehicles has evolved into a new concept - the Internet of Autonomous Vehicles (IoAV). The Internet of Autonomous Vehicles is divided into three layers: physical layer, virtual layer, and management layer.

[0003] At the same time, with the development of autonomous driving technology and computer application technology, there are more types of applications, such as augmented reality, real-time video analysis, human behavior recognition, etc., which require a lot of computing resources when processing these applications. How to provide more computing resources is a key issue that cannot be avoided in the research of autonomous driving vehicle networking. The current answer revolves around a keyword, cloud. However, cloud computing has certain defects due to its inherent limitations. Due to the long distance between the terminal device and the cloud computing center, this greatly increases the cost of communication network deployment and network transmission delay, and also increases the potential transmission security risk. The emergence of edge computing solves the problem in cloud computing: by deploying more cloud-like distributed infrastructure at the edge of the network as close to the data source as possible to provide services to users. If the computing-intensive tasks that cannot be solved by the terminal device are offloaded to the nearby edge device for processing, this can reduce backhaul, transmission delay and bandwidth consumption, and greatly shorten the overall task execution time. As an important extension and extension of mobile cloud computing technology, it is mainly to enhance and expand the computing environment near the user's location, and use local processors to efficiently and timely meet the needs of users at the edge of the network.

[0004] At present, most vehicle task offloading methods use deep reinforcement learning methods, such as DQN, DDQN and other models. However, all of the above algorithms are based on deep network structures, which are bound to have complex structures and numerous parameters. This also leads to a large time and memory cost when using this part of the network training model. At the same time, the offloading strategy based on deep reinforcement learning methods is data-driven, which consumes a lot of computing resources when training and updating models, and greatly increases the task execution and feedback time, which cannot meet the requirements of autonomous driving vehicle networking scenarios for low latency, instant feedback and low energy consumption. Summary of the invention

[0005] In view of the above-mentioned shortcomings of the prior art, the purpose of the present invention is to provide a method for unloading tasks in an autonomous driving vehicle network based on wide reinforcement learning, aiming to solve the problem that the vehicle task unloading method in the prior art has a complex network structure and a large number of parameters, resulting in low data feedback efficiency and unable to meet the requirements of the autonomous driving vehicle network scenario for low latency, instant feedback and low energy consumption.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A method for unloading tasks of an autonomous driving vehicle network based on wide reinforcement learning, comprising:

[0008] Step 1: Obtain vehicle data set through VISSIM simulator. Vehicle data includes vehicle speed and location information.

[0009] Step 2: Propose a width reinforcement learning model; establish a width reinforcement learning model based on a dynamic pricing mechanism, input vehicle data and base station data, train the width reinforcement learning model, and use the dynamic pricing mechanism and the width reinforcement learning model to encourage vehicles to participate in task unloading at the appropriate time and place to determine the number of vehicles that can be unloaded, effectively dealing with distributed task unloading scenarios;

[0010] Step 3: In the task division stage, the type-II fuzzy system is used to comprehensively consider the distance, link reliability, relative speed, available computing resources and dynamic pricing factors of the vehicles to evaluate the suitability of each vehicle, and according to the number of vehicles in the decision, the tasks are assigned to the appropriate vehicles to realize the task offloading of autonomous driving vehicle networking.

[0011] Furthermore, the steps for obtaining the vehicle data set in step 1 are:

[0012] Step 1-1: First, obtain a map of the real scene of the mission vehicle, import the map into the VISSIM simulator, and draw a detailed map of the traffic network, including the width of the road, the number of lanes, traffic signs and signal light information; at the same time, design specific traffic flow, speed distribution and traffic rules parameters to obtain the most realistic data;

[0013] Step 1-2: After the design is completed, run the entire scenario and output the speed and direction information required by the mission vehicle;

[0014] Step 1-3: Process the acquired target vehicle data and convert the task vehicle data format as the input of the width reinforcement learning model.

[0015] Furthermore, the width reinforcement learning model based on the dynamic pricing mechanism in step 2 is composed of two networks, namely Policy-BLS and Target-BLS; wherein, training the distributed task through the width reinforcement learning model includes:

[0016] Step 2-1: The task vehicle sends a request for an uncompleted unloading task to the base station. The base station senses the environmental information, i.e., the vehicle data and base station data in step 1, estimates the wireless channel status and the link duration between the task vehicle and the service vehicle, and sends the status to the Policy-BLS network to obtain feedback action a. n , select the number of vehicles for the unloading task, and then determine the task vehicle and the price that the task vehicle needs to pay to the service vehicle based on the asking price and bid of the service vehicle and the task vehicle; at the same time, the base station writes the experience gained each time into the experience pool;

[0017] Step 2-2: After receiving the allocation message from the base station, the service vehicle reserves some computing resources for the task according to the service price. n , after the task vehicle completes the payment and uploads the task data to the service vehicle, the service vehicle performs the task and finally sends the result receipt to the task vehicle;

[0018] Step 2-3: Pricing based on service¶ n C n And some reserved computing resources, the base station calculates the corresponding reward value R n , and finally written into the experience pool;

[0019] Step 2-4: The experience pool caches the experience learned by the base station at a certain time step (s n , a n , R n ,s n+1 ), which facilitates the base station to perform experience playback learning and random sampling of data and break the correlation between samples;

[0020] Step 2-5: The Training Pool is the storage and training center of the Policy-BLS network. A batch of samples are randomly selected and accumulated based on the experience pool, denoted as {(s i , a i , R i ,s i+1 )} p ; By sending {s i+1} p , and obtain the estimated {max a′ Q T (s i+1 , a′; θ T )} p To prepare the training samples of the Policy-BLS network, the corresponding Q value is recorded as y i ;

[0021] Step 2-6: The Policy-BLS network is used to model the relationship between the input and output of the network to be trained, represented by the action value function Q E Represents; as a set of states s i and the corresponding task assignment action a i As input, y i As output; by minimizing Q E and the output value y i The difference between the two determines the network weight; finally, the computing task offloading action a is determined n , complete the task allocation for service vehicles;

[0022] Step 2-7: The Target-BLS network is consistent with the Policy-BLS network at a certain time step. It is responsible for receiving the input {s i+1} p After that, calculate the optimal action value {max a , Q T (s i+1 , a′; θ T )} p , in preparation for calculating the output value.

[0023] Furthermore, the feedback action a is obtained in step 2-1 n Specifically:

[0024] The Policy-BLS network has two main functions: one is to make decisions, and the other is to train data. The input for making decisions is s t , the output is y, the output y is positively correlated with the Q value of reinforcement learning, and what is ultimately needed is the action decision a t; The dataset for training the network includes input data and label data; the input data is obtained from the replay memory t and a t The label data is based on the reward value r obtained from the replay memory. t Compared with the Q value Q provided by Target-BLS T Combined as follows:

[0025]

[0026] Where a′ represents the action used to maximize the Q value output by Policy-BLS; then, the training data is the basis and prerequisite for training the Policy-BLS network, including the input data and label data required by the network; Policy-BLS is the main neural network in DBRL, which can receive training data, perform network training, and synchronize its parameters α with the Target-BLS network at every K time steps; Target-BLS is the target neural network in DBRL, which receives s t+1 and a′ and calculate the corresponding Q T To support t Calculation of

[0027] To start the algorithm, the agent obtains a state s from the environment t And send it to Policy-BLS, and then Policy-BLS returns the corresponding action; in the action selection process, the ∈ greedy strategy is adopted, that is, a random action is selected with a probability of ∈, and the action with the highest Q value is selected with a probability of 1-∈:

[0028] a t = argmax a Q T (s t , a; α p ).

[0029] Furthermore, the reward R is calculated in step 2-2 and step 2-3. n Specifically:

[0030] In order to measure the effectiveness of vehicle mission execution, a utility function is introduced:

[0031]

[0032] In the above formula, U m Represents the efficiency of completing a task: This equation represents the actual time t required to complete a task m and the maximum acceptable delay in completing the task The logarithmic nature of the equation shows that as the gap between the actual completion time and the maximum allowed delay increases, the decision becomes more effective; conversely, as this difference decreases, the effectiveness of the decision decreases; the overall utility of the service vehicle SV performing the local task in a given time when performing the computational task is expressed as:

[0033]

[0034] in and C l It represents the maximum acceptable delay time for the service vehicle SV to complete the local task, and the number of CPU cycles required to complete the local task. While executing the computing task, the service vehicle SV also needs to process the local task; the minimum allocation frequency of the local task:

[0035]

[0036] Defining SV K The total available computing power is Q K ; To focus on the utility of offloading tasks, the execution time of local tasks in the service vehicle SV can be expressed as To express, where β k is a value between 0 and 1, indicating the proportion of computing resources allocated to local tasks; the computing resource allocation of local tasks is determined by:

[0037]

[0038] It follows that:

[0039]

[0040] From the above, we can know that SV K The total utility of completing the local task in a given time is:

[0041]

[0042] It can be seen that U k (β k ) is a monotonic increment function; when β k =0, indicating SV K Allocate minimal computing resources to prioritize its local tasks; when , it means that the service vehicle mainly allocates a large part of its resources to tasks near it; if TV pays λ for task m m C m , where λ m represents the price per computing cycle, then the computing power allocated to task m is determined by the following formula:

[0043]

[0044] SV K The result of executing task m is:

[0045]

[0046] At the current state S(t) and when executing action A(t), the reward value of the task is determined as:

[0047]

[0048] Furthermore, the output y is obtained in step 2-6 i Specifically:

[0049] First, the input sample X is transformed into the feature space by the following mapping:

[0050]

[0051] in and are random parameters and biases with appropriate dimensions; the transformed features are denoted as E 1 , E 2 …E n , and combined into feature set E n =E 1 , E 2 ...E n ; Then, based on this feature set and randomly generated parameters and biases and Generate enhanced nodes. Enhanced nodes can be obtained in the following ways:

[0052]

[0053] Similar to the process described above, randomly generate the matrix and And use the nonlinear function φ(.) to obtain the enhanced node L j ; These nodes are combined and U s = {U 1 , U 2 ...U s}; After the above conditions are met, the output layer Y is formed by integrating all the features and enhancement nodes obtained from the previous steps, which can be expressed as follows:

[0054]

[0055] in The connection parameters representing the entire structure are approximated by using ridge regression [E n |Ls is efficiently calculated using the pseudoinverse of []; the specific calculation process is as follows:

[0056]

[0057] Furthermore, the method for the type-2 fuzzy system to select vehicles in step three is as follows:

[0058] Step 3-1: To calculate the performance value of each service vehicle SV, five metrics are considered: distance, relative speed, link reliability, the computing resources available to the task vehicle, and the dynamic pricing required by the service vehicle SV; these metrics are transformed into fuzzy values and rules through predefined membership functions; the descriptions of these metrics are outlined as follows:

[0059] (1) Distance: There is a certain distance between the service vehicle SV and the TV; as the distance decreases, the transmission delay between the two vehicles shortens, resulting in a more stable network connection between them; therefore, the closer the service vehicle SV is to the TV, the greater the likelihood of being selected; the distance between the two vehicles is determined by their respective coordinates (x i , y i ) and (x j , y j ); this distance metric can be expressed in the following way:

[0060]

[0061] In this equation, Distance(Vh i , Vh j ) represents the distance between the two vehicles, and the coordinates of Vh i and Vh j are (x i , y i ) and (x j , y j );

[0062] (2) Link reliability: It refers to the possibility that the connection link between vehicles Vh i and Vh j remains available within a certain period of time; the link reliability is denoted by l(r) and can be expressed by the following formula:

[0063] l(r) = Pr{riscontinuouslyavailableuntilt + T l |existsatt}

[0064] where 丨 represents the connected link, and T lrepresents the expected time interval of continuous existence of the link at time t; in order to calculate the link reliability, the speed of the vehicle needs to be considered; the probability density function pdf of the vehicle speed v can be expressed as:

[0065]

[0066] Where φ represents the average speed, σ 2 represents the speed variance; assuming that the communication range of the vehicle is L, 2L represents the maximum communication range between two vehicles; therefore, the link duration T needs to satisfy the following conditions:

[0067] 2L ≥ ΔV i，j T,

[0068] Under this condition, the probability density function PDF of T is derived as:

[0069]

[0070] Now, suppose T l represents the duration that the connection link r between vehicles remains active, which can be expressed as:

[0071] T l =L-Distance(Vh i , Vh j ) / Δv i，j ,

[0072] Where Distance(Vh i , Vh j ) is the distance between vehicles Vh i and Vh j ; By integrating f(T) from t to t+T in the above equation l , it follows that the communication link r will last for a period of time T at a given time t r Therefore, the link reliability model can be expressed as:

[0073]

[0074] By solving the integral in the above equation using the error function erf, we obtain:

[0075]

[0076] (3) Relative speed: expressed as RS i，j , indicating Vh i Relative to Vh j Speed; Mission Vehicle TV iCalculate the relative speed of each service vehicle SV relative to itself when receiving the unloading request response of the nearby service vehicle SV; the relative speed can be obtained by the following formula:

[0077] RS i，j =|v i -v j |,

[0078] where v i Indicates Vh i The speed, v j Indicates Vh j speed; a smaller relative speed value means that the vehicle Vh i and Vh j The speed variation between the two is minimal; therefore, this situation is conducive to a stable network connection and facilitates smooth task transmission;

[0079] (4) Available computing resources: The computing power of a vehicle node is called available computing resources, including RAM, CPU, storage, and other hardware elements. These indicators are used to estimate the inherent computing power of the vehicle. In order to better calculate the available computing resources of the service vehicle SVs, a utility function is introduced. If the task vehicle pays λ for task m, m C m , where λ m represents the price per computing cycle, then the computing power allocated to task m is determined by the following formula:

[0080]

[0081] Service car Vh j The available computing resources at time t determine the computational delay of task $m$ and whether task m can be completed within the specified delay constraints;

[0082] (5) Dynamic pricing: Based on the previous indicators, the service vehicle Vh is obtained. j The utility when executing task m is as follows:

[0083]

[0084] Step 3-2: Fuzzy rule evaluation, based on the definition form of If-Then logic operation, represents the fuzzy meaning between conditions and conclusions; before generating the output, these fuzzy rules are evaluated to obtain the desired results.

[0085] The technical solution adopted by the present invention has the following beneficial effects:

[0086] In the present invention, basic data is obtained through the VISSIM simulator to reduce the data parameter basis, which is used as the input of the wide reinforcement learning environment to provide a basis for subsequent decision-making; wide reinforcement learning based on dynamic pricing is used for decision-making to determine the number of vehicles that can be unloaded, thereby effectively coping with the uncertainty in the unloading process; in the task division stage, the type-II fuzzy system is used to find nearby vehicles suitable for unloading, and the tasks are cleverly assigned to suitable vehicles, which improves the efficiency and accuracy of task unloading; by reusing the wide reinforcement learning based on dynamic pricing for reward solution, better rewards are achieved to further improve resource utilization. This invention aims to solve the problem of task unloading in autonomous driving vehicle networking and achieve the requirements of low latency, instant feedback and low energy consumption in autonomous driving vehicle networking scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] Figure 1 It is the experimental result figure of the present invention;

[0088] Figure 2 A scene diagram of the VISSIM simulator used in the present invention;

[0089] Figure 3 It is the overall flow chart of the present invention;

[0090] Figure 4 This is a framework diagram of the width reinforcement learning model of the present invention;

[0091] Figure 5 is a structural diagram of the type-II fuzzy system of the present invention;

[0092] Figure 6 It is a structural diagram of the dynamic pricing mechanism of the present invention. DETAILED DESCRIPTION

[0093] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0094] The present invention is a method for unloading tasks of autonomous driving vehicle networks based on wide reinforcement learning. First, data is collected and preprocessed, and then distributed task unloading is performed through wide reinforcement learning and type-two fuzzy system. The whole process is as follows: Figure 3 shown.

[0095] A method for unloading tasks in an autonomous driving vehicle network based on wide reinforcement learning, the main steps are as follows:

[0096] Step 1: The data obtained through the VISSIM simulator, including the speed and location information of the vehicle, provides the necessary environmental foundation for the wide reinforcement learning based on dynamic pricing, enabling the wide reinforcement learning model to learn and optimize the decision-making process in the simulated traffic environment;

[0097] Draw in VISSIM according to the real scene map. The steps to obtain data are as follows: Figure 2 As shown, Figure 2 The process of acquiring data on the VISSIM platform is shown. First, the simulation parameters are set to configure the operating conditions of the scenario. Then, by executing the simulation scenario, the required speed and position information is finally obtained.

[0098] Step 2: For the distributed task offloading problem in autonomous vehicle networks, the use of dynamic pricing mechanisms and wide reinforcement learning can effectively incentivize vehicles to participate in task offloading at the right time and place, thereby optimizing resource utilization, reducing task delays, and improving the efficiency and stability of the overall system. Therefore, wide reinforcement learning based on dynamic pricing is used for decision-making to determine the number of vehicles that can be offloaded, thereby effectively addressing the decision-making challenges of distributed task offloading scenarios;

[0099] like Figure 6 As shown, the dynamic pricing mechanism affects the reward function of the wide reinforcement learning model by adjusting the cost of task offloading, thereby optimizing resource utilization.

[0100] The task vehicle sends the uncompleted unloading task request to the base station. The base station senses the environmental information, that is, the data in step 1 and the data generated in the program, estimates the wireless channel status and the link duration between the task vehicle and the service vehicle, and sends the status to the current BLS network to obtain feedback actions, select the number of vehicles for the unloading task, and then determine the task vehicle and the price that the task vehicle needs to pay to the service vehicle based on the asking price and bid price of the service vehicle and the task vehicle. At the same time, the base station writes the experience gained each time into the experience pool.

[0101] like Figure 4 As shown in the figure, the wide reinforcement learning model is mainly divided into two parts: decision-making and training. In the decision-making part, the environmental data is passed to the online network for decision-making; in the training part, a certain batch of state-action pairs is obtained from the replay memory, and the label data is obtained from the target network, and passed to the online network for training to optimize the network parameters. The process of wide reinforcement learning task offloading based on the dynamic pricing mechanism is as follows:

[0102] Step 2-1: Breadth reinforcement learning consists of two networks, Policy-BLS and Target-BLS. The Policy-BLS network has two main functions, one is to make decisions, and the other is training data; the input for making decisions is st , the output is y, which is positively correlated with the Q value of reinforcement learning. What is ultimately needed is the action decision a t ; The dataset for training the network includes input data and label data. The input data is obtained from the replay memory. t and a t The label data is based on the reward value r obtained from the replay memory. t Compared with the Q value Q provided by Target-BLS T Combined as follows:

[0103]

[0104] Where a′ represents the action used to maximize the Q value output by Policy-BLS. Then, the training data is the basis and prerequisite for training the Policy-BLS network, including the input data and label data required by the network. Policy-BLS is the main neural network in DBRL, which can receive training data, perform network training, and synchronize its parameters α with the Target-BLS network at every K time steps. Target-BLS is the target neural network in DBRL, which receives s t+1 and a′ and calculate the corresponding Q T To support t Calculation.

[0105] To start the algorithm, the agent obtains a state s from the environment t And send it to Policy-BLS, and then Policy-BLS returns the corresponding action. In the action selection process, the ∈ greedy strategy is adopted, that is, a random action is selected with a probability of ∈, and the action with the highest Q value is selected with a probability of 1-∈:

[0106] a t = argmax a Q T (s t , a; α p )

[0107] Step 2-2: After receiving the allocation message from the base station, the service vehicle reserves some computing resources for the task according to the service price. n , after the task vehicle completes the payment and uploads the task data to the service vehicle, the service vehicle performs the task and finally sends the result receipt to the task vehicle.

[0108] Step 2-3: Pricing based on service¶ n C n And some reserved computing resources, the base station calculates the corresponding reward value R n , and finally written into the experience pool.

[0109] In order to measure the effectiveness of vehicle task execution, we introduce a utility function. It is defined as follows:

[0110]

[0111] In the above formula, U m represents the efficiency of completing the task. This equation represents the actual time t required to complete the task m and the maximum acceptable delay in completing the task The logarithmic nature of the equation shows that as the gap between the actual completion time and the maximum allowed delay increases, the decision becomes more effective. Conversely, as this difference decreases, the effectiveness of the decision decreases. The overall utility of the service vehicle SV performing the local task in a given time when performing the computational task is expressed as:

[0112]

[0113] in and C 1 It represents the maximum acceptable delay time for the service vehicle SV to complete the local task, and the number of CPU cycles required to complete the local task. While executing the computing task, the service vehicle SV also needs to process the local task. The minimum allocation frequency of local tasks:

[0114]

[0115] Defining SV K The total available computing power is Q K To focus on the utility of offloading tasks, the execution time of local tasks in the service vehicle SV can be expressed as To express, where β k is a value between 0 and 1, indicating the proportion of computing resources allocated to local tasks. The computing resource allocation of local tasks is determined by:

[0116]

[0117] Specifically, there are the following relationships:

[0118]

[0119] From the above, we can know that SV K The total utility of completing the local task in a given time is:

[0120]

[0121] It can be seen that U k (β k ) is a monotonic increment function. When βk =0, indicating SV K Allocate minimal computing resources to prioritize its local tasks. , it indicates that the service vehicle mainly allocates a large part of its resources to tasks near it. If TV pays λ for task m m C m , where λ m represents the price per computing cycle, then the computing power allocated to task m is determined by the following formula:

[0122]

[0123] SV K The result of executing task m is:

[0124]

[0125] A good average utility should be able to shorten the task completion time and reduce the payment. Therefore, at the current state S(t) and when performing action A(t), the reward value of the task is determined as:

[0126]

[0127] Step 2-4: The experience pool caches the experience learned by the base station at a certain time step (s n , a n , R n ,s n+1 ), which makes it easier for the base station to perform experience replay learning, random sampling of data, and breaking the correlation between samples.

[0128] Step 2-5: The Training Pool is the storage and training center of Policy-BLS. A batch of samples are randomly selected and accumulated based on the experience pool, denoted as {(s i , a i , R i ,s i+1 )} p By sending {s i+1} p , and obtain the estimated {max a′ Q T (s i+1 , a′; θ T )} p To prepare the training samples of the Policy-BLS network, the corresponding Q value is recorded as y i .

[0129] Step 2-6: The Policy-BLS network is used to model the relationship between the input and output of the network to be trained, represented by the action value function QE Represents. With a set of states s i and the corresponding task assignment action α i As input, y i As output. By minimizing Q E and the output value y i The difference between the two determines the network weight. Finally, the computing task offloading action a is determined n , complete the task allocation to the service vehicles.

[0130] First, the input sample X is transformed into the feature space by the following mapping:

[0131]

[0132] in and are random parameters and biases with appropriate dimensions. The transformed features are denoted as E 1 , E 2 , ...E n , and combined into feature set E n =E 1 , E 2 ...E n Then, based on this feature set and randomly generated parameters and biases and Generate enhanced nodes. Enhanced nodes can be obtained in the following ways:

[0133]

[0134] Similar to the process described above, randomly generate the matrix and And use the nonlinear function φ(·) to obtain the enhanced node L j These nodes are combined and used with U s = {U 1 , U 2 ...U s After the above conditions are met, the output layer Y is formed by integrating all the features and enhancement nodes obtained from the previous steps, which can be expressed as follows:

[0135]

[0136] in The connection parameter representing the entire structure can be approximated by using ridge regression [E n |L s ] to efficiently calculate the pseudo-inverse. The specific calculation process is as follows:

[0137]

[0138] Step 2-7: The Target-BLS network is consistent with the current BLS at a certain time step. It is responsible for receiving the input {s i+1} p After that, calculate the optimal action value {max a , Q T (s t+1 , a′; θ T )} p , in preparation for calculating the output value.

[0139] Step 3: In the task division stage, such as Figure 5 As shown in the figure, through the type-II fuzzy system, the distance between vehicles, link reliability, relative speed, available computing resources and dynamic pricing factors are comprehensively considered to evaluate the suitability of each vehicle, and the task is assigned to the appropriate vehicle according to the number of vehicles making the decision, which improves the efficiency and accuracy of task offloading.

[0140] Step 3-1: To calculate the performance value of each service vehicle SV, five metrics are considered: distance, relative speed, link reliability, computing resources available to the task vehicle, and dynamic pricing required by the service vehicle SV. These metrics are converted into fuzzy values ​​and rules through predefined membership functions. The description of these metrics is outlined below;

[0141] (1) Distance: There is a certain distance between the service vehicle SV and the TV. As the distance decreases, the transmission delay between the two vehicles decreases, resulting in a more stable network connection between them. Therefore, the service vehicle SV that is closer to the TV is more likely to be selected. The distance between the two vehicles is determined by their corresponding coordinates (x i ,y i ) and (x j ,y j ) is calculated. This distance metric can be expressed as follows:

[0142]

[0143] In this equation, Distance(Vh i , Vh j ) represents the distance between the two vehicles, Vh i and Vh j The coordinates are (x i ,y i ) and (x j ,y j ).

[0144] (2) Link reliability: refers to vehicle Vh i and Vh jThe possibility that the connection link between them remains available within a certain period of time. Link reliability is represented by l(r), which can be expressed by the following formula:

[0145] l(r)=Pr{riscontinuouslyavailableuntilt+T l |existsatt}

[0146] Where l represents the connected link, T l represents the expected time interval for the link to continue to exist at time t. In order to calculate the link reliability, the speed of the vehicle needs to be considered. The probability density function (pdf) of the vehicle speed v can be expressed as:

[0147]

[0148] Where φ represents the average speed, σ 2 represents the speed variance. If the communication range of the vehicle is assumed to be L, then 2L represents the maximum communication range between two vehicles. Therefore, the link duration T needs to satisfy the following conditions:

[0149] 2L≥Δv i，j T,

[0150] Under this condition, the probability density function (PDF) of T can be derived as:

[0151]

[0152] Now, suppose T l represents the duration that the connection link r between vehicles remains active, which can be expressed as:

[0153] T l =L-Distance(Vh i , Vh j ) / Δv i，j ,

[0154] Where Distance(Vh i , Vh j ) is the distance between vehicles Vh i and Vh j By integrating f(T) from t to t+T in the above equation l , it can be concluded that the communication link r will last for a period of time T at a given time t r Therefore, the link reliability model can be expressed as:

[0155]

[0156] By using the error function (erf), the integral in the above equation can be solved. This provides the required link reliability model. Then:

[0157]

[0158] (3) Relative speed: expressed as RS i，j , indicating Vh i Relative to Vh j Speed. Mission Vehicle TV i Calculate the relative speed of each service vehicle SV relative to itself when receiving the unloading request response of the nearby service vehicle SV. The relative speed can be obtained by the following formula:

[0159] RS i，j =|v i -v j |,

[0160] where v i Indicates Vh i The speed, v j Indicates Vh j The smaller the relative speed value, the greater the vehicle Vh i and Vh j The speed variation between the two is minimal. Therefore, this situation is conducive to a stable network connection and facilitates smooth task transmission.

[0161] (4) Available computing resources: The computing power of a vehicle node is called available computing resources, including RAM, CPU,

[0162] storage and other hardware elements. These indicators are used to estimate the inherent computing power of the vehicle. In order to better calculate the available computing resources of the service vehicle SVs, a utility function is introduced. The whole process is shown in claim 6. If the task vehicle pays λ for task m m c m , where λ m represents the price per computing cycle, then the computing power allocated to task m is determined by the following formula:

[0163]

[0164] Service car Vh j The available computing resources at time t determine the computational delay of task m and whether task m can be completed within the specified delay constraints. More available computing resources usually means lower task processing delay, thereby improving the efficiency of task execution.

[0165] (5) Dynamic pricing: reflects the cost or price charged by the service vehicle for providing computing resources. This is an important factor to consider when making offloading decisions because it directly affects the overall cost and benefit of task offloading. Based on the previous indicators, it can be concluded that the service vehicle Vh j The utility when executing task m is as follows:

[0166]

[0167] Step 3-2: Fuzzy rules are a form of definition based on If-Then logic operations to express the fuzzy meaning between conditions and conclusions. It is crucial to evaluate these fuzzy rules before generating output to obtain the desired results. Careful design of fuzzy rules is essential. The following table is a set of fuzzy rule sets that define a set of If-Then rules. The formula of the If-Then rule is as follows:

[0168] Table 1. If-Then rule table

[0169]

[0170] Step 4: Re-apply dynamic pricing-based wide reinforcement learning to solve the reward problem and achieve better rewards to further improve resource utilization.

[0171] Experimental Results

[0172] The present invention conducts the following two comparative experiments: (1) comparison with and without type-two fuzzy system, and (2) comparison with binary offloading using dynamic pricing-based wide reinforcement learning. The comparison indicators are total offloading delay, task completion rate and reward value. Figure 1 As shown in (A), (B), and (C), the present invention introduces dynamic pricing, wide reinforcement learning, and type-two fuzzy systems to reduce the task unloading delay in the field of autonomous driving vehicle networking, improve the task completion rate, and greatly improve the reward page. Compared with other methods, the technical solution adopted by the present invention has the best effect. The experimental results show that the method proposed by the present invention in the task unloading process of autonomous driving vehicle networking has a good effect.

[0173] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the schemes disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed in this disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the claims.

Claims

1. A method for unloading tasks in an autonomous driving vehicle network based on wide reinforcement learning, characterized in that: include: Step 1: Obtain vehicle data set through VISSIM simulator. Vehicle data includes vehicle speed and location information. Step 2: Propose a width reinforcement learning model; A width reinforcement learning model is established based on a dynamic pricing mechanism, and vehicle data and base station data are input to train the width reinforcement learning model. The dynamic pricing mechanism and the width reinforcement learning model are used to encourage vehicles to participate in task unloading at the appropriate time and place to determine the number of vehicles that can be unloaded, effectively coping with distributed task unloading scenarios. Step 3: In the task division stage, the type-II fuzzy system comprehensively considers the distance, link reliability, relative speed, available computing resources and dynamic pricing factors of the vehicles to evaluate the suitability of each vehicle, and assigns the tasks to the appropriate vehicles according to the number of vehicles in the decision-making process, thus realizing the task offloading of autonomous driving vehicle networking; Step 4: Use a wide reinforcement learning model based on a dynamic pricing mechanism to solve the reward problem and realize the reward allocation for the unloading task of the autonomous driving vehicle network.

2. The method for unloading tasks of autonomous driving vehicle networking based on wide reinforcement learning according to claim 1 is characterized in that: The steps to obtain the vehicle dataset in step 1 are: Step 1-1: First, obtain a map of the real scene of the mission vehicle, import the map into the VISSIM simulator, and draw a detailed map of the traffic network, including the width of the road, the number of lanes, traffic signs and signal light information; at the same time, design specific traffic flow, speed distribution and traffic rules parameters to obtain the most realistic data; Step 1-2: After the design is completed, run the entire scenario and output the speed and direction information required by the mission vehicle; Step 1-3: Process the acquired target vehicle data and convert the task vehicle data format as the input of the width reinforcement learning model.

3. The method for unloading tasks of autonomous driving vehicle networking based on wide reinforcement learning according to claim 1, characterized in that: The width reinforcement learning model based on the dynamic pricing mechanism in step 2 is composed of two networks, namely Policy-BLS and Target-BLS; wherein, training distributed tasks through the width reinforcement learning model includes: Step 2-1: The task vehicle sends a request for an uncompleted unloading task to the base station. The base station senses the environmental information, i.e., the vehicle data and base station data in step 1, estimates the wireless channel status and the link duration between the task vehicle and the service vehicle, and sends the status to the Policy-BLS network to obtain feedback action a. n , select the number of vehicles for the unloading task, and then determine the task vehicle and the price that the task vehicle needs to pay to the service vehicle based on the asking price and bid of the service vehicle and the task vehicle; at the same time, the base station writes the experience gained each time into the experience pool; Step 2-2: After receiving the allocation message from the base station, the service vehicle reserves some computing resources for the task according to the service price. n , after the task vehicle completes the payment and uploads the task data to the service vehicle, the service vehicle performs the task and finally sends the result receipt to the task vehicle; Step 2-3: Pricing based on service¶ n C n And some reserved computing resources, the base station calculates the corresponding reward value R n , and finally written into the experience pool; Step 2-4: The experience pool caches the experience learned by the base station at a certain time step {s n , a n , R n ,s n+1 ), which facilitates the base station to perform experience playback learning and random sampling of data and break the correlation between samples; Step 2-5: The Training Pool is the storage and training center of the Policy-BLS network. A batch of samples are randomly selected and accumulated based on the experience pool, denoted as {(s i , a i , R i ,s i+1 )} p ; By sending {s i+1 } p , and obtain the estimated {max a′ Q T (s i+1， a′;θ T )} p To prepare the training samples of the Policy-BLS network, the corresponding Q value is recorded as y i ; Step 2-6: The Policy-BLS network is used to model the relationship between the input and output of the network to be trained, represented by the action value function Q E Represents; as a set of states s i and the corresponding task assignment action a i As input, y i As output; by minimizing Q E and the output value y i The difference between the two determines the network weight; finally, the computing task offloading action a is determined n , complete the task allocation for service vehicles; Step 2-7: The Target-BLS network is consistent with the Policy-BLS network at a certain time step. It is responsible for receiving the input {s i+1 } p After that, calculate the optimal action value {max a , Q T (s i+1 , a′; θ T )} p , in preparation for calculating the output value.

4. The method for unloading tasks of autonomous driving vehicle networking based on wide reinforcement learning according to claim 3 is characterized in that: The feedback action a is obtained in step 2-1 n Specifically: The Policy-BLS network has two main functions: one is to make decisions, and the other is to train data. The input for making decisions is s t , the output is y, the output y is positively correlated with the Q value of reinforcement learning, and what is ultimately needed is the action decision a t ; The dataset for training the network includes input data and label data; the input data is obtained from the replay memory t and a t The label data is based on the reward value r obtained from the replay memory. t Compared with the Q value Q provided by Target-BLS T Combined as follows: Where a′ represents the action used to maximize the Q value output by Policy-BLS; then, the training data is the basis and prerequisite for training the Policy-BLS network, including the input data and label data required by the network; Policy-BLS is the main neural network in DBRL, which can receive training data, perform network training, and synchronize its parameters α with the Target-BLS network at every K time steps; Target-BLS is the target neural network in DBRL, which receives s t+1 and a′ and calculate the corresponding Q T To support t Calculation of To start the algorithm, the agent obtains a state s from the environment t And send it to Policy-BLS, and then Policy-BLS returns the corresponding action; in the action selection process, the ∈ greedy strategy is adopted, that is, a random action is selected with a probability of ∈, and the action with the highest Q value is selected with a probability of 1-∈: a t =argmax a QT(s t ,a;α p )。 5. The method for unloading tasks of autonomous driving vehicle networking based on wide reinforcement learning according to claim 3 is characterized in that: The reward R is calculated in steps 2-2 and 2-3 n Specifically: In order to measure the effectiveness of vehicle mission execution, a utility function is introduced: In the above formula, U m Represents the efficiency of completing a task: This equation represents the actual time t required to complete a task m and the maximum acceptable delay in completing the task The logarithmic nature of the equation shows that as the gap between the actual completion time and the maximum allowed delay increases, the decision becomes more effective; conversely, as this difference decreases, the effectiveness of the decision decreases; the overall utility of the service vehicle SV performing the local task in a given time when performing the computational task is expressed as: in and C l It represents the maximum acceptable delay time for the service vehicle SV to complete the local task, and the number of CPU cycles required to complete the local task; while executing the computing task, the service vehicle SV also needs to process the local task; the minimum allocation frequency of the local task: Defining SV K The total available computing power is Q K ; To focus on the utility of offloading tasks, the execution time of local tasks in the service vehicle SV can be expressed as To express, where β k is a value between 0 and 1, indicating the proportion of computing resources allocated to local tasks; the computing resource allocation of local tasks is determined by: It follows that: From the above, we can know that SV K The total utility of completing the local task in a given time is: It can be seen that U k (β k ) is a monotonic increment function; when β k =0, indicating SV K Allocate minimal computing resources to prioritize its local tasks; when , it means that the service vehicle mainly allocates a large part of its resources to tasks near it; if TV pays λ for task m m C m , where λ m represents the price per computing cycle, then the computing power allocated to task m is determined by the following formula: SV K The result of executing task m is: At the current state S(t) and when executing action A(t), the reward value of the task is determined as:

6. The method for unloading tasks of autonomous driving vehicle networking based on wide reinforcement learning according to claim 3 is characterized in that: The output y is obtained in step 2-6 i Specifically: First, the input sample X is transformed into the feature space by the following mapping: in and are random parameters and biases with appropriate dimensions; the transformed features are denoted as E1, E2, ... n , and combined into feature set E n =E1, E2…E n ; Then, based on this feature set and randomly generated parameters and biases and Generate enhanced nodes. Enhanced nodes can be obtained in the following ways: Similar to the process described above, randomly generate the matrix and And use the nonlinear function φ(·) to obtain the enhanced node L j ; These nodes are combined and U s = {U1, U2…U s }; After the above conditions are met, the output layer Y is formed by integrating all the features and enhancement nodes obtained from the previous steps, which can be expressed as follows: in The connection parameters representing the entire structure are approximated by using ridge regression [E n |L s ] to efficiently calculate the pseudo-inverse; the specific calculation process is as follows:

7. The method for unloading tasks of autonomous driving vehicle networking based on wide reinforcement learning according to claim 1, characterized in that: The method of selecting vehicles by the type-2 fuzzy system in step 3 is as follows: Step 3-1: To calculate the performance value of each service vehicle SV, five metrics are considered: distance, relative speed, link reliability, computing resources available to the task vehicle, and dynamic pricing required by the service vehicle SV; these metrics are converted into fuzzy values ​​and rules through predefined membership functions; the description of these metrics is summarized as follows: (1) Distance: There is a certain distance between the service vehicle SV and TV. As the distance decreases, the transmission delay between the two vehicles decreases, resulting in a more stable network connection between them. Therefore, the closer the service vehicle SV is to the TV, the more likely it is to be selected. The distance between the two vehicles is determined by their corresponding coordinates (x i ,y i ) and (x j ,y j ) is calculated; this distance metric can be expressed as follows: In this equation, Distance(Vh i , Vh j ) represents the distance between the two vehicles, Vh i and Vh j The coordinates are (x i ,y i ) and (x j ,y j ); (2) Link reliability: refers to vehicle Vh i and Vh j The possibility that the connection link between them remains available within a certain period of time; link reliability is represented by l(r), which can be expressed by the following formula: l(r)=Pr{riscontinuouslyavailableuntilt+T l |existsatt} Where | represents a connected link, T t represents the expected time interval of continuous existence of the link at time t; in order to calculate the link reliability, the speed of the vehicle needs to be considered; the probability density function pdf of the vehicle speed v can be expressed as: Where φ represents the average speed, σ 2 represents the speed variance; assuming that the communication range of the vehicle is L, 2L represents the maximum communication range between two vehicles; therefore, the link duration T needs to satisfy the following conditions: 2L≥Δv i，j T, Under this condition, the probability density function PDF of T is derived as: Now, suppose T i represents the duration that the connection link r between vehicles remains active, which can be expressed as: T l =L-Distance(Vh i ,Vh j ) / Δv i，j , Where Distance(Vh i , Vh j ) is the distance between vehicles Vh i and Vh j ; By integrating f(T) from t to t+T in the above equation l , it follows that the communication link r will last for a period of time T at a given time t r Therefore, the link reliability model can be expressed as: By solving the integral in the above equation using the error function erf, we obtain: (3) Relative speed: expressed as RS i，j , indicating Vh i Relative to Vh j Speed; Mission Vehicle TV i Calculating the relative speed of each service vehicle SV relative to itself when receiving the unloading request response of the nearby service vehicle SV; The relative speed can be calculated by the following formula: RS i，j =|v i -v j |, where v i Indicates Vh i The speed, v j Indicates Vh j speed; a smaller relative speed value means that the vehicle Vh i and Vh j The speed variation between the two is minimal; therefore, this situation is conducive to a stable network connection and facilitates smooth task transmission; (4) Available computing resources: The computing power of the vehicle node is called the available computing resources, including RAM, CPU, storage, and other hardware element components; These indicators are used to estimate the inherent computing power of vehicles; in order to better calculate the available computing resources of service vehicles SVs, a utility function is introduced: if the task vehicle pays λ for task m m C m , where λ m represents the price per computing cycle, then the computing power allocated to task m is determined by the following formula: Service car Vh j The available computing resources at time t determine the computational delay of task $m$ and whether task m can be completed within the specified delay constraints; (5) Dynamic pricing: Based on the previous indicators, the service vehicle Vh is obtained. j The utility when executing task m is as follows: Step 3-2: Fuzzy rule evaluation, based on the definition form of If-Then logic operation, represents the fuzzy meaning between conditions and conclusions; Before generating the output, these fuzzy rules are evaluated to obtain the desired results.

Images