Transverse and longitudinal decision-making control method for intelligent network-connected vehicles at vehicle-road cooperative signalized intersection

Abstract

The invention relates to the field of traffic control, and provides a vehicle-road cooperation signalized intersection intelligent network connection vehicle transverse and longitudinal decision control method, which comprises the steps that an expert data set is constructed through a genetic algorithm GA, the expert data set comprises all states of at least one track of an intelligent network connection vehicle CAV intelligent agent, and the expert data set is used for training a neural network; according to the expert data set, the neural network is constructed and trained, and the neural network is used for controlling the action of the CAV intelligent agent and the effect of the action in the current state through a reinforcement learning algorithm; a lane changing model used for controlling the CAV intelligent agent is constructed, the state space of the lane changing model is composed of discrete cells of a lane, and the lane changing model is used for determining lane changing information and vehicle speed information of the CAV intelligent agent through state values of the discrete cells. According to the invention, real-time data exchange and strategy coordination between the signal lamp and the intelligent network connection vehicle are realized, so that the traffic flow is optimized and the processing capability of the intersection is improved.

Description

Technical Field

[0001] This application relates to the field of traffic control. Specifically, it relates to an intelligent connected vehicle's lateral and longitudinal decision-making control method for a vehicle-road collaborative signal intersection. Background Art

[0002] Signal intersections (hereinafter referred to as intersections), as key nodes of the urban traffic network, their traffic efficiency directly affects the overall performance of the urban traffic system. However, due to the increasing annual number of motor vehicles, many intersections have problems such as insufficient traffic capacity and frequent traffic conflicts. The low traffic capacity of intersections is usually caused by various factors, including but not limited to unreasonable traffic signal timing, road design defects, unclear traffic signs and markings, and improper traffic flow management. The above problems usually cause traffic oscillations and conflicts at signal intersections, which in turn lead to traffic accidents. This not only affects the traffic efficiency of vehicles at intersections but also threatens the lives and safety of drivers and pedestrians. Therefore, improving the traffic capacity of intersections and reducing the number of traffic conflicts are of great significance for improving the efficiency and safety of urban traffic.

[0003] In view of these challenges, it is particularly urgent to optimize the design and management strategies of signal intersections. With the rapid development of connected and automated vehicles (CAVs), it provides revolutionary improvement possibilities for the urban traffic system, especially for vehicle control and signal light control at intersections. However, at the current stage, the control methods relying on a single signal intersection have obvious deficiencies and often cannot flexibly cope with the traffic flow changes caused by peak hours or special events, resulting in frequent traffic delays and congestion. In addition, in practical applications, when multiple models cooperate to control a single CAV intelligent agent, the control effect is often not satisfactory and it is difficult to achieve the optimal control solution. Summary of the Invention

[0004] This application provides an intelligent connected vehicle's lateral and longitudinal decision-making control method for a vehicle-road collaborative signal intersection, which can achieve the collaboration between signal lights and intelligent connected vehicles, and solve the problems of information silos and low efficiency in controlling CAVs.

[0005] The method includes: constructing an expert dataset through a genetic algorithm GA, where the expert dataset includes all states of at least one trajectory of a connected and autonomous vehicle (CAV) agent, and the expert dataset is used for the training of a neural network; constructing and training the neural network according to the expert dataset, where the neural network is used to control the state and actions of the CAV agent and the effect of the actions in the current state through a reinforcement learning algorithm, and the expert dataset is also used to construct a reward function of the neural network, where the reward function is used to define the effect of the actions in the current state, and the actions include the acceleration information of the CAV agent, and the acceleration information is used for longitudinal vehicle speed control when the CAV agent passes through a road intersection without stopping; constructing a lane-changing model for controlling the CAV agent, where the lane-changing model includes multiple control models and is centralized on a cloud control platform, and the state space of the lane-changing model is composed of discrete cells of lanes, and the lane-changing model is used to determine the lane-changing information and vehicle speed information of the CAV agent through the state values of the discrete cells, and the lane-changing information is used for the lateral control of the CAV agent, and the vehicle speed information is used for the longitudinal control of the CAV agent.

[0006] Through this method, real-time data exchange and policy coordination between traffic lights and connected and autonomous vehicles are achieved, thereby optimizing traffic flow and improving the processing capacity of intersections. This comprehensive control mode not only helps to reduce traffic congestion and accident rates, but also has a profound impact on enhancing road capacity.

[0007] Combined with the first aspect, in a possible implementation manner of the first aspect, constructing an expert dataset through a genetic algorithm GA includes: constructing a fitness function of the GA, where the fitness function is determined by at least one of fuel consumption, waiting time, intersection passing capacity, and smoothness reward; constructing the state of the CAV agent, where the state includes at least one of position, speed, acceleration, and cumulative time; predefining the remaining time of the traffic light to input into the model of the GA to obtain an output vector, and the output vector is used to construct the expert dataset.

[0008] Combined with the first aspect, in a possible implementation manner of the first aspect, constructing and training the neural network includes: constructing a reward function partially based on the expert dataset according to the expert dataset, and the reward function is used for the training of the neural network.

[0009] This method aims to solve the problems that the design of the reward function is complex and it is difficult to determine a reasonable definition. By using the expert data generated based on the genetic algorithm (GA), some reward items are pre-constructed, and the GA expert experience is integrated to reduce the optimization cost of the model and improve the overall performance, and the difficulty of designing the reward function is reduced. The A-C network is pre-trained using the GA expert experience to reduce the computing power overhead. Through this method, the problem that a large amount of unnecessary computing power is consumed during manual parameter tuning in the derivation of the reward function can be solved. By training the neural network to find the reward function and complete the training, the self-parameter tuning of the model is completed, so as to realize the control decision of the CAV vehicle. Moreover, by pre-generating expert data according to GA in advance, some reward items based on GA expert experience are pre-constructed, thereby reducing the model optimization cost and obtaining a performance improvement.

[0010] Combined with the first aspect, in a possible implementation manner of the first aspect, the neural network is the Actor-Critic network of the deep deterministic policy gradient DDPG.

[0011] Combined with the first aspect, in a possible implementation manner of the first aspect, the input of the Actor network includes displacement, speed, acceleration, and remaining red light time, the output of the Actor network includes acceleration, the input of the Critic network includes the state of the CAV agent and the corresponding state action, the output of the Critic network includes the Q value, and the Q value is used to indicate the state of the CAV agent, the corresponding state action, and the network parameter set.

[0012] Combined with the first aspect, in a possible implementation manner of the first aspect, the sampling method of the neural network is importance sampling, and the importance sampling is used for the neural network to obtain experience from the experience pool through the importance parameter index.

[0013] When randomly taking a batch of data from the experience replay pool to calculate the loss function, the model converges slowly during the training process. This application improves the model convergence speed by introducing the importance sampling method, that is, changing from the original random sampling strategy to importance sampling.

[0014] Combined with the first aspect, in a possible implementation manner of the first aspect, constructing a lane-changing model includes: determining the state space of the lane-changing model through the discrete cells of the lane, and the value of the discrete cell is used to determine whether there is a vehicle in the range of the discrete cell; the action space can be set to 1 dimension, and the action output can represent the selected vehicle lane-changing id; it is determined that lane-changing is only performed when the states of the surrounding cells are all 0, and secondary lane-changing is not allowed; through polynomial generation and model predictive control, the trajectory, speed, and heading angle of the CAV agent are determined. Description of the Drawings

[0015] Figure 1It is a schematic diagram of a signal intersection provided by an embodiment of the present application.

[0016] Figure 2 It is a schematic flowchart of a horizontal and longitudinal decision control method for intelligent connected vehicles at a vehicle-road collaborative signal intersection provided by an embodiment of the present application.

[0017] Figure 3 It is a schematic diagram of the architecture of an Actor network provided by an embodiment of the present application.

[0018] Figure 4 It is a schematic diagram of the architecture of a Critic network provided by an embodiment of the present application.

[0019] Figure 5 It is a schematic diagram of the architecture of a DDPG network provided by an embodiment of the present application.

[0020] Figure 6 It is a schematic diagram of a lane change model provided by an embodiment of the present application.

[0021] Figure 7 It is a schematic diagram of the test results of a horizontal and longitudinal decision control method for intelligent connected vehicles at a vehicle-road collaborative signal intersection provided by an embodiment of the present application.

[0022] Figure 8 It is a schematic diagram of the test results of another horizontal and longitudinal decision control method for intelligent connected vehicles at a vehicle-road collaborative signal intersection provided by an embodiment of the present application.

[0023] Figure 9 It is a schematic structural diagram of a communication device according to an embodiment of the present application.

[0024] Figure 10 It is a schematic structural diagram of a communication device according to another embodiment of the present application. Detailed implementation manners

[0025] The embodiments of the present application will be described in detail below. The examples of the embodiments of the present application are shown in the drawings. In the drawings, the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the drawings are exemplary and are only used to explain the present application and should not be construed as a limitation of the present application.

[0026] Unless otherwise defined, technical or scientific terms used herein shall have the ordinary meanings as understood by those of ordinary skill in the technical field to which this application pertains. In the description of this application, it should be understood that the orientation or positional relationships indicated by the terms "center", "longitudinal", "transverse", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. are based on the orientation or positional relationships shown in the drawings, and are only for the convenience of describing this application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus should not be construed as a limitation to this application.

[0027] The reference to "one embodiment" or "some embodiments" etc. described in this specification means that a specific feature, structure, or characteristic described in connection with that embodiment is included in one or more embodiments of this application. Thus, the phrases "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments", etc. that appear in different places in this specification do not necessarily all refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "comprising", "including", "having" and their variants all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0028] For ease of understanding, first, in combination with Figure 1 the scenario of vehicle lane-changing decision-making applied to the intelligent networked vehicle longitudinal and lateral decision-making control method at a vehicle-road collaborative signal intersection shown in the embodiments of this application is introduced. Due to the tight urban road resources, the lane-changing problem usually occurs when a vehicle enters the intersection area, such as the situation where the number of lanes increases from two lanes to three lanes or even four lanes. Figure 1 The schematic diagram of a signal intersection 100 provided by the embodiments of this application is shown. As Figure 1 shown, after the controlled vehicle 110 passes through the intersection, in order to avoid the following vehicle 120, it needs to make a lane-changing decision; after the controlled vehicle 130 enters the intersection area, the situation where the number of lanes increases from two lanes to three lanes occurs, and then in order to avoid other following vehicles, the controlled vehicle 130 needs to make a lane-changing decision. Similarly, after the controlled vehicles 140 and 150 enter the intersection area, the situation where the number of lanes increases from two lanes to three lanes occurs, and then they need to make lane-changing decisions.

[0029] As mentioned in the background art above, the rapid development of CAVs provides revolutionary possibilities for improving urban transportation systems, especially vehicle control and signal control at intersections. On the one hand, CAV technology enables vehicles to receive and send traffic information in real time during driving and can promptly respond and judge emergencies. This ability greatly optimizes the traffic efficiency and safety of vehicles at intersections. Through communication between vehicle-to-vehicle and vehicle-to-road infrastructure, intelligent connected vehicles can predict the traffic conditions and signal states ahead and thus make driving strategy adjustments in advance. On the other hand, based on the highly intelligent and networked characteristics of CAVs, a high-precision adaptive signal control strategy can be realized. The signal timing phases are dynamically adjusted according to traffic flow information to adapt to the actual needs of traffic flow, reducing traffic congestion and waiting time. This transformation from passive response to active optimization not only improves the operation efficiency of the traffic system but also enhances road safety. Therefore, the integration and application of CAVs will bring unprecedented improvements and optimizations to vehicle control and signal control at intersections.

[0030] However, at the current stage, relying solely on single signal intersection control has obvious deficiencies. This traditional control method often cannot flexibly cope with traffic flow changes caused by peak hours or special events, resulting in frequent traffic delays and congestion. Similarly, although controlling CAVs alone can theoretically optimize the driving efficiency and safety of vehicles, in actual operation, the optimization potential of CAV vehicles cannot be fully exerted, and problems such as long waiting times and suboptimal route selection may still occur. Moreover, in practical applications, when multiple models cooperate to control a single CAV agent, the control effect often fails to meet expectations and it is difficult to achieve the optimal control solution, which will be introduced in detail below.

[0031] To solve the above technical problems, this application proposes a vehicle-road collaborative signal intersection intelligent connected vehicle lateral and longitudinal decision-making control method 200, where "lateral" includes lane changes of CAV vehicles between different lanes, and "longitudinal" includes the vehicle speed and / or acceleration of CAV vehicles in the same lane. Figure 2 Fig. shows a flowchart of the method 200 for lateral and longitudinal collaborative control of intelligent connected vehicles in a signal intersection environment provided by an embodiment of this application. The method 200 can be applied to the action suggestion for lane change decision-making of the movement of CAV agents, thereby guiding vehicles to change lanes before the signal intersection.

[0032] As Figure 2 shown, the method 200 first constructs expert data using a genetic algorithm through step 210, introduces a reinforcement learning algorithm to control CAVs through step 220, and further controls the upstream area of the signal intersection through a lane change model in step 230.

[0033] For example, for multi-objective function non-linear optimization problems such as vehicle trajectory control, there are the following two solutions. For mathematical formula-based models such as particle swarm optimization, genetic algorithm (GA), and mixed integer optimization algorithm, the running time is too long to successfully plan a trajectory for CAV agents in real situations. Learning algorithms such as reinforcement learning and deep reinforcement learning are extremely sensitive to parameters, have high training costs, and poor reusability, but the model has high reusability, fast trajectory planning speed, and high efficiency. The key to reinforcement learning lies in how to set the reward function. This key usually requires strict mathematical derivation. Deriving the corresponding reward function becomes the key to deep reinforcement learning. However, the decision parameters of the reward function often require manual tuning, consuming a large amount of unnecessary computing power. To solve the above problems, this application uses the GA algorithm to construct an expert data set through expert trajectories, uses the expert data to train the Actor-Critic network of deep deterministic policy gradient (DDPG), trains DDPG to find the reward function at the same time, and completes the training to complete the self-tuning of the model, so as to realize the control decision of CAV vehicles.

[0034] In addition, aiming at the obvious deficiencies of existing single signal intersection control (for example, existing single-vehicle intelligent control mainly relies on perception data for decision-making, cannot obtain key information such as signal light status and duration, and is limited by perception accuracy, which may lead to control decision errors), this method 200 not only optimizes vehicle driving through signal intersection control, but also integrates the information of the intersection signal system, and plans a trajectory for the CAV agent by obtaining signal light data, thus solving the problems of information island and low optimization efficiency of traditional control methods at the present stage.

[0035] Through this method, real-time data exchange and policy coordination between signal lights and intelligent connected vehicles are realized, so as to optimize traffic flow and improve the processing capacity of intersections. This comprehensive control mode not only helps to reduce traffic congestion and accident rates, but also has a profound impact on improving road traffic capacity, and is of great significance for achieving the goal of a transportation power. Through intelligent comprehensive coordinated control, this application can effectively meet the complex needs of future urban traffic development and realize the efficient, safe and sustainable development of the traffic system.

[0036] The following will specifically introduce an embodiment of the action suggestion of method 200 applied to the lane mutation at the upstream of the signal light when the CAV agent makes a lane change decision.

[0037] At Figure 2In step 210 shown, the present application uses the GA algorithm to construct an expert data set through expert trajectories, and this expert data set can be used for the training of the Actor-Critic network of subsequent DDPG. Specifically, step 210 may include steps 211 to 214.

[0038] Step 211: Construct a GA fitness function.

[0039] To improve the traffic capacity of signal intersections, usually, reducing traffic flow attributes such as vehicle waiting time, fuel consumption, and the number of stops at signal intersections is used as the objective function. Exemplarily, to successfully plan the trajectory of a CAV intelligent agent that passes through an intersection without stopping, the present application may use fuel consumption f c , waiting time w t , intersection traffic capacity t u , smoothness reward s r to define the GA fitness function, and this GA fitness function is shown in Equation (1):

[0040]

[0041] where w f , w d , w j , w t are weights, and C f , C t , C j , C d are respectively the fuel consumption cost, time efficiency cost, smoothness penalty, and distance cost, which can be defined by Equations (2)-(5) respectively:

[0042]

[0043] C j (a t , a t-1 ) = (a t - a t-1 ) 2 / 3600 (4)

[0044] C d = -10 if T r < T l and x c > x i else -1 (5)

[0045] In Equations (1)-(5), k f in C ij represents a predefined coefficient, which can be generated by a standard normal distribution, v i is the speed, with the unit of m / s, and a jis the acceleration, with the unit of m / s 2 . Since the fuel consumption of the vehicle is affected by various factors, a normalization factor is introduced to standardize the fuel consumption and reflect the true data of the fuel consumption, denoted by m, calculated as 7, dimensionless; C in the above formula t where x represents the current position of the vehicle, x i represents the initial distance of the vehicle, defined as 200, and the units are all m. C in the above formula j where a t represents the acceleration at time t, a t-1 represents the acceleration at time t-1, and the units are all m / s 2 . C in the above formula d where T r represents the driving time of the CAV vehicle, T l represents the remaining red light time when the CAV vehicle enters the intersection, with the unit of s, x c represents the displacement of the CAV vehicle at time T r .

[0046] Step 212: Construct the state of the CAV agent controlled by GA at each moment.

[0047] The state of the CAV agent controlled by GA at each moment can be composed of four parts, as shown in the following formula (6):

[0048]

[0049] In the formula, the state variable s t represents the state of the vehicle, including the position x t , speed v t , acceleration a t and cumulative time τ ts at each time step t. The state of the next moment can be calculated using the vehicle kinematic model, as shown in the following formula (7):

[0050] s t+1 = f(x t , u t ) (7)

[0051] where u t is the control variable at the previous time step solved using GA, representing the acceleration. At the same time, to reduce conflicts at the signal intersection, the following constraint as shown in formula (8) can be added:

[0052]

[0053] where the value of the constraint can be taken as: v min = 3m / s, v max = 20m / s, a min = -3m / s2 , a max = 3 m / s 2 .

[0054] Step 213: Define the remaining time of the traffic signal.

[0055] This application can pre-define the remaining time of the traffic signal as T l ~ U(15, 35), and input it into the above GA model to obtain the output of each step, as shown in the following formula (9):

[0056]

[0057] where t represents the movement time, in s t is the state vector, a t is the acceleration at the current time step, c t is the current evaluation function value, and s t+1 is the state of the next time step planned by the GA algorithm under the current state.

[0058] Step 214: Use the output to construct an expert dataset.

[0059] Construct the expert dataset D as shown in formula (10) through the output of the aforementioned step 213 e , and the expert dataset can contain all the states in n trajectories. The trajectory is the solution process of the GA expert for the above process. For example, an existing CAV enters the intersection at the red light time of the traffic signal, and the remaining red light time is 25 s. Each state s constructed by step 211 t , the GA algorithm mutates the population and genetically programs the acceleration a at the current moment t , uses formula (7) to obtain the state s at the next moment t+1 , and obtains the current reward value, and finally obtains the output o at the current moment t , adds it to the expert dataset D e , updates the state using formula (7), and then constructs the next output until the CAV passes through the signal intersection without stopping, ending the search, constructing a trajectory, and constructing an expert dataset by continuously changing the initial state of the vehicle and the mutation rate of the GA algorithm for the subsequent pre-training process of the A-C network.

[0060] D e = [D1, D2, D3, D4, D5] (10)

[0061] where D1 is the current state vector s t , D2 is the current action a t , D3 is the combination of the current state s t and action a t , D4 is the current reward value r t, D5 is the state s at the next moment t+1 .

[0062] In Figure 2 In step 220 shown, this application introduces a reinforcement learning algorithm to control the CAV. Specifically, step 220 may include steps 221 to 224.

[0063] Step 221: Define the state space, action space, and preset reward function of the reinforcement learning algorithm.

[0064] This application may define the state space as where τ is the remaining time for the traffic signal to switch to green, x is the distance traveled by the CAV vehicle from entering the field to the current moment, v is the speed of the vehicle at the current moment, and a is the acceleration of the vehicle at the current moment.

[0065] The action space is the action generated by the Actor in the current state space. In this application, it can be defined as the acceleration, as shown in Equation (11):

[0066] a = σ(s) (11)

[0067] In the design of the reward function, to accelerate the convergence performance of the model, this application makes some improvements to the original DDPG model. Since the original DDPG is relatively weak in estimating the reward function, and the Actor-Critic network in it is particularly sensitive to the changes in the state and action of the CAV agent at each moment, the time cost consumed is too large when training and modifying the reward function. This application pre-constructs some reward terms based on GA expert experience according to the expert data generated by GA in advance, so as to reduce the model optimization cost and obtain a performance improvement. Among them, for the action evaluation of the Actor, it will be changed, and the reward function is defined as shown in Equation (12):

[0068] R(s,a;φ) = k * R f (s,a) + (1 - k) * R l (s,a;φ) (12)

[0069] where s represents the current state space, a represents the action executed by the Actor in the current state, R f (s,a) is the preset reward function, k is the weight, and R l (s,a;φ) is the reward function set for other forms of purposes except for the expert experience reward function, and

[0070]

[0071] where

[0072]

[0073] x is the displacement at the current moment, and a t is the acceleration at the current moment, and a t+1 is the driving suggestion for the next moment calculated by the CAV agent according to the current state, and v l is the instantaneous speed magnitude when the CAV vehicle passes through the signal intersection.

[0074] Step 222: Construct the A-C network.

[0075] The Actor network can be defined as a function μ that outputs an action a based on the observed state s. This network consists of four fully connected layers. Figure 3 shows a schematic diagram of the architecture of the Actor network 300. As Figure 3 shown, the Actor network has four layers. The first layer is a fully connected layer with an input dimension of 4, and the input parameters include displacement, speed, acceleration, and remaining red light time. The second layer is also a fully connected layer, which receives 256 features input from the upper layer and outputs with a dimension of 128. The third layer is a fully connected layer, which receives 128 features from the upper layer and outputs with a dimension of 64. The fourth layer is also a fully connected layer, which receives 64 features input from the upper layer and outputs with a dimension of 1, and uses the tanh function and multiplies it by three.

[0076] Similar to the Actor, the Critic network is defined as a function Q that outputs the value of the combination of the state s and the action a, and consists of four fully connected layers. Figure 4 shows a schematic diagram of the architecture of the Critic network 400. As Figure 4 shown, the Critic network has four layers. The first layer is a fully connected layer and the input includes the state and the vector of the corresponding state action. The second layer is also a fully connected layer, which receives 256 features input from the upper layer and outputs with a dimension of 128. The third layer is a fully connected layer, which receives 128 features from the upper layer and outputs with a dimension of 64. The fourth layer is also a fully connected layer, which receives 64 features input from the upper layer and outputs with a dimension of 1, which is the Q value.

[0077] In the embodiments of the present application, the loss function of the Critic network can use the mean square error loss as shown in Equation (15) to train the network to minimize the difference between the predicted value and the target value:

[0078]

[0079] where y is the target value, which can be calculated according to the Bellman equation shown in Equation (16):

[0080] y = r + γQ(s′, μ(s′; θ μ′ )); θ Q′ ) (16)

[0081] Here, r is the reward value, γ is the discount factor, s′ is the next state, and μ(s′; θ μ′ ) is the next action output by the target Actor network in the s′ state, and θ μ′ and θ Q′ are the parameters of the target Actor and target Critic networks respectively.

[0082] During the model optimization process, for the Critic network, gradient descent can be used to minimize the parameters of the target Critic network and the current Critic network, as shown in the following equation (17):

[0083]

[0084] For the Actor network, its goal is to maximize the Q value output by the Critic network, and the objective function is

[0085]

[0086] where J(θ μ ) represents the optimization goal of the Actor, Q(s, μ(s|θ μ )) is the Q value output by the Critic network, representing the value of the action μ(s|θ μ ) output by the Actor in the state s. To facilitate the use of the Adam optimizer, the original objective function is negated and minimized, which is to maximize J(θ μ ), and its gradient calculation formula is as shown in the following equation (19):

[0087]

[0088] where, is the gradient of the Q value of the Critic network with respect to the action, is the gradient of the action output by the Actor with respect to its parameters. In the embodiments of the present application, through the chain rule, the Actor network can calculate the parameter update direction, optimize the objective function, and finally update the network parameters of the Actor, as shown in the following equation:

[0089]

[0090] where α and β are the learning rates.

[0091] Step 223: Importance sampling.

[0092] The original DDPG model randomly samples a batch of data from the experience replay pool to calculate the loss function. This measure makes the model converge slowly during training. To solve this problem, this application introduces importance sampling, that is, changing the original random sampling strategy to importance sampling, specifically as follows:

[0093] After the CAV agent executes a, the following four-dimensional output is generated:

[0094]

[0095] Among them, s t represents the current state, a represents the executed action, and s′ t+1 represents the state transition to s t+1 ′ through the execution of this action, and r represents the reward value generated by this action in this state. Mark the reward value function and perform the maximization process of the reward function, which can be set to R m = 200. After representing this state from o, it is put into the experience replay pool. Define the importance parameter i as follows:

[0096]

[0097] When storing experiences, add an importance parameter index to each experience. When the Actor network samples experiences from the experience pool, it is more likely to sample the experience group with a large importance parameter to optimize the action in this state in the next step.

[0098] The above importance sampling method can accelerate convergence. Without the importance sampling method, the reward function converges after about 200 training rounds; when the importance sampling method is added, the reward function converges after about 50 training rounds.

[0099] The final constructed reinforcement learning DDPG model architecture 500 is as Figure 5 shown. Specifically, Figure 5 the model architecture is an energy-saving driving strategy framework based on the enhanced deep deterministic policy gradient algorithm (E-DDPG). The CAV agent in the model architecture can be a connected autonomous vehicle.

[0100] Referring to Figure 5 the reinforcement learning framework, first, the initial state data of the CAV agent and its corresponding signal light phase and time can be randomly generated. Then, as described in step 210 above, the embodiments of this application use the genetic algorithm GA to calculate the acceleration of the CAV agent to construct an expert data set. Correspondingly, the expert data set includes acceleration and other related data such as position and cumulative time.

[0101] Subsequently, before the CAV agent interacts with the simulation environment, the embodiments of the present application use the constructed expert dataset to perform joint training on the aforementioned Actor network and Critic network, and initialize the experience replay buffer of DDPG. Exemplarily, as Figure 5 shown, Gaussian distribution noise can be used to simulate communication delays in the real world. This noise and the acceleration result output by the Actor network can enhance the exploration ability of the CAV agent in the initial stage of training, thereby improving the policy convergence performance.

[0102] In addition, as a data-driven method, the operation process of E-DDPG needs to include a training stage and an inference stage. In the training stage, the CAV continuously interacts with the environment and stores experiences in an importance sampling manner (specifically, see the introduction in the previous step 220). At the same time, by updating the network parameters of E-DDPG, the policy is optimized to maximize the cumulative reward. In the inference stage, the parameters of E-DDPG can no longer be updated, but based on the trained policy, the optimal acceleration control instruction is provided for the CAV at each time step to achieve the online deployment of the energy-saving driving strategy.

[0103] Step 224: Pre-train the Actor network and the Critic network, and construct an experience pool.

[0104] The present application can use the generated feature data D1 and label data D2 mentioned above to train the Actor network so that the action μ(s|θ μ ) it outputs is close to the optimal action generated by GA.

[0105] Among them, the Actor network uses the mean squared error loss as the loss function, and the formula is:

[0106]

[0107] where N is the number of training samples, is the expert action generated by GA, and a i is the action output by the Actor network.

[0108] Exemplarily, the training process can be as follows: 1. Input the feature data D1 into the Actor network to obtain the output action a i = μ(s i |θ μ ). 2. Calculate the MSE loss L A between the output action and the action generated by GA. 3. Use the Adam optimizer to perform gradient updates on the Actor network parameters θ μ . 4. Repeat the training until the loss converges, and the Actor network can gradually approximate the optimal policy generated by GA.

[0109] In addition, the present application can use the generated feature data D3 and label data D4 to train the Critic network so that the estimated Q-value Q(s,a|θ Q ) is close to the reward value r generated by GA to . The Critic network also uses the mean squared error loss, and the formula is:

[0110]

[0111] where N is the number of training samples, is the reward value generated by GA, and q i is the Q-value Q(s,a|θ Q ) output by the Critic network.

[0112] Exemplarily, the training process can be as follows: 1. Input the feature data D3 into the Critic network to obtain the output Q-value q i =Q(s i ,a i |θ Q ). 2. Calculate the MSE loss L C between the output Q-value and the reward value generated by GA. 3. Use the Adam optimizer to perform gradient update on the Critic network parameters θ Q . 4. Repeat the training until the loss converges, and the Critic network can gradually approximate the reward function generated by GA.

[0113] After that, an experience replay pool is constructed using the data generated by the foregoing steps for the subsequent joint training of the Actor and the Critic. The definition of the experience replay pool is:

[0114] D e =[D1,D2,D4,D5] (25)

[0115] The experience replay pool D e can be used in the subsequent reinforcement learning stage to provide batch data samples to improve the training efficiency and stability.

[0116] At Figure 2In step 230 shown above, the present application controls the upstream area of the signal intersection through a lane-changing model. In practical applications, when multiple models cooperate to control a single intelligent agent, the control effect often fails to meet expectations and it is difficult to achieve the optimal control solution. This problem usually stems from the unstable cooperative relationship between models or the limitation of computing resources, which in turn affects the real-time performance and accuracy of control strategies. With the rapid development of vehicle-road-cloud integration technology, by leveraging the high-performance computing power of the cloud, complex control models can be centrally processed, thus significantly improving the control effect of intelligent agents. Specifically, the supercomputer in the cloud can quickly calculate and optimize the control model, and the calculation results are transmitted in real time through the communication link to the execution component of the intelligent agent to guide its actions. However, considering that the scheme of equipping each intelligent agent with a supercomputer to provide computing resources is not practical, but it is possible to provide computing resources by the supercomputer in the cloud, and the communication delay involved is mainly concentrated in the communication link.

[0117] Therefore, to solve the above problems, in the construction of the lane-changing algorithm, step 230 of the present application proposes a centralized control method based on vehicle-road-cloud integration technology. By uniformly deploying and centralizing multiple control models on the cloud control platform, efficient control of a single intelligent agent is achieved, effectively utilizing the powerful computing power of the cloud, optimizing the resource allocation in the control process, reducing the dependence on the local computing power of each intelligent agent, and at the same time reducing the possible computing bottlenecks and delay problems in the multi-model cooperative control process, thereby ensuring the optimization of the control effect and the high efficiency of the system. Among them, step 230 includes step 231 to step 234.

[0118] Step 231: Construction of lane load in the cloud center algorithm

[0119] Figure 6 Shows a schematic diagram of a lane-changing algorithm provided by the present application. As Figure 6 shown in the lane control area in, in the cloud control center algorithm, the state space is composed of discrete cells of the lane. Exemplarily, in the embodiment of the present application, to ensure the safety of lane changing, the lane-changing area can be set to 400 meters and this area can be divided into 100 uniform cells, and the length of each cell is 4 meters. As Figure 6 shown, within each cell, if there is a vehicle, the value of the cell is set to 1; if there is no vehicle in the cell, its value is 0, and if a vehicle straddles two cells, the cell state in the traveling direction is set to 1. Therefore, the state space of the algorithm is the cumulative sum of each cell in the independent lane, which is n-dimensional, where n represents the number of lanes. This construction method of the state space not only simplifies the vehicle behavior modeling, but also can adapt to the situation of the vast majority of road sections, and has high generality and adaptability.

[0120] Through this discretized state - space representation, this application can accurately reflect the distribution of vehicles on the road while ensuring real - time performance and computational efficiency. The state value of each cell provides important input information for the cloud algorithm, enabling the intelligent agent to make reasonable guidance decisions based on the current lane state. This method has strong scalability, can adapt to intelligent lane - changing tasks under different road environments and traffic conditions, and can perform efficient state evaluation and decision optimization through the cloud computing platform, thereby improving the safety and decision - making accuracy of the autonomous driving system in complex traffic scenarios.

[0121] Step 232: Construction of the cloud algorithm action space

[0122] In the embodiment of this application, the goal of the cloud algorithm is to balance the load of the straight - through lanes. Therefore, the action space can be set to 1 - D. For example, a selection is made every 15 s, and the action output can represent the selection of the vehicle lane - changing id, which is the lane with the largest relative load, that is, two scenarios:

[0123] Scenario 1: Edge lanes such as Figure 6 shown in lanes 1 and 4 in

[0124]

[0125]

[0126] Scenario 2: Intermediate lanes such as Figure 6 shown in lanes 2 and 3 in

[0127]

[0128] where e i represents the total number of vehicles in the lane with lane id i. If Y i is negative, then this lane is discarded and selected from the remaining roads. As shown in Figure 6 , the selected lane id = 1 because the relative load of lane 1 is the largest.

[0129] Step 233: Selection of lane - changing vehicles

[0130] In the embodiment of this application, when a connected vehicle changes lanes, the cloud can first send the states of its surrounding cells to the connected vehicle. And to prevent unsafe lane - changing behaviors of vehicles in the vehicle - accumulation area upstream of the intersection, lane - changing can only be performed when the states of all surrounding cells are 0, and secondary lane - changing is not allowed. Therefore, there can be two scenarios for the selection of lane - changing vehicles in this application:

[0131] Scenario 1: Edge lanes

[0132] Such as Figure 6In lanes 1 and 4, these two lanes are edge lanes. At this time, if the lane with the heaviest load is selected as the edge lane in step 232, the following formula (29) can be used to guide lane changing:

[0133]

[0134] In some embodiments of the present application, a method of random sampling can be used to extract vehicles meeting the lane-changing conditions from lane i, following the probability p in the above formula. For example, Figure 6 for the green vehicle in

[0135] Scenario 2: Central lane

[0136] For example, Figure 6 in lanes 2 and 3, which are central lanes. At this time, if the lane with the heaviest load is selected as the central lane in step 232, the following formulas (30) and (31) are used to guide lane changing:

[0137] Left-turn probability:

[0138]

[0139] Right-turn probability:

[0140]

[0141] In the embodiments of the present application, a method of random sampling can be used to extract vehicles meeting the lane-changing conditions from lane i. The left-turn and right-turn probabilities are as shown in the above formula. If the probability value is less than 0, no lane change will be made to the corresponding lane.

[0142] In random sampling, the lane-changing decision of a vehicle is determined by comparing it with a pre-calculated probability. The specific process is as follows: For each vehicle on lane i, a uniformly distributed random number r is generated between [0, 1], that is:

[0143] r ∼ U(0, 1)

[0144] where U(0, 1) represents a uniformly distributed random number between 0 and 1.

[0145] If the random number r is less than or equal to the left-turn probability p l , the vehicle will choose to turn left, that is, change to the left adjacent lane. If r ≤ p l , the vehicle turns left. If the random number r is greater than the left-turn probability p l but less than or equal to the sum of the left-turn probability and the right-turn probability (p l + p r ), the vehicle will choose to turn right, that is, change to the right adjacent lane. If p l < r ≤ p l + p r, The vehicle turns right. If the random number r is greater than the sum of the probabilities of turning left and right, p l +p r , the vehicle remains in the original lane.

[0146] Step 233: Trajectory planning for connected and automated vehicle lane change

[0147] When a connected and automated vehicle changes lanes, its trajectory planning and tracking can be achieved through the following trajectory generation method. For example, in the embodiments of the present application, the trajectory x(t) and y(t), v x (t) (speed) and heading angle φ(t) can be generated by quintic polynomial planning.

[0148] 1) Trajectory in the X direction

[0149] The reference trajectory of the vehicle is represented by a quintic polynomial. The target trajectory is for the vehicle to reach the target position from the starting point (x0, y0)

[0150] (x T , y T ). The calculation formula (32) for the x(t) position of the reference trajectory is as follows:

[0151]

[0152] where x0 is the starting position of the vehicle (initialized according to the vehicle's traveling direction), v x is the initial speed of the vehicle (the actual longitudinal speed of the vehicle), D is the target longitudinal offset (the longitudinal displacement required to change lanes to the optimal target), t f is the total lane change duration (for example, it can be 4 seconds in the present application), and t is the time.

[0153] 2) Trajectory in the Y direction:

[0154] The calculation formula (33) for the y(t) position of the reference trajectory is as follows:

[0155]

[0156] where y0 is the starting y-coordinate of the vehicle (determined according to the actual position of the CAV), w is a parameter affecting the lateral acceleration (set according to the actual CAV vehicle), t f is the total lane change time.

[0157] 3) Derivatives of the reference trajectory: Speed and acceleration

[0158] For x(t) and y(t), calculate their first derivatives (speed) to obtain the tangent direction of the path, as shown in equations (34) and (35).

[0159]

[0160] Acceleration is the first derivative of velocity, as shown in equations (36) and (37):

[0161]

[0162] The curvature k(t) is a quantity that describes the degree of curvature of the trajectory, and it is calculated from the first and second derivatives of x(t) and y(t), as shown in equation (38):

[0163]

[0164] It is used to calculate the degree of curvature of the reference trajectory.

[0165] Step 234: Model predictive control (MPC) tracking control

[0166] MPC is used to calculate the optimal control input based on the vehicle's state and the reference trajectory to minimize the tracking error.

[0167] First, MPC needs to make predictions using the vehicle's dynamic model. The state-space model of the vehicle can be expressed as equation (39):

[0168]

[0169] Where: is the longitudinal velocity of the vehicle, φ t is the heading angle of the vehicle, is the heading angular velocity of the vehicle, Y t is the longitudinal position of the vehicle. The state update equation of the vehicle can be expressed as equation (40):

[0170] x t+1 = Ax t + Bu t (40)

[0171] Where: x t is the state vector of the vehicle, u t is the control input of the front-wheel steering angle δ, and A and B are the state transition matrices of the vehicle dynamic model, which are set according to different vehicle types. For example, they can be set as shown below:

[0172] The state transition matrix A describes how the state of the system is transferred from the current moment to the next moment, taking into account the dynamic characteristics of the system. Specifically, the matrix A is obtained by discretizing the vehicle's dynamic equation (as shown in equation (41) below):

[0173] A = I + T·A t (41)

[0174] Among them, A t is the system matrix under continuous time, representing the mutual relationship between state variables (considering factors such as speed, heading angle, heading angular velocity, position, etc.); T is the time step, which is 0.1 in this patent; A t is given by the following matrix:

[0175]

[0176] where: m is the mass of the vehicle, is the longitudinal speed of the vehicle, C cf and C cr are the stiffnesses of the front and rear wheels, l f and l r are the distances from the front and rear wheels to the vehicle's center of mass, and I is the moment of inertia of the vehicle.

[0177] The control input matrix B describes how the control input affects the system state. It is a column vector representing the influence of the control input on each state. The control input is usually the steering angle δ f of the vehicle, and its corresponding control matrix B is:

[0178]

[0179] After that, the cost function of MPC consists of two parts: the control input cost and the state error cost. The purpose of the cost function is to minimize the error between the vehicle state and the reference trajectory and minimize the control input as much as possible.

[0180] The cost part of the control input is used to penalize the magnitude of the control input and avoid excessive control input (steering angle); it can be represented by the following formula (42):

[0181]

[0182] where, u t is the control input (front wheel steering angle) at the current moment, and R is the cost matrix of the control input (which can be a 1X1 diagonal matrix of 0.1 in this application because the control variable is only the steering angle).

[0183] Then, the cost part of the state error is used to penalize the error between the vehicle state and the reference trajectory. Its formula (43) is:

[0184] C s =(x t -x ref ) T Q(x t -x ref ) (43)

[0185] where, x tis the vehicle state at the current moment, x ref is the state of the reference trajectory, and Q is the state error cost matrix, which is used to weigh and penalize the state error between the vehicle and the reference trajectory in the optimization objective; in this application, it can be set as follows:

[0186]

[0187] Among them, the 8 in the first row and first column represents the penalty weight of the longitudinal speed error, the 10 in the second row and second column represents the penalty weight of the heading angle error, the 1 in the third row and third column represents the penalty weight of the heading angular velocity error, and the 10 in the fourth row and fourth column represents the penalty weight of the position error.

[0188] In addition, the final state cost part is used to penalize the error between the final state of the vehicle and the final state of the reference trajectory. Its formula (44) is:

[0189] C f =(x N -x ref,N ) T Q(x N -x ref,N ) (44)

[0190] Finally, constraint conditions need to be set, which can include the following constraints:

[0191] 1. Limit of control input (maximum steering angle):

[0192] |u t |≤0.5

[0193] 2. Dynamic constraints of state update:

[0194] x t+1 =Ax t +Bu t

[0195] Finally, MPC calculates the optimal control input by solving the following optimization problem, as shown in Equation (45):

[0196]

[0197] And, by solving the above optimization problem, MPC will output the optimal control input steering angle, and the calculation process of the control input is as shown in the following Equation (46):

[0198] δ o =argmin(C I ) (46)

[0199] Then, the calculated optimal control input δ oApplied to the status update of the vehicle, status variables such as the position, heading angle, and speed of the vehicle are updated.

[0200] Furthermore, the method 200 implements lane change guidance control within 400m upstream of the signal light control as shown in Figure 6 to dynamically adjust the lane usage rate through cloud computing and real-time scheduling, so as to optimize the load balance of the straight lane. And, as shown in Figure 6 the vehicle speed control area, this application can implement vehicle speed guidance control within 200m downstream of the signal light, and accurately adjust the vehicle driving speed through an intelligent speed regulation strategy to improve the traffic efficiency and achieve the goal of passing through the intersection without stopping. This sub-region collaborative control method effectively reduces the complexity of a single control task, improves the success rate of control decisions, and thus enhances the overall operation efficiency and safety of the intelligent transportation system.

[0201] Figure 7 Shows a test result of the method 200 provided by the embodiment of the present application. Figure 7 Shows the longitudinal trajectory of passing through the signal intersection without stopping (single vehicle). As shown in Figure 7 the orange curve corresponds to the experimental result without guiding the vehicle speed. The test shows that in the absence of vehicle speed optimization, the vehicle may run a red light due to improper speed control, affecting traffic safety and traffic efficiency. The blue curve represents the experimental result after adopting the vehicle speed guidance control strategy proposed by the present application. The test vehicle can decelerate in advance within the vehicle speed control area, dynamically adjust the driving speed, and pass through the intersection smoothly after the green light is turned on without having to stop and wait. The experimental results verify the effectiveness of the present application, prove that it can optimize the traffic efficiency of the signal intersection, reduce the risk of red light violations at the same time, and meet the requirements of intelligent transportation optimization.

[0202] In addition, the algorithm test results provided by the present application show that within the set vehicle speed control area (200m) as shown in Figure 6 by precisely planning the vehicle acceleration, its speed is dynamically adjusted, so as to ensure that the vehicle passes through the intersection at the optimal speed without stopping. The horizontal axis represents time, and the vertical axis represents the vehicle position. The test results verify the effectiveness of the present application, prove that this control strategy can meet the patent requirements of intelligent transportation optimization, improve traffic efficiency and reduce signal delay.

[0203] Figure 8 Shows another test result of the method 200 provided by the embodiment of the present application. As shown in Figure 8As shown, the lane change test results indicate that the control algorithm proposed in this application can effectively track the reference trajectory during the lane change process. The red curve represents the preset reference trajectory, and the blue curve represents the steering trajectory actually executed by the vehicle after using MPC for trajectory tracking control. The experimental results show that the blue curve can fit the reference trajectory to a large extent, verifying the effectiveness of the lane change control strategy proposed in this application in terms of accuracy and stability, and ensuring the controllability and safety of the lane change process. In addition, Table 1 shows the effects of different control strategies on the load balance of the straight lane under the conditions of 4 import lanes and a traffic flow of 2400 veh / h.

[0204] Table 1

[0205]

[0206] As shown in Table 1, from the core evaluation indicators such as the number of passing vehicles, the number of collision events (time to collision, TTC) (TTC is the minimum time for two vehicles to collide), and the waiting time, when only cloud control is used but speed control is not used, the number of passing vehicles is 1564 vehicles / h, the number of TTC occurrences is 268 times / h, and the average waiting time is 19.53 h; when cloud control is not used but speed control is used, the number of passing vehicles is 1471 vehicles / h, the number of TTC occurrences is 227 times / h, and the waiting time is 24.11 h. When both cloud control and speed control are used (i.e., using the solution of this application), the number of passing vehicles is 1971 vehicles / h, the number of TTC occurrences is 204 times / h, and the waiting time is the shortest, which is 16.57 h.

[0207] In addition, when no cloud or speed control is used, the number of passing vehicles is the least, only 1346 vehicles / h, the number of TTC occurrences is the most (346 times / h), and the waiting time is the longest (29.96 h), indicating that without a control strategy, the traffic efficiency at the intersection is the lowest and the potential safety hazards are relatively large. Overall, the experimental results show that the lane change and vehicle speed guidance optimization strategy based on cloud control proposed in this application has significant advantages in improving traffic capacity, reducing collision risks, and reducing waiting time, verifying its application value in intelligent traffic signal optimization.

[0208] Figure 9 It is a schematic block diagram of the communication device 900 according to an embodiment of the present application. The processing module 910 is used to implement the method 200 as Figure 2 shown.

[0209] It should be understood that the communication device 900 is only an example. The device according to the embodiment of the present application may further include other modules or units, or include modules similar to the functions of each module in Figure 9 , or not necessarily include all the modules in Figure 9 .

[0210] Figure 10 It is a schematic structural diagram of a communication device 1000 according to an embodiment of the present application. It should be understood that Figure 10 the shown communication device 1000 is only an example. The communication device 1000 in the embodiment of the present application may further include other modules or units, or include modules similar to the functions of each module in Figure 10 it.

[0211] The communication device 1000 may include one or more processors 1010, one or more memories 1020, a receiver 1030, and a transmitter 1040. The receiver 1030 and the transmitter 1040 may be integrated together and are referred to as a transceiver. The memory 1020 is used to store program codes executed by the processor 1010. Among them, the memory 1020 may be integrated in the processor 1010, or the processor 1010 is coupled to one or more memories 1020 for retrieving instructions in the memory 1020.

[0212] It should be understood that the processor in the embodiment of the present application may be a central processing unit (CPU), and this processor may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or this processor may also be any conventional processor, etc.

[0213] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0214] The above embodiments can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that contains one or more sets of available media. The available media can be magnetic media (such as floppy disks, hard disks, magnetic tapes), optical media (such as DVDs), or semiconductor media. The semiconductor media can be a solid-state drive.

[0215] It should be understood that the term "and / or" in this document is merely a description of the association relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this document generally represents an "or" relationship between the associated objects before and after.

[0216] It should be understood that in various embodiments of the present application, the sequence numbers of the above processes do not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0217] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0218] Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0219] In several embodiments provided in the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of the devices or units can be in electrical, mechanical, or other forms.

[0220] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0221] In addition, in each embodiment of the present application, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.

[0222] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs that can store program codes.

[0223] The above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed in the present application can easily think of changes or substitutions, which should all be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claimed rights.

Claims

1. A lateral and longitudinal decision control method for intelligent networked vehicles at a vehicle-road cooperative signal intersection, characterized in that: include: Constructing an expert data set by a genetic algorithm GA, wherein the expert data set includes all states of at least one trajectory of a CAV intelligent body, and the expert data set is used for training a neural network; According to the expert data set, construct and train the neural network, the neural network is used to control the action of the CAV agent and the effect of the action in the current state through a reinforcement learning algorithm, the expert data set is also used to construct a reward function of the neural network, the reward function is used to define the effect of the action in the current state, the action includes acceleration information of the CAV agent, and the acceleration information is used to control the longitudinal speed of the CAV agent when passing through a road intersection without stopping; A lane changing model for controlling the CAV agent is constructed, wherein the lane changing model includes multiple control models and is centralized on a cloud control platform, wherein the state space of the lane changing model is composed of discrete cells of lanes, and the lane changing model is used to determine lane changing information and vehicle speed information of the CAV agent through the state values ​​of the discrete cells, wherein the lane changing information is used for lateral control of the CAV agent, wherein the lateral control includes lane changing of the CAV agent between different lanes, and wherein the vehicle speed information is used for longitudinal control of the CAV agent, wherein the longitudinal control includes the speed change of the CAV agent in the same lane.

2. The method according to claim 1, characterized in that The constructing of the expert data set by the genetic algorithm GA comprises: Constructing a fitness function of the GA, wherein the fitness function is determined by at least one of fuel consumption, waiting time, intersection capacity, and flatness reward; Constructing a state of the CAV agent, the state including position, velocity, acceleration and accumulated time; The remaining time of the signal light is predefined to be input into the model of the GA to obtain an output vector, and the output vector is used to construct the expert dataset.

3. The method according to claim 1, characterized in that The neural network is an Actor-Critic network of deep deterministic policy gradient DDPG.

4. The method according to claim 3, characterized in that The input of the Actor network includes displacement, velocity, acceleration and remaining red light time, the output of the Actor network includes acceleration, the input of the Critic network includes the state of the CAV agent and the corresponding state action, the output of the Critic network includes a Q value, and the Q value is used to indicate the state of the CAV agent, the corresponding state action and the network parameter set.

5. The method according to claim 3 or 4, characterized in that: The sampling method of the neural network is importance sampling, and the importance sampling is used for the neural network to obtain experience from the experience pool through importance parameter indicators.

6. The method according to claim 1, characterized in that The lane-changing model construction comprises: Determine the state space of the lane change model through discrete cells of the lane, wherein the value of the discrete cells is used to determine whether there is a vehicle within the range of the discrete cells; The action space can be set to 1 dimension, and the action output can represent the lane change identification id of the selected vehicle; Determine that a single CAV agent changes lanes only when the states of surrounding cells are all 0, and cannot change lanes twice; The trajectory, velocity and heading angle of the CAV agent are determined through polynomial generation and model predictive control.

Images