Mixed traffic flow multi-vehicle particle flow cooperative control method and related equipment

Abstract

The invention discloses a mixed traffic flow multi-vehicle particle flow cooperative control method and related equipment, and can be applied to the technical field of intelligent traffic and automatic driving cooperative control. According to the invention, after the fixed dimension joint state representation corresponding to the multi-dimensional state information is input into the shared feature extraction backbone network for feature extraction, the fixed dimension joint state representation is respectively input into the preset TD3 network and the preset DQN network for vehicle action decision making, so that multi-vehicle longitudinal and transverse cooperative control can be realized by using the two reinforcement learning networks TD3 and DQN. And the second Critic network is combined with the central collaborative value function, so that the network is not newly added, the overestimation risk is not increased, and the vehicle driving safety is improved. And a multi-target collaborative reward function in the composite action control process is calculated through the average fuel consumption of the motorcade and the traffic light passing rate standardization value and the traffic light parking frequency standardization value corresponding to the intelligent networking automobile, so that the dimension of the reward function is fixed, and the reward function does not need to be redeployed for a real automobile.

Description

Technical Field

[0001] This application relates to the field of intelligent transportation and autonomous driving cooperative control technology, and in particular to a multi-vehicle granular flow cooperative control method and related equipment for mixed traffic flow. Background Technology

[0002] In related technologies, the coexistence of connected and automated vehicles (CAVs) and human-driving vehicles (HDVs) in mixed traffic flow has become the new normal in urban transportation. CAVs possess V2V (vehicle-to-vehicle) and V2I (vehicle-to-infrastructure) communication capabilities, enabling them to acquire high-precision maps, traffic light phases, and neighboring vehicle status in real time, thus supporting proactive cooperative control. HDVs, on the other hand, lack communication and cooperative capabilities, and their driving behavior is random and unpredictable, leading to a significant increase in overall system uncertainty. This heterogeneous mixed traffic flow of "communicable + non-communicable" traffic brings unprecedented complexity to existing traffic control methods.

[0003] In mixed traffic flow environments, signalized intersections remain a bottleneck for energy consumption and efficiency. Frequent stop-and-go traffic, rapid acceleration and deceleration, and narrow green light windows directly lead to increased fuel consumption, decreased comfort, and a surge in emissions. While existing model predictive control (MPC) methods can solve for optimal trajectories, they require real-time processing of non-convex optimization, are sensitive to HDV behavior disturbances, and struggle to adapt to dynamic traffic flow. Rule-based ECO-Driving strategies, on the other hand, often fail in real-world scenarios because they cannot model HDV stochasticity.

[0004] Most existing reinforcement learning (RL) methods focus on single-vehicle trajectory optimization, lacking multi-vehicle granular-level coordination. If each CAV only optimizes its own trajectory, a chain reaction of "the car in front running a light and the car behind braking suddenly" can easily occur, leading to a rebound in overall platoon fuel consumption, a low green light grouping rate, and an increase in the number of stops at red lights. Secondly, multi-vehicle longitudinal-lateral coordinated control involves a mixed action space of continuous acceleration and discrete lane changing. Traditional multi-algorithm superposition leads to parameter explosion, difficulty in training convergence, and complexity in real-vehicle deployment.

[0005] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0006] The main objective of this application is to propose a multi-vehicle granular flow cooperative control method and related equipment for mixed traffic flow, which can achieve multi-vehicle longitudinal and transverse cooperative control without adding new networks or increasing parameter explosion, thereby effectively improving the safety of vehicle driving in mixed traffic flow.

[0007] To achieve the above objectives, one aspect of this application proposes a multi-vehicle granular flow cooperative control method for mixed traffic flow, the method comprising the following steps: Multidimensional state information is collected to form a fixed-dimensional original state vector. The multidimensional state information includes the vehicle's kinematic state, its relative state with the vehicle in front, its relative state with adjacent vehicles in the left and right lanes, the SPaT information of the preset traffic lights, and the tag information of the connected automatic vehicle. The original state vector is subjected to pooling compression to obtain a fixed-dimensional joint state representation; The fixed-dimensional joint state representation is input into the shared feature extraction backbone network for feature extraction to obtain a high-level feature vector. The high-level feature vectors are input into a preset TD3 network and a preset DQN network respectively for vehicle action decision-making to obtain composite actions, which include continuous longitudinal acceleration commands and discrete lateral lane change commands. The preset TD3 network includes a first Critic network, a second Critic network, and an Actor network. The second Critic network also serves as the central collaborative value function, and the target collaborative advantage function corresponding to the central collaborative value function is used to guide the strategy training process of all vehicles to learn to maximize global benefits. The multi-objective cooperative reward function in the composite motion control process is calculated. The multi-objective cooperative reward function is calculated by the average fuel consumption of the fleet at a preset time step and the standardized values ​​of the traffic light pass rate and the number of traffic light stops corresponding to the intelligent connected vehicle. The network parameters of the preset TD3 network and the preset DQN network are updated based on the batch data composed of the fixed-dimensional joint state representation, the composite action, and the multi-objective cooperative reward function. After deploying and testing the preset TD3 network and preset DQN network with updated network parameters, as well as the corresponding pooling compression weights and the weights of the multi-objective collaborative reward function in a real vehicle, driving control of the real vehicle is then performed.

[0008] In some embodiments, the collection of multi-dimensional state information to form a fixed-dimensional original state vector includes: The multi-dimensional status information is collected synchronously through the vehicle CAN bus, vehicle radar, and V2V communication module of the intelligent connected vehicle at a preset time period. The multidimensional state information is used to form the fixed-dimensional original state vector.

[0009] In some embodiments, the pooling compression process performed on the original state vector to obtain a fixed-dimensional joint state representation includes: The mean, maximum value, standard deviation, summation, and latest value of the original state vector are calculated along the feature dimension to obtain the extended statistical feature vector; The remaining green and red times of the traffic lights at multiple intersections ahead of the vehicle path are concatenated to obtain a global SPaT feature vector; The extended statistical feature vector and the global SPaT feature vector are concatenated to obtain the intermediate state vector; The intermediate state vector is linearly transformed and dimensionality reduced by a pooling compression layer to obtain the fixed-dimensional joint state representation, which is independent of the total number of vehicles in the traffic flow.

[0010] In some embodiments, the multi-objective collaborative reward function is as follows: r coop (t)=-w f *F(t)+w f *S(t)-w l *L(t); In the formula, r coop (t) represents the multi-objective collaborative reward value; F(t) represents the average fuel consumption of the fleet at the preset time step t; S(t) represents the standardized value of the traffic light pass rate of the intelligent connected vehicle at the preset time step t; L(t) represents the standardized value of the number of times the intelligent connected vehicle stops at traffic lights at the preset time step t; w f w f and w l All are greater than 0 and are pre-set fixed weight coefficients.

[0011] In some embodiments, the target synergistic advantage function is as follows: A coop (t)=r coop (t)+γ*V coop (V coop (t+1))-V coop (s joint (t)); In the formula, A coop (t) represents the target collaborative advantage value at time step t; V coop (s joint (t) represents the value of the central collaborative value function at time step t; V coop (V coop (t+1)) represents the value of the central collaborative value function corresponding to time step t+1; γ represents the discount factor.

[0012] In some embodiments, updating the network parameters of the preset TD3 network and the preset DQN network based on batch data composed of the fixed-dimensional joint state representation, the composite action, and the multi-objective cooperative reward function includes: The composite action, the Q value corresponding to the composite action at adjacent time steps, and the multi-objective cooperative reward function are combined to form a single-vehicle experience tuple; the fixed-dimensional joint state representation corresponding to adjacent time steps and the multi-objective cooperative reward function are combined to form a joint state transition tuple; Store all the single-vehicle experience tuples and the joint state transition tuples in the global experience replay pool; After environmental interaction based on a preset step size, batch data is obtained from the global experience replay pool to update the network parameters of the preset TD3 network and the preset DQN network.

[0013] In some embodiments, updating the network parameters of the preset TD3 network and the preset DQN network includes: Update the central collaborative value function and the parameters of the first Critic network according to the first loss function and the second loss function; Update the parameters of the Actor network according to the policy gradient function; The parameters of the preset DQN network are updated according to the third loss function.

[0014] To achieve the above objectives, another aspect of this application proposes a multi-vehicle granular flow cooperative control device for mixed traffic flow, the device comprising: The first module is used to collect multi-dimensional state information to form a fixed-dimensional original state vector. The multi-dimensional state information includes the vehicle's kinematic state, the relative state with the vehicle in front, the relative state with adjacent vehicles in the left and right lanes, the SPaT information of the preset traffic lights, and the tag information of the connected automatic vehicle. The second module is used to perform pooling compression on the original state vector to obtain a fixed-dimensional joint state representation; The third module is used to input the fixed-dimensional joint state representation into the shared feature extraction backbone network for feature extraction to obtain a high-level feature vector. The fourth module is used to input the high-level feature vectors into a preset TD3 network and a preset DQN network respectively to make vehicle action decisions and obtain composite actions. The composite actions include continuous longitudinal acceleration commands and discrete lateral lane change commands. The preset TD3 network includes a first Critic network, a second Critic network, and an Actor network. The second Critic network also serves as the central collaborative value function. The target collaborative advantage function corresponding to the central collaborative value function is used to guide the strategy training process of all vehicles to learn to maximize global benefits. The fifth module is used to calculate the multi-objective cooperative reward function in the composite action control process. The multi-objective cooperative reward function is calculated by the average fuel consumption of the fleet at a preset time step and the standardized value of the traffic light pass rate and the standardized value of the number of traffic light stops corresponding to the intelligent connected vehicle. The sixth module is used to update the network parameters of the preset TD3 network and the preset DQN network based on the batch data composed of the fixed-dimensional joint state representation, the composite action, and the multi-objective cooperative reward function; The seventh module is used to perform driving control on the real vehicle after deploying and testing the preset TD3 network and preset DQN network based on the updated network parameters, as well as the corresponding pooled compression weights and the weights of the multi-objective collaborative reward function.

[0015] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0016] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method.

[0017] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer program product, including a computer program that, when executed by a processor, implements the aforementioned method.

[0018] The embodiments of this application include at least the following beneficial effects: This application provides a multi-vehicle granular flow cooperative control method and related equipment for mixed traffic flow. This scheme collects multi-dimensional state information to form a fixed-dimensional original state vector, performs pooling compression on the original state vector to obtain a fixed-dimensional joint state representation, and then inputs the fixed-dimensional joint state representation into a shared feature extraction backbone network to extract features to obtain a high-level feature vector. Then, the high-level feature vector is input into a preset TD3 network and a preset DQN network respectively to make vehicle action decisions to obtain composite actions. Then, by using the two reinforcement learning networks TD3 and DQN, multi-vehicle longitudinal and transverse cooperative control can be realized, effectively improving the safety of vehicle driving in mixed traffic flow. At the same time, in the cooperative control process, the second Critic network is used as the central cooperative value function, so that no new network is added and no overestimation risk is increased, making the training more stable and fuel-efficient. The multi-objective cooperative reward function in the composite action control process is calculated by the average fuel consumption of the fleet and the standardized values ​​of the traffic light pass rate and the number of traffic light stops corresponding to the intelligent connected vehicles. This makes the dimension of the reward function fixed, and the actual vehicle does not need to redeploy the reward function, reducing the deployment workload. Attached Figure Description

[0019] Figure 1 This is a flowchart of the multi-vehicle granular flow cooperative control method for mixed traffic flow provided in the embodiments of this application; Figure 2 This is a schematic diagram illustrating the application control of the multi-vehicle particle flow cooperative control method for mixed traffic flow provided in the embodiments of this application; Figure 3 This is a schematic diagram of the structure of the multi-vehicle particle flow coordinated control device for mixed traffic flow provided in the embodiments of this application; Figure 4 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0021] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0022] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit the scope of this application.

[0024] Before providing a detailed description of the embodiments of this application, some of the nouns and terms used in the embodiments of this application will be explained first. The nouns and terms used in the embodiments of this application shall be interpreted as follows: TD3 (Twin Delayed Deep Deterministic Policy Gradient) in reinforcement learning is a deep learning-based off-policy reinforcement learning algorithm designed to solve the sample efficiency problem in continuous action space.

[0025] Deep Q-Network (DQN) in reinforcement learning is an algorithm that combines deep learning and Q-Learning. It solves decision-making problems in high-dimensional state spaces by approximating the state-action value function (Q(s,a)) through a neural network.

[0026] MAPPO (Multi-Agent Proximal Policy Optimization) is a multi-agent reinforcement learning algorithm based on policy gradients. It extends PPO (Proximal Policy Optimization) to achieve efficient policy learning in multi-agent cooperative and competitive scenarios.

[0027] Centralized MPC (Model Predictive Control) is an advanced control method based on system models. Its core idea is to achieve efficient control of multivariable systems by predicting future states and optimizing current control inputs.

[0028] In related technologies, the coexistence of connected and automated vehicles (CAVs) and human-driving vehicles (HDVs) in mixed traffic flow has become the new normal in urban transportation. CAVs possess V2V (vehicle-to-vehicle) and V2I (vehicle-to-infrastructure) communication capabilities, enabling them to acquire high-precision maps, traffic light phases, and neighboring vehicle status in real time, thus supporting proactive cooperative control. HDVs, on the other hand, lack communication and cooperative capabilities, and their driving behavior is random and unpredictable, leading to a significant increase in overall system uncertainty. This heterogeneous mixed traffic flow of "communicable + non-communicable" traffic brings unprecedented complexity to existing traffic control methods.

[0029] In mixed traffic flow environments, signalized intersections remain a bottleneck for energy consumption and efficiency. Frequent stop-and-go traffic, rapid acceleration and deceleration, and narrow green light windows directly lead to increased fuel consumption, decreased comfort, and a surge in emissions. While existing model predictive control (MPC) methods can solve for optimal trajectories, they require real-time processing of non-convex optimization, are sensitive to HDV behavior disturbances, and struggle to adapt to dynamic traffic flow. Rule-based ECO-Driving strategies, on the other hand, often fail in real-world scenarios because they cannot model HDV stochasticity.

[0030] Most existing reinforcement learning (RL) methods focus on single-vehicle trajectory optimization, lacking multi-vehicle granular-level coordination. If each CAV only optimizes its own trajectory, a chain reaction of "the car in front running a light and the car behind braking suddenly" can easily occur, leading to a rebound in overall platoon fuel consumption, a low green light grouping rate, and an increase in the number of stops at red lights. Secondly, multi-vehicle longitudinal-lateral coordinated control involves a mixed action space of continuous acceleration and discrete lane changing. Traditional multi-algorithm superposition leads to parameter explosion, difficulty in training convergence, and complexity in real-vehicle deployment.

[0031] In view of this, this application provides a multi-vehicle granular flow cooperative control method and related equipment for mixed traffic flow, which can achieve multi-vehicle longitudinal and transverse cooperative control without adding new networks or increasing parameter explosion, thereby effectively improving the safety of vehicle driving in mixed traffic flow.

[0032] The mixed traffic flow multi-vehicle granular flow cooperative control method provided in this application relates to the field of intelligent transportation and autonomous driving cooperative control technology. The mixed traffic flow multi-vehicle granular flow cooperative control method provided in this application can be applied to a terminal, a server, or software running on a terminal or server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the mixed traffic flow multi-vehicle granular flow cooperative control method, but is not limited to the above forms.

[0033] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0034] The embodiments of this application will be described in detail below with reference to the accompanying drawings: Figure 1 This is an optional flowchart of the multi-vehicle granular flow cooperative control method for mixed traffic flow provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S106.

[0035] Step S110: Collect multi-dimensional state information to form a fixed-dimensional original state vector. The multi-dimensional state information includes the vehicle's kinematic state, relative state with the vehicle in front, relative state with adjacent vehicles in the left and right lanes, SPaT information of preset traffic lights, and tag information of connected automatic vehicles. Step S120: Perform pooling compression on the original state vector to obtain a fixed-dimensional joint state representation; Step S130: Input the fixed-dimensional joint state representation into the shared feature extraction backbone network for feature extraction to obtain the high-level feature vector; Step S140: Input the high-level feature vectors into the preset TD3 network and the preset DQN network respectively to make vehicle action decisions and obtain composite actions. The composite actions include continuous longitudinal acceleration commands and discrete lateral lane change commands. The preset TD3 network includes a first Critic network, a second Critic network and an Actor network. The second Critic network also serves as the central collaborative value function. The target collaborative advantage function corresponding to the central collaborative value function is used to guide the strategy training process of all vehicles to learn to maximize global benefits. Step S150: Calculate the multi-objective cooperative reward function in the compound motion control process. The multi-objective cooperative reward function is calculated by the average fuel consumption of the fleet at a preset time step and the standardized value of the traffic light pass rate and the standardized value of the number of traffic light stops for the intelligent connected vehicle. Step S160: Update the network parameters of the preset TD3 network and the preset DQN network based on the batch data composed of fixed-dimensional joint state representation, compound actions and multi-objective cooperative reward function; Step S170: After deploying and testing the preset TD3 network and preset DQN network based on the updated network parameters, as well as the corresponding pooling compression weights and the weights of the multi-objective collaborative reward function on a real vehicle, driving control is performed on the real vehicle.

[0036] It is understood that the hardware environment used in this embodiment is deployed on an onboard electronic control unit (ECU) with a computing power of at least 8 cores CPU and a memory capacity of ≥4GB. The ECU has a built-in hardware module supporting V2V (vehicle-to-vehicle) and V2I (vehicle-to-infrastructure) communication, used to receive real-time traffic light phase and time (SPaT) information and the status of surrounding vehicles. The image corresponding to the method in this embodiment is pre-stored in the onboard storage medium, containing the following pre-trained weight files corresponding to the neural network model: shared feature extraction backbone network weights, TD3 algorithm double-Q network weights, DQN algorithm lateral decision head weights, pooling compression layer weights, and multi-objective collaborative weights. The software environment in this embodiment uses SUMO 3.1.0 traffic simulation software and the PyTorch 2.0 deep learning framework to build the software stack. During initialization, the pre-trained weights are loaded sequentially to construct a complete network model. This network model includes a shared feature extraction backbone network (fully connected layers of 128 and 64 dimensions), a pre-defined TD3 network Actor network and a dual Critic(Q) network, a pre-defined DQN network lateral decision head, a pooling compression layer (compressing the 110-dimensional input to a 50-dimensional fully connected layer), and fixed scalar weights (w) for reward coordination. f ,w s ,w l ,λ).

[0037] It is understood that this embodiment models the cooperative driving problem of intelligent connected vehicles (CAVs) in traffic light-controlled road networks as a partially observable Markov decision process, and uses a cooperative reinforcement learning framework with centralized evaluation and distributed execution to solve it.

[0038] Specifically, the state, action, and reward space of the method in this embodiment during the solution process can be formally defined as follows: (1) Multidimensional state information of a single vehicle: For the i-th vehicle in the traffic flow, at time step t, its original local observation state is denoted as vector o. i (t)∈ 20In this embodiment, the fixed-dimensional original state vector is constructed by synchronously collecting multi-dimensional state information through the vehicle CAN bus, vehicle radar, and the V2V communication module of the intelligent connected vehicle at a preset time interval. This multi-dimensional state information includes the vehicle's kinematic state (3D, including speed, acceleration, and position), its relative state with the vehicle in front (3D, including relative distance, relative speed, and relative acceleration), its relative state with adjacent vehicles in the left and right lanes (8D), the SPaT information of the nearest traffic light (3D, including traffic light status, remaining time, and distance to the stop line), and the tag information of the connected autonomous vehicle (CAV) (3D). The collected information is then combined to form a 20-dimensional fixed-dimensional original state vector. i (t).

[0039] In this embodiment, before training the neural network model, the variance inflation factor (VIF) is diagnosed on the 20-dimensional fixed-dimensional original state vector Oi(t) to ensure that the VIF value of each dimension is less than 1.15, thereby eliminating the influence of multicollinearity on the stability of the model.

[0040] (2) Fixed-dimensional joint state representation.

[0041] In this embodiment, after obtaining the original state vector, multi-statistic pooling is performed on it. The pooling process in this embodiment calculates the mean, maximum, standard deviation, sum, and latest value along the feature dimensions of the original state vector to obtain an extended statistical feature vector. For example, a 100-dimensional extended statistical feature vector is obtained by calculating five statistics—mean, maximum, standard deviation, sum, and last—along the feature dimensions of a 20-dimensional fixed-dimensional original state vector Oi(t).

[0042] Simultaneously, the remaining green / red times of traffic lights at multiple intersections ahead of the vehicle's path are concatenated to obtain a global SPaT feature vector. For example, concatenating the remaining green / red times of traffic lights at up to 5 intersections ahead of the vehicle's path yields a 10-dimensional global SPaT feature vector S. global (t).

[0043] Then, the extended statistical feature vector and the global SPaT feature vector are concatenated to obtain an intermediate state vector. This intermediate state vector is then linearly transformed and dimensionality-reduced using a pooling compression layer to obtain a fixed-dimensional joint state representation. For example, a 100-dimensional extended statistical feature vector is concatenated with a 10-dimensional global SPaT feature vector to form a 110-dimensional intermediate state vector. This is then further compressed using a pooling compression layer Φ. θ(A fixed fully connected layer, FC110→50) performs linear transformation and dimensionality reduction on the intermediate state vector, finally outputting a 50-dimensional joint state representation s that is independent of the total number of vehicles in the traffic flow. joint (t), denoted as s joint (t) ∈ 50 The process is formally defined as: s joint (t) = Φ θ (Pool(o1(t), o2(t), ..., o N (t)), S global (t)), where Φ θ For the pooling compression layer with parameter θ, Pool(·) is a 5-statistic pooling operation, S global (t) represents global SPAT information.

[0044] (3) Compound actions: the compound actions a of each vehicle i (t) is a two-dimensional composite action, consisting of continuous longitudinal acceleration commands and discrete lateral lane-changing commands, i.e., a i (t) = (ai acc(t), ai lane(t)). Where, ai acc(t) ∈ [-4, 4] is a continuous value; ai lane(t)∈{-1,0,1} is a discrete value, representing changing lanes to the left, keeping the lane, and changing lanes to the right, respectively.

[0045] (4) Multi-objective collaborative reward function: In this embodiment, a scalarized multi-objective collaborative reward function r is defined. coop (t), the multi-objective collaborative reward function is a standardized weighted sum of three key performance indicators, and the calculation formula is as follows: r coop (t)=-w f *F(t)+w f *S(t)-w l *L(t); In the formula, r coop (t) represents the multi-objective collaborative reward value; F(t) represents the average fuel consumption of the fleet at the preset time step t; S(t) represents the standardized value of the traffic light pass rate of the intelligent connected vehicle at the preset time step t; L(t) represents the standardized value of the number of times the intelligent connected vehicle stops at traffic lights at the preset time step t; w f w f and w l All are greater than 0 and are pre-set fixed weight coefficients used to balance the priorities of different optimization objectives.

[0046] Specifically, in this embodiment, after each decision step, the multi-objective collaborative reward value r is calculated in real time using a multi-objective collaborative reward function. coop (t). The calculation process in this embodiment has been optimized, taking less than 5 milliseconds per step, and its computational complexity is independent of the total number of vehicles N in the current scenario. The r calculated in this embodiment... coop (t) is used for Critic network updates for both TD3 and DQN.

[0047] It is understood that the neural network model in this embodiment includes a shared feature extraction backbone network, a preset TD3 network, and a preset DQN network. The function of the shared feature extraction backbone network is f. φ (·), with parameter φ. This shared feature extraction backbone network receives the fixed-dimensional original state vector O of a single vehicle's local area. i (t), outputting the high-level feature vector h i (t), i.e., h i (t)=f φ (O i (t)). Among them, the high-level feature vector h i (t) serves as the common input to the decision heads of the subsequent preset TD3 network and preset DQN network.

[0048] The default TD3 network includes the first Critic network Q. ψ1 (h,a acc ), the second Critic network Q ψ2 (h,a acc ) and Actor Network π θ (h). The first and second Critic networks form a dual-Q network, with parameters ψ1 and ψ2 respectively. The training objective for each Critic network is to minimize the temporal difference error: L(ψ k )= {(h,a acc ,r coop ,h')~D}[(Q ψk (h,a acc )-y) 2 ],fork=1,2; In the formula, the target value y is calculated by the target network: y = r coop +γ*min{k=1,2}Q ψ'k (h', π) θ' (h') + ); γ is the discount factor, ψ' k And θ' are the target network parameters. To trim noise.

[0049] Actor Network π θ (h) Outputs continuous acceleration, and its training objective is to maximize the expected reward through the policy gradient. The policy gradient function is as follows: θ J(θ)≈E{h ~D}[ a Q ψ1 (h,a)|{a=π θ( h )}· θ π θ( h ) ].

[0050] Preset DQN network Q network ω (h,a lane The parameter is ω. The preset training objective of the DQN network is to minimize the following loss function: L(ω)= {(h,a lane ,r coop ,h')~D}[(Q ω (h,a lane )-y dqn ) 2 ]; In the formula, the target value y dqn =r coop +γ*max{a' lane}Q ω' (h', a' lane), where ω' is the target Q-network parameter.

[0051] In the neural network model of this embodiment, the second Critic network Q in the preset TD3 network is set. ψ2 Directly concurrently serving as the central collaborative value function V coop V coop (s joint )=Q ψ2 (g( s joint The baseline value of g(·) is given by the formula, where g(·) is the feature adaptation mapping within the network. The target collaborative advantage function corresponding to the central collaborative value function is as follows: A coop (t)=r coop (t)+γ*V coop (V coop (t+1))-V coop (s joint (t)); In the formula, A coop (t) represents the target collaborative advantage value at time step t; V coop (s joint(t) represents the value of the central collaborative value function at time step t; V coop (V coop (t+1)) represents the value of the central collaborative value function corresponding to time step t+1; γ represents the discount factor.

[0052] In this embodiment, during the training process, the second Critic network Q... ψ2 It is directly used as the central collaborative value function (t), whose input is the fixed-dimensional joint state representation s generated above. joint (t), the output is the joint state representation s of this fixed dimension. joint Conservative value estimate under (t).

[0053] In this embodiment, after each decision step, the collaborative advantage value A is calculated using the target collaborative advantage function. coop (t)=r coop (t)+γ·(t)(s joint (t+1))-V coop (r coop (t)). This advantage value is backpropagated through the loss function of the Critic network and is implicitly used to guide the policy updates of the Actor and DQN to achieve collaborative optimization.

[0054] In this embodiment, the target collaborative advantage function is updated during training by updating Q. ψ2 And the influence of the policy gradient, guiding the policy of all vehicles (π) θ and Q ω By maximizing global collaborative benefits, centralized training and distributed execution can be achieved without adding new independent network parameters.

[0055] It is understandable that the aforementioned neural network model pre-configures a TD3 network for continuous vertical decision-making. Specifically, in this embodiment, a 50-dimensional fixed-dimensional joint state representation is input into the shared backbone network f. φ Feature extraction is performed to obtain the high-level feature vector h. i (t) The Actor network π further feeds into the preset TD3 network. θ A continuous longitudinal acceleration command ai acc(t) is generated, with a value range of [-4, 4] m / s². This Actor network follows a delayed update strategy, updating its parameters every two decision steps.

[0056] The aforementioned neural network model pre-defines a DQN network for lateral discrete decision-making. Specifically, it uses the same high-level feature vector h... i (t) The Q network fed into the preset DQN network Q ω Output each discrete lane-changing action a laneThe Q-values ​​∈ {-1,0,1}. In this embodiment, an ε-greedy exploration strategy is used in the early stage of training, where the exploration rate ε decreases linearly from 1 to 0.01, and finally the lane-changing decision ai lane(t) is selected based on the maximum Q-value.

[0057] Specifically, at each decision moment (100ms), this embodiment outputs lateral and longitudinal decisions in parallel. The actions are executed sequentially: first, a lane-changing decision is executed (this process is completed within 3 seconds), followed by longitudinal acceleration control. This embodiment completes the composite action a in one step. i Simultaneous decision-making (t)=(ai acc(t),ai lane(t)).

[0058] It is understandable that, such as Figure 2 As shown, after completing the hardware platform construction, simulation environment building, and collaborative reinforcement learning theoretical modeling, this embodiment performs distributed training and updating of the neural network model. Specifically, this embodiment can adopt an 8-core parallel computing architecture, starting 8 independent SUMO simulation processes as a parallel environment for collecting experience samples. The collected experience samples include a single-vehicle experience tuple (O) composed of compound actions, Q-values ​​corresponding to compound actions at adjacent time steps, and multi-objective collaborative reward functions. i (t),a i (t),r coop (t),O i (t+1)), and a joint state transition tuple (s) consisting of the fixed-dimensional joint state representation corresponding to adjacent time steps and the multi-objective cooperative reward function. joint (t),r coop (t),s joint (t+1)). All samples are stored uniformly in a global experience replay pool D. Then, after environmental interaction based on a preset step size, batch data is obtained from the global experience replay pool to update the network parameters of the preset TD3 network and the preset DQN network.

[0059] Specifically, in this embodiment, a centralized parameter update can be performed after every 20 steps of environmental interaction. This embodiment can involve sampling batch data from the global replay pool D and performing the following updates sequentially: a) Update the collaborative value function / Critic: Update the central collaborative value function Q based on the loss functions L(ψ2) and L(ψ1). ψ2 and another Critic network Q ψ1 The parameters.

[0060] b) Update vertical Actor: based on policy gradient θ J(θ) updates TD3 Actor network πθ The parameters.

[0061] c) Update the transverse DQN: Update the Q-network Q of the DQN according to the loss function L(ω). ω The parameters.

[0062] d) Update the target network: Update all target network parameters (θ', ψ' 1, ψ' 2) of TD3 using a soft update method; update the target network parameter ω' of DQN using either a hard update or a soft update method.

[0063] In this embodiment, using an RTX 5060 graphics card, approximately 8,000 steps of effective training can be completed in just 4 hours, with system memory usage of approximately 5GB during the training period.

[0064] Specifically, such as Figure 2 As shown, in this embodiment, after the training of the neural network model is completed, the shared backbone network, TD3 Actor network, DQN lateral decision head, pooling compression layer and collaborative weights are loaded onto the real vehicle. The corresponding system of the real vehicle receives 20-dimensional original state input, and after processing through the above fixed process, it finally outputs longitudinal acceleration and lateral lane change commands, with the dimensions fixed throughout.

[0065] Furthermore, the vehicle system corresponding to the method in this embodiment supports incremental upgrades via over-the-air (OTA). When optimizing the algorithm, only the weight files of parts such as the pooling compression layer need to be updated remotely; there is no need to recompile or reinstall the entire vehicle software. Verification shows that the system in this embodiment can run stably on vehicle ECUs with ≥4GB of memory, and the end-to-end inference latency from state input to action output is less than 10 milliseconds, meeting real-time control requirements.

[0066] As can be seen from the above, the method of this application embodiment has the following beneficial effects: (1) This embodiment uses only two RL networks, TD3 and DQN. By sharing the backbone network and multi-task head, the parameter increment can be effectively controlled.

[0067] (2) In this embodiment, a multi-objective collaborative reward function is used to replace pairwise reward, which is based on fleet average fuel consumption, CAV pass rate and number of red light stops. The dimensions are fixed and the reward function does not need to be rewritten for actual vehicles. (3) In this embodiment, the second Critic of TD3 (naturally conservative) directly serves as the collaborative value function, without adding a new network or overestimation problem, making the actual vehicle safer.

[0068] Please see Figure 3 This application also provides a multi-vehicle particle flow cooperative control device for mixed traffic flow, the device comprising: The first module is used to collect multi-dimensional state information to form a fixed-dimensional original state vector. The multi-dimensional state information includes the vehicle's kinematic state, its relative state with the vehicle in front, its relative state with adjacent vehicles in the left and right lanes, the SPaT information of the preset traffic lights, and the tag information of the connected automatic vehicle. The second module is used to perform pooling compression on the original state vector to obtain a fixed-dimensional joint state representation; The third module is used to input the fixed-dimensional joint state representation into the shared feature extraction backbone network for feature extraction, and obtain the high-level feature vector. The fourth module is used to input high-level feature vectors into a preset TD3 network and a preset DQN network respectively to make vehicle action decisions and obtain composite actions. The composite actions include continuous longitudinal acceleration commands and discrete lateral lane change commands. The preset TD3 network includes a first Critic network, a second Critic network and an Actor network. The second Critic network also serves as the central collaborative value function. The target collaborative advantage function corresponding to the central collaborative value function is used to guide the strategy training process of all vehicles to learn to maximize global benefits. The fifth module is used to calculate the multi-objective cooperative reward function in the process of compound motion control. The multi-objective cooperative reward function is calculated by the average fuel consumption of the fleet at a preset time step and the standardized values ​​of the traffic light pass rate and the number of traffic light stops for the intelligent connected vehicles. The sixth module is used to update the network parameters of the preset TD3 network and the preset DQN network based on batch data composed of fixed-dimensional joint state representation, compound actions, and multi-objective cooperative reward functions; The seventh module is used to perform driving control on the real vehicle after deploying and testing the preset TD3 network and preset DQN network based on the updated network parameters, as well as the corresponding pooled compression weights and the weights of the multi-objective collaborative reward function.

[0069] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0070] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0071] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0072] Please see Figure 4 , Figure 4 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 410 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 420 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 420 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 420 and is called and executed by the processor 410 using the methods described above in the embodiments of this application. Input / output interface 430 is used to realize information input and output; The communication interface 440 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 450 transmits information between various components of the device (e.g., processor 410, memory 420, input / output interface 430, and communication interface 440); The processor 410, memory 420, input / output interface 430 and communication interface 440 are connected to each other within the device via bus 450.

[0073] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0074] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0075] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0076] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0077] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0078] This application provides a multi-vehicle granular flow cooperative control method and related equipment for mixed traffic flow. After collecting multi-dimensional state information to form a fixed-dimensional original state vector, the original state vector is pooled and compressed to obtain a fixed-dimensional joint state representation. This fixed-dimensional joint state representation is then input into a shared feature extraction backbone network for feature extraction to obtain a high-level feature vector. This high-level feature vector is then input into a preset TD3 network and a preset DQN network respectively for vehicle action decision-making to obtain composite actions. The TD3 and DQN reinforcement learning networks are then used to achieve multi-vehicle longitudinal and transverse cooperative control, effectively improving the safety of vehicle driving in mixed traffic flow. Simultaneously, during the cooperative control process, a second Critic network serves as the central cooperative value function, thus avoiding the need to add new networks or increase overestimation risk, resulting in more stable training and lower fuel consumption. Furthermore, the multi-objective cooperative reward function in the composite action control process is calculated using the fleet's average fuel consumption and the standardized values ​​of traffic light pass rates and traffic light stop counts for intelligent connected vehicles. This ensures a fixed reward function dimension, eliminating the need to redeploy the reward function in actual vehicles and reducing deployment workload.

[0079] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0080] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0081] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0082] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0083] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0084] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0085] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0086] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0087] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0088] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0089] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A mixed traffic flow multi-vehicle flow cooperative control method, characterized in that, The method includes the following steps: Multidimensional state information is collected to form a fixed-dimensional original state vector. The multidimensional state information includes the vehicle's kinematic state, its relative state with the vehicle in front, its relative state with adjacent vehicles in the left and right lanes, the SPaT information of the preset traffic lights, and the tag information of the connected automatic vehicle. The original state vector is subjected to pooling compression to obtain a fixed-dimensional joint state representation; The fixed-dimensional joint state representation is input into the shared feature extraction backbone network for feature extraction to obtain a high-level feature vector. The high-level feature vectors are input into a preset TD3 network and a preset DQN network respectively for vehicle action decision-making to obtain composite actions, which include continuous longitudinal acceleration commands and discrete lateral lane change commands. The preset TD3 network includes a first Critic network, a second Critic network, and an Actor network. The second Critic network also serves as the central collaborative value function, and the target collaborative advantage function corresponding to the central collaborative value function is used to guide the strategy training process of all vehicles to learn to maximize global benefits. The multi-objective cooperative reward function in the composite motion control process is calculated. The multi-objective cooperative reward function is calculated by the average fuel consumption of the fleet at a preset time step and the standardized values ​​of the traffic light pass rate and the number of traffic light stops corresponding to the intelligent connected vehicle. The network parameters of the preset TD3 network and the preset DQN network are updated based on the batch data composed of the fixed-dimensional joint state representation, the composite action, and the multi-objective cooperative reward function. After deploying and testing the preset TD3 network and preset DQN network with updated network parameters, as well as the corresponding pooling compression weights and the weights of the multi-objective collaborative reward function in a real vehicle, driving control of the real vehicle is then performed.

2. The method of claim 1, wherein, The collected multi-dimensional state information forms a fixed-dimensional original state vector, including: The multi-dimensional status information is collected synchronously through the vehicle CAN bus, vehicle radar, and V2V communication module of the intelligent connected vehicle at a preset time period. The multidimensional state information is used to form the fixed-dimensional original state vector.

3. The method of claim 1, wherein, The step of performing pooling compression on the original state vector to obtain a fixed-dimensional joint state representation includes: The mean, maximum value, standard deviation, summation, and latest value of the original state vector are calculated along the feature dimension to obtain the extended statistical feature vector; The remaining green and red times of the traffic lights at multiple intersections ahead of the vehicle path are concatenated to obtain a global SPaT feature vector; The extended statistical feature vector and the global SPaT feature vector are concatenated to obtain the intermediate state vector; The intermediate state vector is linearly transformed and dimensionality reduced by a pooling compression layer to obtain the fixed-dimensional joint state representation, which is independent of the total number of vehicles in the traffic flow.

4. The method of claim 1, wherein, The multi-objective collaborative reward function is as follows: r coop (t) = -w f *F(t) + w f *S(t) - w l *L(t); In the formula, r coop (t) represents the multi-objective collaborative reward value; F(t) represents the average fuel consumption of the fleet at the preset time step t; S(t) represents the standardized value of the traffic light pass rate of the intelligent connected vehicle at the preset time step t; L(t) represents the standardized value of the number of times the intelligent connected vehicle stops at traffic lights at the preset time step t; w f w f and w l All are greater than 0 and are pre-set fixed weight coefficients.

5. The method of claim 4, wherein, The target collaborative advantage function is as follows: A coop (t)=r coop (t)+γ*V coop (V coop (t+1))-V coop (s joint (t)); In the formula, A coop (t) represents the target synergistic advantage value corresponding to time step t; V coop (s joint (t) represents the central synergistic value function value corresponding to time step t; V coop (V coop (t+1)) represents the central synergistic value function value corresponding to time step t+1; and γ represents a discount factor.

6. The method of claim 1, wherein, The step of updating the network parameters of the preset TD3 network and the preset DQN network based on the batch data composed of the fixed-dimensional joint state representation, the composite action, and the multi-objective cooperative reward function includes: The composite action, the Q value corresponding to the composite action at adjacent time steps, and the multi-objective cooperative reward function are combined to form a single-vehicle experience tuple; the fixed-dimensional joint state representation corresponding to adjacent time steps and the multi-objective cooperative reward function are combined to form a joint state transition tuple; Store all the single-vehicle experience tuples and the joint state transition tuples in the global experience replay pool; After environmental interaction based on a preset step size, batch data is obtained from the global experience replay pool to update the network parameters of the preset TD3 network and the preset DQN network.

7. The method of claim 6, wherein, The updating of the network parameters of the preset TD3 network and the preset DQN network includes: Update the central collaborative value function and the parameters of the first Critic network according to the first loss function and the second loss function; Update the parameters of the Actor network according to the policy gradient function; The parameters of the preset DQN network are updated according to the third loss function.

8. A hybrid traffic flow multi-vehicle flow cooperative control device characterized by comprising: The device includes: The first module is used to collect multi-dimensional state information to form a fixed-dimensional original state vector. The multi-dimensional state information includes the vehicle's kinematic state, the relative state with the vehicle in front, the relative state with adjacent vehicles in the left and right lanes, the SPaT information of the preset traffic lights, and the tag information of the connected automatic vehicle. The second module is used to perform pooling compression on the original state vector to obtain a fixed-dimensional joint state representation; The third module is used to input the fixed-dimensional joint state representation into the shared feature extraction backbone network for feature extraction to obtain a high-level feature vector. The fourth module is used to input the high-level feature vectors into a preset TD3 network and a preset DQN network respectively to make vehicle action decisions and obtain composite actions. The composite actions include continuous longitudinal acceleration commands and discrete lateral lane change commands. The preset TD3 network includes a first Critic network, a second Critic network, and an Actor network. The second Critic network also serves as the central collaborative value function. The target collaborative advantage function corresponding to the central collaborative value function is used to guide the strategy training process of all vehicles to learn to maximize global benefits. The fifth module is used to calculate the multi-objective cooperative reward function in the composite action control process. The multi-objective cooperative reward function is calculated by the average fuel consumption of the fleet at a preset time step and the standardized value of the traffic light pass rate and the standardized value of the number of traffic light stops corresponding to the intelligent connected vehicle. The sixth module is used to update the network parameters of the preset TD3 network and the preset DQN network based on the batch data composed of the fixed-dimensional joint state representation, the composite action, and the multi-objective cooperative reward function; The seventh module is used to perform driving control on the real vehicle after deploying and testing the preset TD3 network and preset DQN network based on the updated network parameters, as well as the corresponding pooled compression weights and the weights of the multi-objective collaborative reward function.

9. An electronic device, comprising: include: at least one processor; at least one memory for storing at least one program; the at least one program, when executed by the at least one processor, causing the at least one processor to implement the method according to any one of claims 1 to 7.

10. A computer program product comprising a computer program, characterized in that, the computer program, which when executed by a processor, implements the method according to any one of claims 1 to 7.

Images