Vehicle infrastructure collaborative awareness method based on deep reinforcement learning

Abstract

The invention provides a vehicle-road cooperative sensing method based on deep reinforcement learning, and the method comprises the steps: obtaining the positions of all CAVs through a road side unit (RSU) through a cooperative object selection method based on the RSU, distributing an optimal cooperative object for each CAV through employing a DQN network, and enabling the CAVs to achieve the cooperative object after obtaining an object needing to be cooperated, and selecting an optimal three-dimensional detection frame and a corresponding confidence score by using a DQN network based on invalid action shielding, and sending the optimal three-dimensional detection frame and the corresponding confidence score to a collaborative object to realize vehicle-road collaborative perception.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent transportation systems, and specifically relates to a vehicle-road collaborative perception method based on deep reinforcement learning. Background Art

[0002] In the intelligent transportation system, the Internet of Vehicles can improve the connectivity of all-round network connections of vehicles, enable vehicles to have a higher level of intelligence, and ultimately achieve the deep integration of people, vehicles, and the environment. Vehicles in the same area share cooperative perception messages (CPMs) with each other to collaboratively perceive the surrounding environment, which is called multi-vehicle beyond-visual-range cooperative perception. Although the development of cooperative perception technology has greatly improved the perception capabilities of vehicles, it also faces some challenges brought by the communication bandwidth limitations of the actual system. In actual situations, if the complete original perception data or a large number of perception intermediate features are to be transmitted, it will be difficult to bear the huge real-time communication consumption. Therefore, in reality, we cannot only improve the environmental perception performance of vehicles without evaluating the precious communication bandwidth cost. In order to mitigate the negative impact caused by broadcasting CPMs between vehicles and achieve a balance between perception performance and communication bandwidth, vehicles should intelligently select collaborative vehicles and data content.

[0003] Due to the urgent need for collaborative perception environment in the Internet of Vehicles, some vehicle collaborative perception methods have been proposed in the prior art. In order to improve the perception accuracy and ensure the reliability of perception, it is necessary to select from two aspects: collaborative perception content and collaborative perception object. In terms of collaborative perception content selection, the Actor-Critic reinforcement learning algorithm adaptively adjusts the transmission ratio of raw data, feature data and object-level data in a highly dynamic Internet of Vehicles environment, thereby enhancing the average detection accuracy. Cobevt generates bird's-eye view map predictions in collaboration with surrounding vehicles through vehicle-to-vehicle communication in the perception framework of cameras with multiple perspectives, uses the Transformer architecture to fuse image features from multiple views and multiple vehicles, and designs a fusion axial attention module to capture sparse local features and global spatial interactions. The above method only considers the selection of collaborative content and broadcasts the selected collaborative message to all nearby autonomous vehicles, resulting in redundant information exchange between vehicles. In terms of collaborative object association, Who2com uses a handshake communication mechanism to calculate the matching score between the requester's query vector and the supporter's keyword vector to measure the correlation between the two vectors, thereby determining the collaborative object. However, the model assumes that each agent always needs to communicate with one of the other agents, and sending unnecessary perception information causes additional communication overhead. To solve this problem, When2com uses activation functions to prune unimportant connections, learns to decide when to communicate and create communication groups, and further reduces the consumption of communication bandwidth. In terms of joint selection of collaborative content and collaborative objects, the perception method of action branching and joint reinforcement learning uses early collaboration to store and exchange raw sensor information between vehicles. In order to alleviate the problem of a large amount of communication resources required by this method, regional quadtree compression technology is used to store the perception data of each vehicle in two-dimensional space. However, due to the large amount of raw point cloud data transmitted by early collaboration, a large amount of communication bandwidth will be consumed. Where2comm uses mid-term collaboration to extract spatial confidence maps and request maps from the feature maps obtained by each agent through encoding multimodal sensor information, and reflects the perception level of each spatial area in the same coordinate system, thereby constructing a communication map, determining the communicating vehicles and the perception messages sent. Finally, perception message fusion is performed on the feature map to obtain a more comprehensive target detection prediction result. However, due to the inconsistent data formats and different processing methods from different devices in mid-term collaboration, computing resources will be wasted and the time required for data processing will be increased. Summary of the invention

[0004] In order to overcome the problems and challenges of the above-mentioned prior art, in order to improve the average perception accuracy of the vehicle and save the consumption of collaborative data, the present invention proposes a vehicle-road collaborative perception method based on deep reinforcement learning, and uses a deep reinforcement learning method based on invalid action shielding to reduce the consumption of vehicle computing resources. In order to solve the problem of too large dimensions in the joint selection of collaborative objects and collaborative content, the present invention models the action space as a two-dimensional discrete action space, decouples the selection problems of collaborative objects and collaborative content, and designs an RSU-based collaborative object selection method and a vehicle-based collaborative content selection method. Among them, the RSU-based collaborative object method uses Dueling DQN to select the best vehicle collaborative object according to the state space of RSU. After the vehicle receives the collaborative object, it knows the object that needs to be collaborated, and sends the collaborative content to the collaborative object. Since the actual action selection is only the result perceived by the vehicle, invalid action shielding is used to narrow the range of selected actions, and Dueling DQN based on invalid action shielding is proposed to select the 3D bounding box and the corresponding confidence score that need to be sent to its collaborative object.

[0005] The present invention relates to a collaborative perception method in V2X (Vehicle to Everything), which decouples and separates the two actions of collaborative object selection and collaborative content selection, and proposes a vehicle-road collaborative perception method based on deep reinforcement learning, which is used to improve the perception ability of autonomous driving vehicles. First, a collaborative object selection method based on a roadside unit is used. The roadside unit obtains the positions of all autonomous driving vehicles, and uses a deep Q network to assign the best collaborative object to each autonomous driving vehicle. The autonomous driving vehicle obtains its own radar posture and its own perception results with the collaborative vehicle, and uses a deep Q network based on invalid action shielding to select the best three-dimensional detection frame and the corresponding confidence score, and sends the best three-dimensional detection frame and the corresponding confidence score to the collaborative object. The present invention effectively improves the accuracy of collaborative perception of autonomous driving vehicles and reduces the communication volume of collaborative perception.

[0006] The technical solution specifically adopted by the present invention to solve the technical problem is:

[0007] A vehicle-road collaborative perception method based on deep reinforcement learning: through a collaborative object selection method based on a roadside unit RSU, the roadside unit RSU obtains the positions of all autonomous driving vehicles CAV, and uses a DQN network to assign the best collaborative object to each autonomous driving vehicle CAV. After the autonomous driving vehicle CAV learns the object that needs to be collaborated, it uses a DQN network based on invalid action shielding to select the best three-dimensional detection frame and the corresponding confidence score, and sends the best three-dimensional detection frame and the corresponding confidence score to the collaborative object to achieve vehicle-road collaborative perception.

[0008] Furthermore, assume that at time slot t, N autonomous driving vehicles CAV are within the service coverage of the roadside unit RSU, and RSU is used to obtain the position information p of each vehicle in real time. n (t) = [x n (t),y n (t)], each CAV independently senses and allows communication to other CAVs within the coverage area of ​​the RSU at time slot t;

[0009] The RSU uses the DQN network to assign a perception object to each vehicle according to the RSU state space, and sends the assigned cooperative CAV number to the corresponding CAV without delay;

[0010] After the corresponding CAV receives the collaboration object number assigned by the RSU, it learns the object that needs to collaborate. On this basis, CAV n selects the 3D bounding box and corresponding confidence score that need to be sent to its collaboration object according to the vehicle state space through the DQN network based on invalid action masking to achieve vehicle-road collaborative perception.

[0011] Furthermore, the method for selecting a cooperative object based on a roadside unit RSU includes:

[0012] The state space of the RSU is given by the positions of all CAVs served by the RSU. The state space of the RSU-agent at time slot t is expressed as:

[0013] s RSU (t) = {p n (t)}, n∈N

[0014] Based on the RSU state at the start time of each time slot, the RSU takes the cooperative sensing vehicle association action matrix E(t):

[0015] E(t)=[e nn′ (t)]

[0016] Where E(t) is the global vehicle association matrix controlled by the RSU in time slot t. nn′ When e(t)=1, it means that CAV n sends a cooperative sensing message to CAV n′ in time slot t. On the contrary, when e(t)=1, it means that CAV n sends a cooperative sensing message to CAV n′ in time slot t. nn′ When (t) = 0, it means that CAV n does not send a collaborative sensing message to CAV n′ at time slot t; the RSU sends the collaborative object association of each vehicle to the corresponding CAV through the V2I link at time slot t;

[0017] Assume that the RSU action limits the association selection between vehicles to be bidirectional, that is, e nn′ (t) = e n′n(t), which means that when CAV n sends to CAV n′ in time slot t, CAV n′ also sends to CAV n; then the dimension of the action space of RSU in time slot t is Action a allowed by the RSU in time slot t RSU (t) is expressed as:

[0018]

[0019] Therefore, after the RSU assigns a cooperation object to each vehicle at the beginning of each time slot, it sends the number of the assigned cooperation CAV to the corresponding CAV without delay.

[0020] Furthermore, the specific implementation process of using the DQN network to assign the best collaboration partner to each autonomous driving vehicle CAV is as follows:

[0021] The goal of the DQN network is to allow the agent to learn and optimize strategies through interaction with the environment to maximize long-term cumulative rewards: where γ is the discount factor, r t is the immediate reward; the action value function Q Π Is the cumulative reward U t Expectations, Q Π Depends only on the current state s t and action a t , and is related to the strategy function Π; Q Π It is expressed as:

[0022]

[0023] Q Π Maximize Π to further eliminate Π, that is, among all possible strategy functions Π, choose the one that makes Q Π The largest strategy Π obtains the optimal value function Q*; the optimal value function Q* is expressed as:

[0024] Q * (s t , a t )=max π Q π (s t , a t )

[0025] The DQN network uses a deep neural network to approximate the Q value function. The fitting process is:

[0026] Q(s, a; w)≈Q*(s, a)

[0027] Where w represents the parameters of the neural network; the best action taken by RSU is a t = argmax aQ(s t , a; w);

[0028] Using Dueling DQN, the Q-value function is decomposed into the state value function V(s) and the advantage function A(s, a), which represent the average value of the state and the advantage of the state, respectively, expressed as:

[0029]

[0030] in represents the state-value network, represents the dominant network, θ=(θ v ,θ a ), which represents the parameters of the network.

[0031] Furthermore, the process of selecting the best three-dimensional detection frame and the corresponding confidence score using the DQN network based on invalid action shielding, and sending the best three-dimensional detection frame and the corresponding confidence score to the collaborative object is specifically as follows:

[0032] After receiving the cooperation object number assigned by RSU, CAV learns the object that needs to cooperate; CAV uses Dueling DQN to send the 3D bounding box and the corresponding confidence score to its cooperation object; the state space of CAV n in time slot t is expressed as:

[0033] s n (t) = {l n (t), l n′ (t), B n (t), C n (t)}

[0034] Among them l n (t), l n′ (t) are the LiDAR positions of the user and the neighbor, respectively. n (t), C n (t) are the 3D bounding box and confidence score of the independent prediction result at time slot t;

[0035] Each vehicle shares the same action space at time slot t. The possible cooperative message selected by CAV n at time slot t is A vehicles including all autonomous vehicles CAV and background vehicles. The total number of possible actions in the cooperative content selection problem is:

[0036]

[0037] The action space of CAV n in time slot t It is expressed as:

[0038]

[0039] For each perception frame τ , the valid action of each CAV n is determined by its own perceived bounding box and the corresponding score, and the invalid action masking method is used to ensure that CAV does not continuously output invalid actions; the value function in the case of invalid action masking of Dueling DQN is:

[0040]

[0041] in The action space of CAV n at time t From to Where A is the number of all autonomous driving vehicles CAV and background vehicles, A collection of confidence scores representing the detected 3D bounding boxes.

[0042] And, a vehicle-road collaborative perception system based on deep reinforcement learning, comprising: at least one roadside unit RSU and multiple autonomous driving vehicles CAV; through a collaborative object selection method based on the roadside unit RSU, the roadside unit RSU obtains the positions of all autonomous driving vehicles CAV, and uses a DQN network to assign the best collaborative object to each autonomous driving vehicle CAV. After the autonomous driving vehicle CAV learns the object that needs to be collaborated, it uses a DQN network based on invalid action shielding to select the best three-dimensional detection frame and the corresponding confidence score, and sends the best three-dimensional detection frame and the corresponding confidence score to the collaborative object to achieve vehicle-road collaborative perception.

[0043] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of a vehicle-road collaborative perception method based on deep reinforcement learning as described above are implemented.

[0044] A non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a vehicle-road collaborative perception method based on deep reinforcement learning as described above.

[0045] Compared with the prior art, the present invention and its preferred solution select the cooperative object based on RSU and the cooperative content based on the vehicle, which improves the perception accuracy and reduces the cooperative communication volume. On this basis, a deep reinforcement learning method based on invalid action shielding is proposed. For complex scenarios in the Internet of Vehicles, a deep network is used to fit the value Q function in cooperative perception, and an invalid action shielding method is used to ensure that the intelligent agent does not continuously output invalid actions, thereby accelerating the convergence of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0047] Figure 1 A vehicle-road collaborative perception method based on deep reinforcement learning is provided in an embodiment of the present invention.

[0048] Figure 2 It is a comparison of CAV average perception confidence scores during the training of different strategies provided by the embodiments of the present invention.

[0049] Figure 3 This is a comparison of the amount of collaborative data consumption during the training of different strategies provided by the embodiments of the present invention. DETAILED DESCRIPTION

[0050] In order to make the features and advantages of the present invention more clearly understood, the following embodiments are specifically described in detail as follows:

[0051] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used in this specification have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0052] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0053] The present invention provides a vehicle-road cooperative perception method based on deep reinforcement learning, comprising the following steps:

[0054] Step S1: At time slot t, N CAVs are within the service coverage of the RSU. The RSU can obtain the location information p of each vehicle in real time. n (t) = [x n (t), y n (t)], each CAV senses independently in time slot t and can communicate to other CAVs within the coverage area of ​​the RSU;

[0055] Step S2, the RSU uses a deep Q network (DQN) model to assign a perception object to each vehicle according to the RSU state space, and sends the assigned cooperative CAV number to the corresponding CAV without delay;

[0056] Step S3: After receiving the collaboration object number assigned by the RSU, the CAV learns the object that needs to be collaborated with. The CAV selects the 3D bounding box and the corresponding confidence score that need to be sent to its collaboration object according to the vehicle state space through the deep Q network (DQN) model based on invalid action masking.

[0057] The following is the specific implementation process of the present invention.

[0058] like Figure 1 As shown, this example is a vehicle-road cooperative perception method based on deep reinforcement learning, including the following steps:

[0059] Step S1: At time slot t, N CAVs are within the service coverage of the RSU. The RSU can obtain the location information p of each vehicle in real time. n (t) = [x n (t), y n (t)], each CAV senses independently in time slot t and can communicate to other CAVs within the coverage area of ​​the RSU;

[0060] Step S2, the RSU uses a deep Q network (DQN) model to assign a perception object to each vehicle according to the RSU state space, and sends the assigned cooperative CAV number to the corresponding CAV without delay;

[0061] Step S3: After receiving the collaboration object number assigned by the RSU, the CAV learns the object that needs to be collaborated with. The CAV selects the 3D bounding box and the corresponding confidence score that need to be sent to its collaboration object according to the vehicle state space through the deep Q network (DQN) model based on invalid action masking.

[0062] Furthermore, the specific implementation process of step S1 is as follows:

[0063] At time slot t, N CAVs are within the service coverage of the RSU, and the RSU can obtain the location information p of each vehicle in real time. n (t) = [x n (t), y n (t)]. The state space of RSU in reinforcement learning is given by the positions of all CAVs served by RSU, so the state space of RSU-agent at time slot t is expressed as:

[0064] s RSU (t) = {p n (t)}, n∈N

[0065] Based on the RSU state at the start time of each time slot, the RSU takes the cooperative sensing vehicle association action matrix E(t):

[0066] E(t)=[e nn′ (t)]

[0067] E(t) is the global vehicle association matrix controlled by the RSU in time slot t. nn′ When e(t)=1, it means that CAV n sends a cooperative sensing message to CAV n′ in time slot t. On the contrary, when e(t)=1, it means that CAV n sends a cooperative sensing message to CAV n′ in time slot t. nn′ When (t) = 0, it means that CAV n does not send a cooperative sensing message to CAV n′ at time slot t. The RSU sends the cooperative object association of each vehicle to the corresponding CAV through the V2I link at time slot t. In addition, it is assumed that the vehicle association is bidirectional, that is, e nn′ (t) = e n′n (t) indicates that when CAV n sends to CAV n′ in time slot t, CAV n′ also sends to CAV n.

[0068] Since the RSU action limits the association selection between vehicles to be bidirectional, the dimension of the RSU action space in time slot t is Therefore, the action a that the RSU can perform in time slot t is RSU (t) is:

[0069]

[0070] After the RSU assigns a cooperative object to each vehicle at the beginning of each time slot, it sends the number of the assigned cooperative CAV to the corresponding CAV without delay.

[0071] Furthermore, the specific implementation process of step S2 is as follows:

[0072] The goal of reinforcement learning is to enable the agent to learn and optimize strategies through interaction with the environment to maximize long-term cumulative rewards: Where γ is the discount factor and rt is the immediate reward. The action value function QΠ is the expectation of the cumulative reward Ut, which is equivalent to eliminating the randomness of all future states and actions. Π Depends only on the current state s t and action a t , and is related to the strategy function Π. Π It is expressed as:

[0073]

[0074] Q Π Regarding maximizing Π, we can further eliminate Π, that is, among all possible strategy functions Π, we choose to make Q Π The largest strategy Π, thus we get the optimal value function Q*. The optimal value function Q* is expressed as:

[0075] Q*(s t , a t )=maxπ Q π (s t , a t )

[0076] The Deep Q-learning Network (DQN) algorithm approximates the Q-value function by using a deep neural network. The fitting process of DQN is:

[0077] Q(s, a; w)≈Q*(s, a)

[0078] Where w represents the parameters of the neural network. The best action taken by RSU is a t = argmax a Q(s t , a; w).

[0079] To improve the estimation efficiency and performance of the Q-value function, Dueling DQN is used to decompose the Q-value function into a state value function.

[0080] V(s) and advantage function A(s, a) represent the average value and advantage of the state, that is, the relative value of the state corresponding to different actions. The architecture is expressed as:

[0081]

[0082] in represents the state-value network, represents the dominant network, θ=(θ v ,θ a ), which represents the parameters of the network.

[0083] Furthermore, the specific implementation process of step S3 is as follows:

[0084] After receiving the cooperation object number assigned by RSU, CAV learns the object to be cooperated. CAV uses Dueling DQN to send the 3D bounding box and the corresponding confidence score to its cooperation object. The state space of CAV n in time slot t is represented as:

[0085] s n (t) = {l n (t), l n′ (t), B n (t), C n (t)}

[0086] Among them l n (t), l n′ (t) are the LiDAR positions of the user and the neighbor, respectively. n (t), C n(t) are the 3D bounding box and confidence score of the independent prediction results at time slot t, respectively.

[0087] Each vehicle shares the same action space at time slot t, and the possible cooperative message selected by CAV n at time slot t is vehicle A including all autonomous vehicles and background vehicles. The total number of possible actions in the cooperative content selection problem is:

[0088]

[0089] The action space of CAV n in time slot t It is expressed as:

[0090]

[0091] In practice, for each perception frame τ , the valid action of each CAV n is determined by its own perceived bounding box and the corresponding score. The invalid action shielding method is used to ensure that the agent does not continuously output invalid actions, thereby preventing the agent from always selecting invalid actions. The value function in the case of invalid action shielding of Dueling DQN is:

[0092]

[0093] in The action space of CAV n at time t From to Where A is the number of all autonomous driving vehicles and background vehicles, A collection of confidence scores representing the detected 3D bounding boxes.

[0094] The method of the present invention is improved in terms of collaborative perception, and compared with other collaborative perception methods, it is found that compared with existing algorithms, this solution can consume the lowest communication volume while maintaining the highest perception accuracy. Figure 2 It is a comparison of CAV average perceptual confidence scores during the training of different strategies in an embodiment of the present invention. Figure 3 This is a comparison of the amount of data consumed in collaboration during the training of different strategies in one embodiment of the present invention.

[0095] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0096] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0097] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0098] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0099] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0100] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

[0101] The present invention is not limited to the above-mentioned optimal implementation mode. Anyone can derive other various forms of a vehicle-road collaborative perception method based on deep reinforcement learning under the inspiration of the present invention. All equal changes and modifications made within the scope of the invention application of the present invention should fall within the scope of the present invention.

Claims

1. A vehicle-road cooperative perception method based on deep reinforcement learning, characterized by: Through the collaborative object selection method based on the roadside unit RSU, the roadside unit RSU obtains the positions of all autonomous driving vehicles CAV, and uses the DQN network to assign the best collaborative object to each autonomous driving vehicle CAV. After the autonomous driving vehicle CAV learns the object that needs to cooperate, it uses the DQN network based on invalid action shielding to select the best three-dimensional detection frame and the corresponding confidence score, and sends the best three-dimensional detection frame and the corresponding confidence score to the collaborative object to realize vehicle-road collaborative perception.

2. The vehicle-road cooperative perception method based on deep reinforcement learning according to claim 1 is characterized in that: Assume that at time slot t, N autonomous driving vehicles CAV are within the service coverage of the roadside unit RSU, and RSU is used to obtain the position information p of each vehicle in real time n (t) = [x n (t),y n (t)], each CAV independently senses and allows communication to other CAVs within the coverage area of ​​the RSU at time slot t; The RSU uses the DQN network to assign a perception object to each vehicle according to the RSU state space, and sends the assigned cooperative CAV number to the corresponding CAV without delay; After the corresponding CAV receives the collaboration object number assigned by the RSU, it learns the object that needs to collaborate. On this basis, CAV n selects the 3D bounding box and corresponding confidence score that need to be sent to its collaboration object according to the vehicle state space through the DQN network based on invalid action masking to achieve vehicle-road collaborative perception.

3. The vehicle-road cooperative perception method based on deep reinforcement learning according to claim 1, characterized in that: The method for selecting a cooperative object based on a roadside unit RSU includes: The state space of the RSU is given by the positions of all CAVs served by the RSU. The state space of the RSU-agent at time slot t is expressed as: s RSU (t)={p n (t)},n∈N Based on the RSU state at the start time of each time slot, the RSU takes the cooperative sensing vehicle association action matrix E(t): E(t)=[e nn' (t)] Where E(t) is the global vehicle association matrix controlled by the RSU in time slot t. nn' When (t) = 1, it means that CAV n sends a cooperative sensing message to CAV n' at time slot t. On the contrary, when e nn' When (t) = 0, it means that CAV n does not send a collaborative sensing message to CAV n' at time slot t; the RSU sends the collaborative object association of each vehicle to the corresponding CAV through the V2I link at time slot t; Assume that the RSU action limits the association selection between vehicles to be bidirectional, that is, e nn' (t) = e n'n (t), which means that when CAV n sends to CAV n' in time slot t, CAV n' also sends to CAV n; then the dimension of the action space of RSU in time slot t is Action a allowed by the RSU in time slot t RSU (t) is expressed as: Therefore, after the RSU assigns a cooperation object to each vehicle at the beginning of each time slot, it sends the number of the assigned cooperation CAV to the corresponding CAV without delay.

4. The vehicle-road cooperative perception method based on deep reinforcement learning according to claim 1, characterized in that: The specific implementation process of using the DQN network to assign the best collaboration object to each autonomous driving vehicle CAV is as follows: The goal of the DQN network is to allow the agent to learn and optimize strategies through interaction with the environment to maximize long-term cumulative rewards: where γ is the discount factor, r t For immediate rewards; Action-value function Q ∏ Is the cumulative reward U t Expectations, Q ∏ Depends only on the current state s t and action a t , and is related to the policy function ∏; Q ∏ It is expressed as: Q ∏ Maximize ∏ to further eliminate ∏, that is, among all possible strategy functions ∏, choose to make Q ∏ The largest strategy ∏ obtains the optimal value function Q*; the optimal value function Q* is expressed as: Q*(s t ,a t )=max π Q π (s t ,a t ) The DQN network uses a deep neural network to approximate the Q value function. The fitting process is: Q(s,a;w)≈Q * (s,a) Where w represents the parameters of the neural network; the best action taken by RSU is a t = argmax a Q(s t ,a;w); Using Dueling DQN, the Q value function is decomposed into the state value function V(s) and the advantage function A(s,a), which represent the average value and advantage of the state respectively, expressed as: in represents the state-value network, represents the dominant network, θ=(θ v ,θ a ), which represents the parameters of the network.

5. The vehicle-road cooperative perception method based on deep reinforcement learning according to claim 1, characterized in that: The process of selecting the best three-dimensional detection frame and the corresponding confidence score using the DQN network based on invalid action shielding, and sending the best three-dimensional detection frame and the corresponding confidence score to the collaborative object is specifically as follows: After receiving the collaboration object number assigned by RSU, CAV learns the object that needs to be collaborated; CAV uses DuelingDQN to send the 3D bounding box and the corresponding confidence score to its collaboration object; the state space of CAV n in time slot t is expressed as: s n (t)={l n (t),l n' (t),B n (t),C n (t)} Among them l n (t),l n' (t) are the LiDAR positions of the user and the neighbor, respectively. n (t), C n (t) are the 3D bounding box and confidence score of the independent prediction result at time slot t; Each vehicle shares the same action space at time slot t. The possible cooperative message selected by CAV n at time slot t is A vehicles including all autonomous vehicles CAV and background vehicles. The total number of possible actions in the cooperative content selection problem is: The action space of CAV n in time slot t It is expressed as: For each perception frame τ, the valid action of each CAV n is determined by its own perceived bounding box and the corresponding score. The invalid action masking method is used to ensure that the CAV does not continuously output invalid actions; the value function in the case of invalid action masking of Dueling DQN is: in The action space of CAV n at time t From to Where A is the number of all autonomous driving vehicles CAV and background vehicles, A collection of confidence scores representing the detected 3D bounding boxes.

6. A vehicle-road cooperative perception system based on deep reinforcement learning, characterized in that: include: At least one roadside unit RSU and multiple autonomous driving vehicles CAV; Through the collaborative object selection method based on the roadside unit RSU, the roadside unit RSU obtains the positions of all autonomous driving vehicles CAV, and uses the DQN network to assign the best collaborative object to each autonomous driving vehicle CAV. After the autonomous driving vehicle CAV learns the object that needs to cooperate, the DQN network based on invalid action shielding is used to select the best three-dimensional detection frame and the corresponding confidence score, and the best three-dimensional detection frame and the corresponding confidence score are sent to the collaborative object to realize vehicle-road collaborative perception.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it implements the steps of a vehicle-road collaborative perception method based on deep reinforcement learning as described in any one of claims 1-5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of a vehicle-road collaborative perception method based on deep reinforcement learning are implemented as described in any one of claims 1 to 5.

Images