Signal control method and system based on radar-vision fusion and deep reinforcement learning

Abstract

The present invention discloses a signal control method and system based on radar-visual fusion and deep reinforcement learning. The method includes: obtaining the spatiotemporal information of all vehicles passing through the intersection and the phase, stage, driving lane and prohibited lane contained in each cycle, calculating the queue length of the lane granularity, the vehicle state and stop time, vacant time, vacant ratio, vehicle extension time and traffic environment information of the vehicle granularity, and taking multiple index parameters as the evaluation index set, taking the maintenance of the current phase and switching to the target phase set as the control strategy set; constructing an index reward function according to the evaluation index set; constructing a maximum expectation function according to the evaluation index set, the control strategy set and the index reward function; taking the maximum expectation function as the target, inputting the evaluation index set and the control strategy set into the neural network for iterative calculation, obtaining the expected control strategy, and controlling the operation state of the signal.

Description

Technical Field

[0001] The present application relates to the field of intelligent transportation technology, and in particular to a signal control method and system based on radar-vision fusion and deep reinforcement learning. Background Art

[0002] With the rapid development of social economy, the scale of urbanization is getting larger and larger, and the population and vehicles are also growing continuously. In urban traffic, the increase of vehicles and population has caused more and more traffic congestion, especially during the morning and evening rush hours, which is prone to traffic accidents and road congestion. Therefore, the control of traffic lights at intersections is particularly important.

[0003] The traditional traffic light control method uses a fixed signal configuration. The fixed signal is obtained based on historical traffic data, which is used to set fixed time signals such as green light time, red light time, yellow light time, phase and cycle. However, the control method of the traditional traffic light control method is a fixed signal configuration derived from historical traffic data. The control method is fixed and difficult to change. It is impossible to reasonably control complex intersections and roads with large traffic flows, which makes intersections prone to congestion or traffic accidents. Summary of the invention

[0004] The purpose of this application is to solve the technical problem that the control method of traditional traffic light control methods is fixed and difficult to change, and it is impossible to reasonably control complex intersections and roads with large traffic flows. To achieve the above purpose, this application provides a traffic light control method and system based on radar-vision fusion and deep reinforcement learning.

[0005] The present application provides a signal control method based on radar-vision fusion and deep reinforcement learning, comprising:

[0006] Obtaining the spatiotemporal information of all vehicles passing through the intersection and the phase, stage, pass lane and prohibited lane contained in each cycle, wherein the spatiotemporal information is obtained based on the radar-visual fusion data;

[0007] According to the spatiotemporal information of all vehicles at the intersection and the phases, stages, traffic lanes and prohibited lanes contained in the cycle, the queue length at the lane granularity, the vehicle status and stop time, idle time, idle ratio, vehicle extension time and traffic environment information at the vehicle granularity are calculated;

[0008] The queue length of the lane granularity, the vehicle state and stop time of the vehicle granularity, the idle time, the idle ratio, the vehicle extension time and multiple index parameters in the traffic environment information are used as an evaluation index set, and maintaining the current phase and switching to the target phase set are used as a control strategy set;

[0009] Constructing an indicator reward function according to the evaluation indicator set;

[0010] Constructing a maximum expectation function according to the evaluation index set, the control strategy set and the index reward function;

[0011] Taking the maximum expected function as the target, inputting the evaluation index set into the neural network for iterative calculation to obtain the expected control strategy in the control strategy set;

[0012] The operating state of the signal is controlled according to the desired control strategy.

[0013] In one embodiment, constructing an indicator reward function according to the evaluation indicator set includes:

[0014] According to the vehicle state and the stop time of the vehicle granularity in the evaluation indicator set, a first cumulative parking time of all vehicles in a first state and a second cumulative parking time of all vehicles in a second state are calculated;

[0015] constructing the indicator reward function according to the first cumulative parking time and the second cumulative parking time;

[0016] Among them, the indicator reward function is:

[0017] r=r1-r2, r1 represents the first accumulated parking time, and r2 represents the second accumulated parking time.

[0018] In one embodiment, constructing an indicator reward function according to the evaluation indicator set further includes:

[0019] According to the vacant time in the evaluation indicator set, calculate and obtain the cumulative vacant time of the first lane of all lanes in the first state and the cumulative vacant time of the second lane of all lanes in the second state;

[0020] Constructing the indicator reward function according to the cumulative vacant time of the first lane and the cumulative vacant time of the second lane;

[0021] Among them, the indicator reward function is:

[0022] r=r3-r4, r3 represents the accumulated vacant time of the second lane, and r4 represents the accumulated vacant time of the second lane.

[0023] In one embodiment, constructing an indicator reward function according to the evaluation indicator set further includes:

[0024] According to the vehicle extension time in the evaluation indicator set, a first cumulative extension time between the first category of vehicles and the second category of vehicles in the first state and a second cumulative extension time between the first category of vehicles and the second category of vehicles in the second state are calculated;

[0025] Constructing the indicator reward function according to the first cumulative extension time and the second cumulative extension time;

[0026] Among them, the indicator reward function is:

[0027] r=r5-r6=(k1×d1+k2×d2)-(k1×d3+k2×d4), r5 represents the first cumulative extension time, r6 represents the second cumulative extension time, k1 represents the weight of the first category of vehicles, k2 represents the weight of the second category of vehicles, d1 represents the first category time of the first category of vehicles in the first state, d2 represents the second category time of the second category of vehicles in the first state, d3 represents the third category time of the first category of vehicles in the second state, and d4 represents the fourth category time of the second category of vehicles in the second state.

[0028] In one embodiment, constructing a maximum expectation function according to the evaluation index set, the control strategy set, and the index reward function includes:

[0029] According to the evaluation index set, the control strategy set and the index reward function, the maximum expectation function is constructed as follows:

[0030] Among them, s represents the evaluation index set, a represents the control strategy set, r represents the index reward function, and γ represents the discount coefficient.

[0031] In one embodiment, the present application provides a signal control system based on radar-vision fusion and deep reinforcement learning, including:

[0032] A data acquisition module is used to acquire the spatiotemporal information of all vehicles passing through the intersection and the phase, stage, pass lane and prohibited lane contained in each cycle, wherein the spatiotemporal information is acquired based on the radar-visual fusion data;

[0033] A control index acquisition module is used to calculate the queue length at the lane granularity, the vehicle status and stop time, idle time, idle ratio, vehicle extension time and traffic environment information at the vehicle granularity according to the spatiotemporal information of all vehicles at the intersection and the phase, stage, pass lane and prohibited lane contained in the cycle;

[0034] A control index and strategy building module, used to use the queue length of the lane granularity, the vehicle status and stop time of the vehicle granularity, the idle time, the idle ratio, the vehicle extension time and multiple index parameters in the traffic environment information as an evaluation index set, and to use the set of maintaining the current phase and switching to the target phase as a control strategy set;

[0035] An indicator reward function construction module, used to construct an indicator reward function according to the evaluation indicator set;

[0036] A maximum expectation function construction module, used to construct a maximum expectation function according to the evaluation index set, the control strategy set and the index reward function;

[0037] An expected control strategy acquisition module is used to take the maximum expected function as a target, input the evaluation index set into a neural network for iterative calculation, and obtain the expected control strategy in the control strategy set;

[0038] The control module is used to control the operating state of the signal according to the desired control strategy.

[0039] In one embodiment, the indicator reward function building module includes:

[0040] A cumulative parking time calculation module, used to calculate a first cumulative parking time of all vehicles in a first state and a second cumulative parking time of all vehicles in a second state according to the vehicle state and the stop time of the vehicle granularity in the evaluation indicator set;

[0041] A first reward construction module, configured to construct the indicator reward function according to the first cumulative parking time and the second cumulative parking time;

[0042] Among them, the indicator reward function is:

[0043] r=r1-r2, r1 represents the first accumulated parking time, and r2 represents the second accumulated parking time.

[0044] In one embodiment, the indicator reward function construction module further includes:

[0045] A cumulative vacant time calculation module, used to calculate the cumulative vacant time of the first lane of all lanes in the first state and the cumulative vacant time of the second lane of all lanes in the second state according to the vacant time in the evaluation index set;

[0046] A second reward construction module, used to construct the indicator reward function according to the cumulative vacant time of the first lane and the cumulative vacant time of the second lane;

[0047] Among them, the indicator reward function is:

[0048] r=r3-r4, r3 represents the accumulated vacant time of the second lane, and r4 represents the accumulated vacant time of the second lane.

[0049] In one embodiment, the indicator reward function construction module further includes:

[0050] A cumulative extension time calculation module, configured to calculate, based on the vehicle extension time of the evaluation index set, a first cumulative extension time between the first type of vehicle and the second type of vehicle in the first state, and a second cumulative extension time between the first type of vehicle and the second type of vehicle in the second state;

[0051] A third reward construction module, configured to construct the indicator reward function according to the first cumulative extension time and the second cumulative extension time;

[0052] Among them, the indicator reward function is:

[0053] r=r5-r6=(k1×d1+k2×d2)-(k1×d3+k2×d4), r5 represents the first cumulative extension time, r6 represents the second cumulative extension time, k1 represents the weight of the first category of vehicles, k2 represents the weight of the second category of vehicles, d1 represents the first category time of the first category of vehicles in the first state, d2 represents the second category time of the second category of vehicles in the first state, d3 represents the third category time of the first category of vehicles in the second state, and d4 represents the fourth category time of the second category of vehicles in the second state.

[0054] In one embodiment, the maximum expected function building block includes:

[0055] The target construction module is used to construct the maximum expectation function according to the evaluation index set, the control strategy set and the index reward function:

[0056] Among them, s represents the evaluation index set, a represents the control strategy set, r represents the index reward function, and γ represents the discount coefficient.

[0057] In the above-mentioned signal control method and system based on radar-visual fusion and deep reinforcement learning, the spatiotemporal information of all vehicles passing through the intersection and the phases, stages, traffic lanes and prohibited lanes contained in each cycle are obtained according to the radar-visual fusion data, timing plan information and high-precision map information, and the queue length at the lane granularity, the vehicle status and stop time, vacant time, vacant ratio, vehicle extension time and traffic environment information at the vehicle granularity are further obtained to form an evaluation index set. A control strategy set and an index reward function are constructed, and a maximum expectation function is constructed based on the evaluation index set, the control strategy set and the index reward function as the target of the iterative calculation of the neural network, and the control strategy corresponding to the optimal value is calculated as the expected control strategy. The signal is controlled according to the expected control strategy to achieve signal regulation at the intersection.

[0058] The signal control method based on radar-vision fusion and deep reinforcement learning provided in the present application is not only based on the multi-dimensional data of the intersection, but also based on the deep reinforcement learning of the neural network algorithm, and iteratively calculates to obtain the expected control strategy, so as to manage the traffic at the intersection more flexibly and adaptably, improve the traffic efficiency of the intersection, and be more conducive to the intelligent management of the intersection, solving the problems such as intersection congestion or traffic accidents. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is a schematic diagram of the step flow of the signal control method based on radar-vision fusion and deep reinforcement learning provided in this application.

[0060] Figure 2 It is a structural diagram of a signal control system based on radar-vision fusion and deep reinforcement learning provided in this application. DETAILED DESCRIPTION

[0061] The technical solution of the present application is further described in detail below through the accompanying drawings and embodiments.

[0062] See also Figure 1 The present application provides a signal control method based on radar-vision fusion and deep reinforcement learning, comprising:

[0063] S10, obtaining the spatiotemporal information of all vehicles passing through the intersection and the phase, stage, pass lane and prohibited lane contained in each cycle, wherein the spatiotemporal information is obtained based on the radar-visual fusion data;

[0064] S20, based on the spatiotemporal information of all vehicles at the intersection and the phases, stages, driving lanes and prohibited lanes included in the cycle, the queue length at the lane granularity, the vehicle status and stop time, idle time, idle ratio, vehicle extension time and traffic environment information at the vehicle granularity are calculated;

[0065] S30, taking the queue length at the lane granularity, the vehicle status and stop time at the vehicle granularity, the idle time, the idle ratio, the vehicle extension time and multiple indicator parameters in the traffic environment information as the evaluation indicator set, and taking the maintaining current phase and switching to the target phase set as the control strategy set;

[0066] S40, constructing an indicator reward function according to the evaluation indicator set;

[0067] S50, constructing a maximum expectation function according to the evaluation indicator set, the control strategy set and the indicator reward function;

[0068] S60, taking the maximum expected function as the target, inputting the evaluation index set into the neural network for iterative calculation to obtain the expected control strategy in the control strategy set;

[0069] S70: Control the operating state of the signal according to the desired control strategy.

[0070] In this embodiment, the radar-visual fusion data is derived from the fusion of radar data and video data. The radar-visual integrated machine is set at the intersection to obtain video image data and radar detection data. The video image data and radar detection data are fused to obtain the spatiotemporal information of all vehicles. The vehicle is captured through video image data, and vehicle attribute information such as license plate number, license plate color, vehicle color, etc. is identified. The vehicle distance, vehicle speed, etc. are identified through radar detection data. The video image data and radar detection data are transmitted via optical fiber, and the demodulator demodulates and transmits them to the platform end for fusion.

[0071] When the camera-based vehicle information and the radar-based vehicle information are fused in time and space, they can be divided into time fusion and space fusion. The radar coordinate system and the camera coordinate system are converted to realize the conversion of the radar coordinate system to the pixel coordinate system, and the spatial fusion of the video image data and the radar detection data is completed. The video image data and the radar detection data are collected synchronously in time to realize the fusion in time. High-precision maps can provide the stop line contour position information, lane line contour position information and longitude and latitude information of the intersection. The relative time and space information includes the relative spatial position of the vehicle relative to the intersection. According to the distance of the vehicle relative to the intersection, the relative spatial position can be converted into the spatial position in the geographic coordinate system, and the corresponding longitude and latitude position of the vehicle in the geographic coordinate system is obtained. Thus, the relative time and space information of each vehicle relative to the intersection is converted into time and space information, and then the driving trajectory of each vehicle is known.

[0072] The spatiotemporal information includes the latitude and longitude position of a vehicle at a certain moment, time information, vehicle location information, vehicle speed information, vehicle driving direction information, vehicle model information, vehicle license plate information, and vehicle color information, etc. The corresponding vehicle can be marked on the map according to the corresponding latitude and longitude position of the vehicle, providing relevant geographic location information support.

[0073] The timing scheme information is obtained through the signal control machine. The timing scheme information is calculated together with the lane information displayed in the high-precision map to obtain multiple stages and phases contained in a cycle and the corresponding lanes for passage and prohibition of passage. A cycle can be understood as the time required for the color of the signal light to change one cycle according to the set signal phase sequence. A cycle contains n stages or n phases. The phase represents a moment state, and the stage represents the duration of the state. The stages and phases contained in the cycle are obtained by timing. The traffic lights indicate the flow direction allowed to pass, and then the lanes are matched with the flow direction to obtain the passable lanes. For example: the phases that allow passage are north straight, north right, south straight, and south right. The right turn lane and the straight + right turn lane in the north-south straight lane are all passable, that is, the passable lane is formed. The corresponding time is assigned to each phase to form a stage.

[0074] Through the phase information in the timing plan, we can know the flow direction allowed at any time (such as south to north, east to south, etc.). By obtaining the position and direction of the lane (such as the first lane on the west side, left turn) from the high-precision map, we can know the lane flow direction (such as west to north). By analogy, we can know whether the lanes corresponding to each phase at any time are open or prohibited, and then obtain the phase, stage, open lane and prohibited lane information contained in each cycle.

[0075] Granularity can be understood as data refinement. Granularity can be day, hour, minute, second, millisecond, etc. According to the status of each vehicle at each moment, the length of the section between the front position of the first vehicle in the queue and the rear position of the last vehicle in the queue corresponding to the queue formed by multiple consecutive stopped vehicles on the lane is used as the queue length. Alternatively, the queue length of the queue is obtained based on the number of vehicles between the front position of the first vehicle in the queue and the rear vehicle in the queue. The queue length can be the length of the distance section or the length of the number of vehicles. The vehicle status is divided into stopped vehicles or non-stopped vehicles based on whether the vehicle is stopped at each moment.

[0076] The extended time period, the starter sub-loss time period, and the clearing sub-loss time period of each vehicle in the stage time of each lane are removed, and the remaining time is the idle time of each lane. Furthermore, according to the idle time of each lane, the direction-level idle time, the intersection idle time, the stage idle time, etc. can be obtained. The idle time can be any one of the lane idle time, the direction-level idle time, the intersection idle time, and the stage idle time, which can be set according to actual needs. The idle ratio can be understood as the ratio of the idle time to a stage or a cycle.

[0077] Each vehicle passing through the stop line within the phase time of each lane is time-marked to obtain the marking time of each vehicle, and the saturated headway time of each vehicle is extended to obtain the vehicle extension time of each vehicle. Traffic environment information includes upstream intersection flow, weather, working / rest days and other information.

[0078] The evaluation index set can be understood as a combined data set of one or more of the queue length at the lane granularity, the vehicle status and stop time at the vehicle granularity, the vacant time, the vacant ratio, the vehicle extension time, and the traffic environment information. The evaluation index set is not limited to the queue length at the lane granularity, the vehicle status and stop time at the vehicle granularity, the vacant time, the vacant ratio, the vehicle extension time, and the traffic environment information, but can also include other traffic information parameters related to the intersection.

[0079] The control strategy set includes two execution actions: maintaining the current phase and switching to the target phase set. The current phase can be understood as the phase corresponding to the current state. Maintaining the current phase can be understood as continuing to provide the green light direction for the duration of the current phase. Switching to the target phase set can be understood as switching to other phases in the target phase set, or it can be understood as the red light corresponding to the current phase is on, allowing the flow direction of other phases to pass. The target phase set includes phase one, phase two, phase three, phase four, etc. Phase one, phase two, phase three, phase four, etc. in the target phase set are different from the current phase and do not conflict with the current phase. For example, a left turn and waiting to stop requires the left turn to be placed in the current direction of straight driving. The target phase set can be understood as a set of phases that need to be switched to, and the current phase can be switched to any target phase in the target phase set.

[0080] The evaluation indicator set represents a certain state, and the control strategy set represents the actions that can be performed. For example, if the queue length becomes longer, the light color can be switched. The queue length is the state in the evaluation indicator set, and switching the light color is the action of switching to the target phase set in the control strategy set.

[0081] The indicator return function can be understood as the attribute of the strategy composed of the evaluation indicator set and the control strategy set. Different control indicators correspond to different control strategies and produce different returns. The greater the return corresponding to the indicator return function, the greater the value. For different states in the evaluation indicator set, different actions in the control strategy set can be executed to produce different returns. The evaluation indicator set, the control strategy set, and the indicator return function are used to construct a value function. The value function conforms to the basic form of the Bellman equation and can be iteratively calculated through a neural network to obtain the optimal value function, that is, the maximum expectation function, and then obtain the optimal strategy corresponding to the maximum expectation function.

[0082] The maximum expected function is used as the target of iterative calculation, the evaluation index set and the control strategy set are input into the neural network for iterative calculation, the value function is calculated, different value values ​​corresponding to different control strategies are obtained, and the control strategy corresponding to the optimal value value is further obtained as the expected control strategy. The expected control strategy is used as the optimal strategy. Thus, according to the expected control strategy (for example, maintaining the current phase or switching to the target phase set), the operation state of the signal is controlled.

[0083] The signal control method based on radar-visual fusion and deep reinforcement learning provided in the present application obtains the spatiotemporal information of all vehicles passing through the intersection and the phases, stages, traffic lanes and prohibited lanes contained in each cycle based on the radar-visual fusion data, timing plan information and high-precision map information, and further obtains the queue length at the lane granularity, the vehicle status and stop time at the vehicle granularity, the idle time, the idle ratio, the vehicle extension time and the traffic environment information, etc., to form an evaluation index set. Construct a control strategy set and an index reward function, and construct a maximum expectation function based on the evaluation index set, the control strategy set and the index reward function as the target of the iterative calculation of the neural network, and calculate the control strategy corresponding to the optimal value as the expected control strategy. Control the signal according to the expected control strategy to achieve signal regulation at the intersection.

[0084] The signal control method based on radar-vision fusion and deep reinforcement learning provided in the present application is not only based on the multi-dimensional data of the intersection, but also based on the deep reinforcement learning of the neural network algorithm, and iteratively calculates to obtain the expected control strategy, so as to manage the traffic at the intersection more flexibly and adaptably, improve the traffic efficiency of the intersection, and be more conducive to the intelligent management of the intersection, solving the problems such as intersection congestion or traffic accidents.

[0085] In one embodiment, S40, constructing an indicator reward function according to the evaluation indicator set includes:

[0086] S411, calculating a first cumulative parking time of all vehicles in a first state and a second cumulative parking time of all vehicles in a second state according to the vehicle state and the stop time at the vehicle granularity in the evaluation indicator set;

[0087] S412, constructing an indicator reward function according to the first cumulative parking time and the second cumulative parking time;

[0088] Among them, the indicator return function is:

[0089] r=r1-r2, r1 represents the first accumulated parking time, and r2 represents the second accumulated parking time.

[0090] In this embodiment, the vehicle state can be divided into a parking state and a non-parking state, and the vehicle state can be determined by comparing the instantaneous speed at the current moment with the speed threshold. In one embodiment, if the state at the previous moment is a non-parking state, it is determined whether the instantaneous speed of the vehicle at the current moment is less than the low speed threshold. If so, the state at the current moment is a parking state, and if not, the state at the current moment is a non-parking state; if the state at the previous moment is a parking state, it is determined whether the instantaneous speed of the vehicle at the current moment is less than the high speed threshold. If so, the state at the current moment is a non-parking state, and if not, the state at the current moment is a parking state.

[0091] In one embodiment, the first state and the second state are adjacent states. The first state and the second state can be understood as the previous state and the current state, which represent the state of a certain moment or period. For a certain lane or a certain direction or a certain intersection, according to the parking state of each vehicle at each moment within a period of time, the duration corresponding to multiple continuous parking states is summed to obtain the parking duration of each vehicle. The parking durations of all vehicles in the first state are summed to obtain the first cumulative parking duration. The parking durations of all vehicles in the second state are summed to obtain the second cumulative parking duration. The first cumulative parking duration and the second cumulative parking duration may correspond to a certain lane or a certain direction or a certain intersection. An indicator reward function is constructed based on the difference between the first cumulative parking duration and the second cumulative parking duration.

[0092] By constructing an indicator reward function based on the first accumulated parking time and the second accumulated parking time corresponding to the first state and the second state in this embodiment, the vehicle delay time can be shortened as much as possible, allowing the vehicle to pass the intersection as quickly as possible.

[0093] In one embodiment, S40, constructing an indicator reward function according to the evaluation indicator set, further includes:

[0094] S421, calculating the accumulated vacant time of the first lane of all lanes in the first state and the accumulated vacant time of the second lane of all lanes in the second state according to the evaluation indicator concentration vacant time;

[0095] S422, constructing an indicator reward function according to the cumulative vacant time of the first lane and the cumulative vacant time of the second lane;

[0096] Among them, the indicator return function is:

[0097] r=r3-r4, r3 represents the cumulative vacant time of the second lane, and r4 represents the cumulative vacant time of the second lane.

[0098] In this embodiment, the vacant time may be the vacant time of the lane, the vacant time at the direction level, the vacant time at the intersection, and the vacant time at the stage. For the vacant time of each lane, the extended time period of each vehicle in the stage time of each lane, the starter loss time period, and the clearing sub-loss time period are removed to obtain the vacant time. The vacant time of each lane in the same direction is intersected to obtain the vacant time at the direction level. The vacant time period at the direction level of each direction at the intersection is intersected to obtain the vacant time period at the intersection. The vacant time period of each lane at each stage in a cycle is intersected to obtain the stage vacant time period.

[0099] When the vacant time is the vacant time of the lane, the vacant time of all lanes in the first state is summed up to obtain the cumulative vacant time of the first lane. The vacant time of all lanes in the second state is summed up to obtain the cumulative vacant time of the second lane. Using the difference between the cumulative vacant time of the first lane and the cumulative vacant time of the second lane as the indicator return function, the vacant time of the intersection can be reduced and the overall utilization rate can be improved.

[0100] In one embodiment, S40, constructing an indicator reward function according to the evaluation indicator set, further includes:

[0101] S431, according to the evaluation index, the vehicle extension time is concentrated, and a first cumulative extension time between the first category vehicle and the second category vehicle in the first state and a second cumulative extension time between the first category vehicle and the second category vehicle in the second state are calculated;

[0102] S432, constructing an indicator reward function according to the first cumulative extension time and the second cumulative extension time;

[0103] Among them, the indicator return function is:

[0104] r=r5-r6=(k1×d1+k2×d2)-(k1×d3+k2×d4), r5 represents the first cumulative extended time, r6 represents the second cumulative extended time, k1 represents the weight of the first category of vehicles, k2 represents the weight of the second category of vehicles, d1 represents the first category time of the first category of vehicles in the first state, d2 represents the second category time of the second category of vehicles in the first state, d3 represents the third category time of the first category of vehicles in the second state, and d4 represents the fourth category time of the second category of vehicles in the second state.

[0105] In this embodiment, the vehicle extension time can be calculated based on the difference between the actual driving time of the non-free crossing intersection and the standard passing time of the free crossing intersection. The standard passing time represents the time of passing the intersection freely and without any obstruction during the green light period. The vehicle extension time represents the time consumed by the vehicle being blocked at the intersection and the loss of driving time.

[0106] The vehicle types of the first category vehicles and the second category vehicles are different. In one embodiment, the first category vehicles may be special vehicles, such as buses, ambulances, fire trucks, etc. The second category vehicles may be ordinary vehicles, such as private cars, motorcycles, bicycles, etc. Different calculation weights are assigned to different types of vehicles, and the reward ratios assigned to different types in the reward calculation process are divided. k1 represents the weight of the first category of vehicles, k2 represents the weight of the second category of vehicles, k1 is 2 to 4 times the value of k2, and the specific value can be adjusted according to actual conditions. Through the present embodiment, the weight setting of each different vehicle type in the index reward function can greatly improve the vehicle weight of special vehicles, so that the release of special vehicles has a greater reward, and the control of the signal control machine signal is more flexible and applicable to various traffic conditions.

[0107] In one embodiment, S50, constructing a maximum expectation function according to the evaluation index set, the control strategy set, and the index reward function includes:

[0108] S510, based on the evaluation index set, the control strategy set and the index reward function, a maximum expectation function is constructed as follows:

[0109] Among them, s represents the evaluation index set, a represents the control strategy set, r represents the index return function, and γ represents the discount coefficient.

[0110] In this embodiment, a value function can be constructed based on the evaluation index set, the control strategy set and the index reward function. In this embodiment, the value function adopts the Q function. The value of the Q function is calculated iteratively through a neural network to obtain multiple different value values. In the iterative calculation process, the maximum expected function is constructed, which can also be understood as the maximum Q function expected value as the target of the iterative calculation, and finally the optimal Q value is obtained, and then the optimal strategy corresponding to the optimal Q value is obtained to realize the control of the operating state of the signal machine.

[0111] See also Figure 2 The present application provides a signal control system 100 based on radar-visual fusion and deep reinforcement learning. The signal control system 100 based on radar-visual fusion and deep reinforcement learning includes a data acquisition module 10, a control index acquisition module 20, a control index and strategy construction module 30, an index reward function construction module 40, a maximum expectation function construction module 50, an expected control strategy acquisition module 60 and a control module 70. The data acquisition module 10 is used to obtain the spatiotemporal information of all vehicles passing through the intersection and the phase, stage, pass lane and prohibited lane contained in each cycle. The spatiotemporal information is obtained based on the radar-visual fusion data.

[0112] The control index acquisition module 20 is used to calculate the queue length at the lane granularity, the vehicle status and stop time, vacant time, vacant ratio, vehicle extension time and traffic environment information at the vehicle granularity according to the spatiotemporal information of all vehicles at the intersection and the phase, stage, passable lane and prohibited lane contained in the cycle. The control index and strategy construction module 30 is used to use the queue length at the lane granularity, the vehicle status and stop time, vacant time, vacant ratio, vehicle extension time and traffic environment information at the vehicle granularity as the evaluation index set, and to use the set of maintaining the current phase and switching to the target phase as the control strategy set. The index reward function construction module 40 is used to construct an index reward function based on the evaluation index set.

[0113] The maximum expectation function construction module 50 is used to construct the maximum expectation function according to the evaluation index set, the control strategy set and the indicator reward function. The expected control strategy acquisition module 60 is used to input the evaluation index set into the neural network for iterative calculation with the maximum expectation function as the target, and obtain the expected control strategy in the control strategy set. The control module 70 is used to control the operating state of the signal according to the expected control strategy.

[0114] In this embodiment, the relevant description of the data acquisition module 10 may refer to the relevant description of S10 in the above embodiment. The relevant description of the control indicator acquisition module 20 may refer to the relevant description of S20 in the above embodiment. The relevant description of the control indicator and strategy construction module 30 may refer to the relevant description of S30 in the above embodiment. The relevant description of the indicator reward function construction module 40 may refer to the relevant description of S40 in the above embodiment. The relevant description of the maximum expectation function construction module 50 may refer to the relevant description of S50 in the above embodiment. The relevant description of the expected control strategy acquisition module 60 may refer to the relevant description of S60 in the above embodiment. The relevant description of the control module 70 may refer to the relevant description of S70 in the above embodiment.

[0115] In one embodiment, the indicator reward function construction module 40 includes a cumulative parking time calculation module and a first reward construction module. The cumulative parking time calculation module is used to calculate the first cumulative parking time of all vehicles in the first state and the second cumulative parking time of all vehicles in the second state according to the vehicle state and stop time of the vehicle granularity in the evaluation indicator set. The first reward construction module is used to construct an indicator reward function according to the first cumulative parking time and the second cumulative parking time.

[0116] Among them, the indicator return function is:

[0117] r=r1-r2, r1 represents the first accumulated parking time, and r2 represents the second accumulated parking time.

[0118] In this embodiment, the description of the accumulated parking time calculation module can refer to the description of S411 in the above embodiment. The description of the first reporting construction module can refer to the description of S412 in the above embodiment.

[0119] In one embodiment, the indicator reward function construction module 40 also includes a cumulative vacant time calculation module and a second reward construction module. The cumulative vacant time calculation module is used to calculate the cumulative vacant time of the first lane of all lanes in the first state and the cumulative vacant time of the second lane of all lanes in the second state according to the evaluation index. The second reward construction module is used to construct an indicator reward function according to the cumulative vacant time of the first lane and the cumulative vacant time of the second lane.

[0120] Among them, the indicator return function is:

[0121] r=r3-r4, r3 represents the cumulative vacant time of the second lane, and r4 represents the cumulative vacant time of the second lane.

[0122] In this embodiment, the description of the accumulated idle time calculation module can refer to the description of S421 in the above embodiment. The description of the second report construction module can refer to the description of S422 in the above embodiment.

[0123] In one embodiment, the indicator reward function construction module 40 also includes a cumulative extension time calculation module and a third reward construction module. The cumulative extension time calculation module is used to calculate the first cumulative extension time of the first type of vehicle and the second type of vehicle in the first state and the second cumulative extension time of the first type of vehicle and the second type of vehicle in the second state according to the evaluation index. The third reward construction module is used to construct an indicator reward function according to the first cumulative extension time and the second cumulative extension time.

[0124] Among them, the indicator return function is:

[0125] r=r5-r6=(k1×d1+k2×d2)-(k1×d3+k2×d4), r5 represents the first cumulative extended time, r6 represents the second cumulative extended time, k1 represents the weight of the first category of vehicles, k2 represents the weight of the second category of vehicles, d1 represents the first category time of the first category of vehicles in the first state, d2 represents the second category time of the second category of vehicles in the first state, d3 represents the third category time of the first category of vehicles in the second state, and d4 represents the fourth category time of the second category of vehicles in the second state.

[0126] In this embodiment, the description of the cumulative extension time calculation module can refer to the description of S431 in the above embodiment. The description of the third report construction module can refer to the description of S432 in the above embodiment.

[0127] In one embodiment, the maximum expected function construction module 50 includes a target construction module. The target construction module is used to construct the maximum expected function according to the evaluation index set, the control strategy set and the index reward function.

[0128] Among them, s represents the evaluation index set, a represents the control strategy set, r represents the index return function, and γ represents the discount coefficient.

[0129] In this embodiment, the relevant description of the target construction module may refer to the relevant description of S510 in the above embodiment.

[0130] In the above-mentioned various embodiments, the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process can be rearranged without departing from the scope of protection of the present disclosure. The attached method claims provide the elements of various steps in an exemplary order and are not intended to be limited to a specific order or hierarchy.

[0131] Those skilled in the art may also understand that the various illustrative logical blocks, modules and steps listed in the embodiments of the present application may be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly demonstrate the interchangeability of hardware and software, the various illustrative components, modules and steps described above have generally described their functions. Whether such functions are implemented by hardware or software depends on the specific application and the design requirements of the entire system. Those skilled in the art may use various methods to implement the described functions for each specific application, but such implementation should not be understood as exceeding the scope of protection of the embodiments of the present application.

[0132] The various illustrative logic blocks or modules described in the embodiments of the present application can be implemented or operated by a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or the design of any combination of the above functions. The general-purpose processor can be a microprocessor, and optionally, the general-purpose processor can also be any traditional processor, controller, microcontroller or state machine. The processor can also be implemented by a combination of computing devices, such as a digital signal processor and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a digital signal processor core, or any other similar configuration.

[0133] The steps of the method or algorithm described in the embodiments of the present application can be directly embedded in the software module executed by the hardware, the processor, or a combination of the two. The software module can be stored in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM or other storage media of any form in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from the storage medium and can write information to the storage medium. Alternatively, the storage medium can also be integrated into the processor. The processor and the storage medium can be arranged in an ASIC, and the ASIC can be arranged in a user terminal. Optionally, the processor and the storage medium can also be arranged in different components in the user terminal.

[0134] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only the specific implementation method of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A signal control method based on radar-vision fusion and deep reinforcement learning, characterized in that: include: Obtaining the spatiotemporal information of all vehicles passing through the intersection and the phase, stage, pass lane and prohibited lane contained in each cycle, wherein the spatiotemporal information is obtained based on the radar-visual fusion data; According to the spatiotemporal information of all vehicles at the intersection and the phases, stages, traffic lanes and prohibited lanes contained in the cycle, the queue length at the lane granularity, the vehicle status and stop time, idle time, idle ratio, vehicle extension time and traffic environment information at the vehicle granularity are calculated; The queue length of the lane granularity, the vehicle state and stop time of the vehicle granularity, the idle time, the idle ratio, the vehicle extension time and multiple index parameters in the traffic environment information are used as an evaluation index set, and maintaining the current phase and switching to the target phase set are used as a control strategy set; Constructing an indicator reward function according to the evaluation indicator set; Constructing a maximum expectation function according to the evaluation index set, the control strategy set and the index reward function; Taking the maximum expected function as the target, inputting the evaluation index set into the neural network for iterative calculation to obtain the expected control strategy in the control strategy set; Controlling the operating state of the signal according to the desired control strategy; The constructing of the maximum expectation function according to the evaluation index set, the control strategy set and the index reward function comprises: According to the evaluation index set, the control strategy set and the index reward function, the maximum expectation function is constructed as follows: Among them, s represents the evaluation index set, a represents the control strategy set, r represents the index reward function, and γ represents the discount coefficient.

2. The signal control method based on radar-visual fusion and deep reinforcement learning according to claim 1 is characterized in that: The constructing an indicator reward function according to the evaluation indicator set includes: According to the vehicle state and the stop time of the vehicle granularity in the evaluation indicator set, a first cumulative parking time of all vehicles in a first state and a second cumulative parking time of all vehicles in a second state are calculated; constructing the indicator reward function according to the first cumulative parking time and the second cumulative parking time; Among them, the indicator reward function is: r=r1-r2, r1 represents the first accumulated parking time, and r2 represents the second accumulated parking time.

3. The signal control method based on radar-visual fusion and deep reinforcement learning according to claim 1 is characterized in that: The constructing of the indicator reward function according to the evaluation indicator set also includes: According to the vacant time in the evaluation indicator set, calculate and obtain the cumulative vacant time of the first lane of all lanes in the first state and the cumulative vacant time of the second lane of all lanes in the second state; Constructing the indicator reward function according to the cumulative vacant time of the first lane and the cumulative vacant time of the second lane; Among them, the indicator reward function is: r=r3-r4, r3 represents the accumulated vacant time of the second lane, and r4 represents the accumulated vacant time of the second lane.

4. The signal control method based on radar-visual fusion and deep reinforcement learning according to claim 1 is characterized in that: The constructing of the indicator reward function according to the evaluation indicator set also includes: According to the vehicle extension time in the evaluation indicator set, a first cumulative extension time between the first category of vehicles and the second category of vehicles in the first state and a second cumulative extension time between the first category of vehicles and the second category of vehicles in the second state are calculated; Constructing the indicator reward function according to the first cumulative extension time and the second cumulative extension time; Among them, the indicator reward function is: r=r5-r6=(k1×d1+k2×d2)-(k1×d3+k2×d4), r5 represents the first cumulative extension time, r6 represents the second cumulative extension time, k1 represents the weight of the first category of vehicles, k2 represents the weight of the second category of vehicles, d1 represents the first category time of the first category of vehicles in the first state, d2 represents the second category time of the second category of vehicles in the first state, d3 represents the third category time of the first category of vehicles in the second state, and d4 represents the fourth category time of the second category of vehicles in the second state.

5. A signal control system based on radar-vision fusion and deep reinforcement learning, characterized in that: include: A data acquisition module is used to acquire the spatiotemporal information of all vehicles passing through the intersection and the phase, stage, pass lane and prohibited lane contained in each cycle, wherein the spatiotemporal information is acquired based on the radar-visual fusion data; A control index acquisition module is used to calculate the queue length at the lane granularity, the vehicle status and stop time, idle time, idle ratio, vehicle extension time and traffic environment information at the vehicle granularity according to the spatiotemporal information of all vehicles at the intersection and the phase, stage, pass lane and prohibited lane contained in the cycle; A control index and strategy building module, used to use the queue length of the lane granularity, the vehicle status and stop time of the vehicle granularity, the idle time, the idle ratio, the vehicle extension time and multiple index parameters in the traffic environment information as an evaluation index set, and to use the set of maintaining the current phase and switching to the target phase as a control strategy set; An indicator reward function construction module, used to construct an indicator reward function according to the evaluation indicator set; A maximum expectation function construction module, used to construct a maximum expectation function according to the evaluation index set, the control strategy set and the index reward function; An expected control strategy acquisition module is used to take the maximum expected function as a target, input the evaluation index set into a neural network for iterative calculation, and obtain the expected control strategy in the control strategy set; A control module, used for controlling the operation state of the signal according to the desired control strategy; The maximum expected function building block includes: The target construction module is used to construct the maximum expectation function according to the evaluation index set, the control strategy set and the index reward function: Among them, s represents the evaluation index set, a represents the control strategy set, r represents the index reward function, and γ represents the discount coefficient.

6. The signal control system based on radar-visual fusion and deep reinforcement learning according to claim 5 is characterized in that: The indicator return function building module includes: A cumulative parking time calculation module, used to calculate a first cumulative parking time of all vehicles in a first state and a second cumulative parking time of all vehicles in a second state according to the vehicle state and the stop time of the vehicle granularity in the evaluation indicator set; A first reward construction module, configured to construct the indicator reward function according to the first cumulative parking time and the second cumulative parking time; Among them, the indicator reward function is: r=r1-r2, r1 represents the first accumulated parking time, and r2 represents the second accumulated parking time.

7. The signal control system based on radar-visual fusion and deep reinforcement learning according to claim 5 is characterized in that: The indicator reward function building module also includes: A cumulative vacant time calculation module, used to calculate the cumulative vacant time of the first lane of all lanes in the first state and the cumulative vacant time of the second lane of all lanes in the second state according to the vacant time in the evaluation index set; A second reward construction module, used to construct the indicator reward function according to the cumulative vacant time of the first lane and the cumulative vacant time of the second lane; Among them, the indicator reward function is: r=r3-r4, r3 represents the accumulated vacant time of the second lane, and r4 represents the accumulated vacant time of the second lane.

8. The signal control system based on radar-visual fusion and deep reinforcement learning according to claim 5 is characterized in that: The indicator reward function building module also includes: A cumulative extension time calculation module, configured to calculate, based on the vehicle extension time of the evaluation index set, a first cumulative extension time between the first type of vehicle and the second type of vehicle in the first state, and a second cumulative extension time between the first type of vehicle and the second type of vehicle in the second state; A third reward construction module, configured to construct the indicator reward function according to the first cumulative extension time and the second cumulative extension time; Among them, the indicator reward function is: r=r5-r6=(k1×d1+k2×d2)-(k1×d3+k2×d4), r5 represents the first cumulative extension time, r6 represents the second cumulative extension time, k1 represents the weight of the first category of vehicles, k2 represents the weight of the second category of vehicles, d1 represents the first category time of the first category of vehicles in the first state, d2 represents the second category time of the second category of vehicles in the first state, d3 represents the third category time of the first category of vehicles in the second state, and d4 represents the fourth category time of the second category of vehicles in the second state.

Images