A highway lane control method and system based on real-time traffic flow

Abstract

The embodiment of the present invention discloses a highway lane control method and system based on real-time traffic flow. The method includes: constructing multiple intelligent agents based on reinforcement learning, and different intelligent agents have different corresponding prediction durations; selecting a target intelligent agent corresponding to a prediction duration, and predicting the optimal control measures within the future prediction duration; using a simulation model to simulate the time-varying curve of the traffic flow data of the controlled section under the optimal control measures; using the optimal control measures to actually control the controlled section; if the difference between the actual traffic flow data at a certain moment and the time-varying curve exceeds the set range, record the actual control duration up to the certain moment; provide feedback rewards to the target intelligent agent according to the actual control duration; reselect the prediction duration according to the actual control duration, and start a new round of prediction. This embodiment improves the accuracy and anti-interference of highway control.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of intelligent transportation technology, and in particular to a highway lane control method and system based on real-time traffic flow. Background Art

[0002] At present, in the process of highway traffic control, traffic control strategies are mostly formulated by managers based on experience. The control effect is not ideal and cannot adapt to the dynamic changes of traffic flow.

[0003] There are also some methods in the prior art that use artificial intelligence technology based on reinforcement learning to formulate highway control strategies, but most methods are difficult to detect in time when the control strategy deviates from the expected control effect and take corresponding measures, resulting in limited accuracy and anti-interference of highway control. In addition, patent CN117894174A provides a road traffic safety active control method device and system, and patent CN114170793A provides a highway lane dynamic control method, both of which cannot solve the above problems well. Summary of the invention

[0004] The embodiments of the present invention provide a highway lane control method and system based on real-time traffic flow to solve the above technical problems.

[0005] In a first aspect, an embodiment of the present invention provides a highway lane control method based on real-time traffic flow, comprising:

[0006] Construct multiple reinforcement learning-based intelligent agents, where each intelligent agent is used to predict the optimal lane-level control measures for the controlled road section at different time periods in the future, and different intelligent agents have different corresponding prediction time periods;

[0007] Selecting a target intelligent agent corresponding to a prediction duration, and using the target intelligent agent to predict optimal control measures within the prediction duration in the future according to current traffic flow data of the controlled road section;

[0008] Using a simulation model, simulating a time-varying curve of traffic flow data of the controlled road section under the optimal control measure, wherein the time-varying curve represents an ideal control state of the optimal control measure;

[0009] Using the optimal control measures to actually control the controlled road section, and recording the actual traffic flow data after the control in real time;

[0010] If the difference between the actual traffic flow data at a certain moment and the time-varying curve exceeds the set range, the actual control duration up to the certain moment is recorded; if the certain moment does not exist, the predicted duration of the target intelligent agent is used as the actual control duration;

[0011] According to the actual control time, the target intelligent agent is given a feedback reward, wherein the feedback reward is used to characterize the anti-interference ability of the target intelligent agent to actual traffic factors, and the longer the actual control time, the higher the reward value;

[0012] According to the actual control time and the accumulated reward value of each agent under the actual control time, the predicted time is reselected, and the agent corresponding to the new predicted time is used as the new target agent. The operation of using the target agent to predict the optimal control measures is returned to start a new round of prediction.

[0013] In a second aspect, an embodiment of the present invention provides an electronic device, the electronic device comprising:

[0014] one or more processors;

[0015] a memory for storing one or more programs,

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the highway lane control method based on real-time traffic flow described in any embodiment.

[0017] In a third aspect, an embodiment of the present invention further provides a highway lane control system based on real-time traffic flow, comprising: a plurality of intelligent agents based on reinforcement learning, a traffic flow simulation model and a control module;

[0018] Among them, each intelligent agent is used to predict the optimal lane-level control measures for the controlled road section at different time periods in the future, and different intelligent agents have different corresponding prediction time periods;

[0019] The control module is used to select a target intelligent agent corresponding to a predicted duration, and use the target intelligent agent to predict the optimal control measures within the predicted duration in the future according to the current traffic flow data of the controlled road section;

[0020] The simulation model is used to simulate the time-varying curve of the traffic flow data of the controlled road section under the optimal control measure, wherein the time-varying curve represents the ideal control state of the optimal control measure;

[0021] The control module is also used to use the optimal control measures to actually control the controlled road section, and record the actual traffic flow data after the control in real time; if the difference between the actual traffic flow data at a certain moment and the time-varying curve exceeds the set range, record the actual control time up to the certain moment; if the certain moment does not exist, use the predicted time of the target intelligent agent as the actual control time; provide feedback reward to the target intelligent agent based on the actual control time, wherein the feedback reward is used to characterize the anti-interference ability of the target intelligent agent to actual traffic factors, and the longer the actual control time, the higher the reward value; based on the actual control time and the accumulated reward value of each intelligent agent under the actual control time, reselect the predicted time, and use the intelligent agent corresponding to the new predicted time as the new target intelligent agent, return to the operation of using the target intelligent agent to predict the optimal control measures, and start a new round of prediction.

[0022] The embodiments of the present invention provide a highway lane control method and system based on real-time traffic flow, which can achieve the following beneficial effects:

[0023] 1. Since traffic control measures take some time to improve traffic conditions, the agents in this embodiment predict the optimal control measures for a period of time in the future to provide enough time for the control measures to take effect;

[0024] 2. We have built multiple agents with different prediction durations, and reduced or increased the prediction duration for the next round based on the actual control duration of the previous round of predictions and the accumulated rewards of each agent under the control duration, thus achieving adaptive dynamic changes in the prediction duration and improving the accuracy of prediction and control;

[0025] 3. Compare the actual traffic status under control measures with the ideal control status in real time, promptly identify the period when the actual situation deviates from the ideal state and stop the control, and conduct a second feedback training for the agent used according to the actual control duration to enhance the agent's ability to resist interference from actual traffic;

[0026] 4. Provide comprehensive feedback to the intelligent agent used based on the predicted duration and the actual control duration, and accurately reflect the influence of the two in the reward feedback by setting a reasonable reward function, so that it is consistent with the action law and optimization law of each parameter in the control measure prediction, thereby gradually making up for the difference between the simulation model and the actual traffic status, and improving the accuracy and anti-interference ability of the entire control method. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0028] Figure 1 is a flow chart of a highway lane control method based on real-time traffic flow provided by an embodiment of the present invention;

[0029] Figure 2 It is a schematic diagram of the structure and data flow of each intelligent agent provided by an embodiment of the present invention;

[0030] Figure 3 It is a structural schematic diagram of a highway lane control system based on real-time traffic flow provided by an embodiment of the present invention;

[0031] Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.

[0033] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0034] In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0035] Figure 1 is a flow chart of a highway lane control method based on real-time traffic flow provided by an embodiment of the present invention. The method is executed by an electronic device, such as Figure 1 As shown, the method specifically includes:

[0036] S110, construct multiple intelligent agents based on reinforcement learning, wherein each intelligent agent is used to predict the lane-level optimal control measures of the controlled road section in different time periods in the future, and different intelligent agents have different corresponding prediction time periods.

[0037] This embodiment constructs multiple independent agents based on the principle of reinforcement learning. Each agent is used to predict the lane-level optimal control measures for different time periods in the future based on the current traffic flow data of the controlled section. Among them, the traffic flow data may include at least one traffic flow parameter, such as at least one of traffic flow, vehicle occupancy rate, traffic flow density, headway, and average vehicle speed; the control measures may include variable speed limits for each lane, and ramp merging flow restrictions may be included in the ramp area. Since the duration of effective control of control measures under different traffic conditions is different, and the prediction difficulty of control measures under different prediction time periods is also different, this embodiment constructs different agents for different prediction time periods. For example, if a certain agent is used to predict the optimal control measures within the next 2 hours, then under normal circumstances, the optimal control measures will continue to be put into operation within the next 2 hours; another agent is used to predict the optimal control measures within the next 4 hours, then under normal circumstances, the optimal control measures will continue to be put into operation within the next 4 hours.

[0038] Furthermore, each intelligent agent can be implemented based on a virtual traffic flow prediction model. The prediction model can adopt a traffic flow simulation model or other neural network models. The following will take the simulation model as an example of a traffic flow prediction model. Optionally, the simulation model can be implemented using simulation software such as VISSIM and SUMO, or using simulation methods such as cellular machines. After calibrating the simulation model based on the actual traffic flow of the controlled section, the simulation model can simulate the changes in traffic flow in the future based on the current traffic flow data and control input of each lane.

[0039] Based on the above traffic flow simulation model, each intelligent agent uses the traffic flow data of the controlled section as the state variable, the lane-level control measures within the corresponding prediction time as the action variable, the total number of vehicles passing through and the congestion index within the corresponding prediction time as the reward function, and uses the simulation model to calculate the change of the state variable. Exemplarily, the traffic flow parameters of each lane in each segment of the controlled section are arranged in a certain order to obtain a state variable vector. At the same time, when the action variable includes a variable speed limit, the value range of the variable speed limit of each lane can be discretized, and the discrete value combination of the variable speed limit of each lane constitutes an action variable. All discrete value combinations together constitute the action space. Exemplarily, the value range of the variable speed limit of each lane is [v1, v2]. The range is discretized into 3 values, with a total of 4 lanes, and a total of 3 4 action variables; the action space of ramp merging flow restriction is similar. In addition, since the optimal control measures predicted by the target agent need to run for a period of time to reflect the control effect, this embodiment uses the total number of vehicles passing through and the congestion index within the prediction time as the reward function. For a single agent with a fixed prediction time, the larger the total number of vehicles passing through, the lower the congestion index, and the better the control effect. Optionally, the reward function can be expressed as:

[0040] (1)

[0041] in, Represents the reward value, Indicates the total number of vehicles passing through. represents the congestion index, and are hyperparameters respectively. Congestion index The calculation formula can be:

[0042] (2)

[0043] The expected speed The calculation is based on the value corresponding to the 85th percentile on the speed cumulative distribution curve under the free flow state of the road, combined with the road grade, design speed and expert scores. For the is the average speed of vehicles passing through the road section, and N is the number of vehicles passing through the road section.

[0044] In a specific embodiment, each agent can use Figure 2In the structure shown, each agent includes an input layer, a feature extraction layer, a Q-value calculation layer, and an output layer. Among them, the input layer (not shown in the figure) is a state variable, the feature extraction layer is used to extract the implicit features of the state variables, the Q-value calculation layer is used to calculate the Q value of each action variable in the action space according to the implicit features, and the output layer (not shown in the figure) is the control measure with the maximum Q value. The structure of the feature extraction layer and the Q-value calculation layer can be flexibly set as needed. Optionally, the feature extraction layer and the Q-value calculation layer can both adopt a fully connected structure, the number of nodes in the feature extraction layer increases layer by layer, and the number of nodes in the Q-value calculation layer gradually decreases until the dimension of the action space.

[0045] Based on the above structure, each agent can be trained in advance in the following way: Taking any agent as an example, first, obtain the current traffic flow data of the controlled section as the state variable ; and generate a control measure in the action space as an action variable Then, the current traffic flow data and the one control measure are input into the simulation model to simulate the traffic flow data of the controlled section after the predicted duration corresponding to the intelligent agent as the new state variable ; Calculate the total number of vehicles passing through and the congestion index within the predicted time according to the simulation results, and obtain the reward function value based on the total number of vehicles passing through and the congestion index ; Then according to the principle of reinforcement learning, use the reward function value Update the Q value of each control measure in the agent:

[0046] (3)

[0047] in, Indicated in the state variable Next action variable The Q value, Indicated in the state variable Next action variable The updated Q value, Represents the new state variable The largest of all Q values, and is a hyperparameter.

[0048] Finally, according to the expected value and actual value The difference between them is used to calculate the loss function and update the network parameters of the agent. After the final Q value converges, the training is completed.

[0049] S120, selecting a target intelligent agent corresponding to a predicted duration, and using the target intelligent agent to predict optimal control measures within the predicted duration in the future based on current traffic flow data of the controlled road section.

[0050] After pre-training, each agent can be used to predict the optimal control measures within the corresponding time. In the first prediction of control measures, you can choose any agent with the prediction time as the target agent to predict the first round of optimal control measures.

[0051] S130. Using a simulation model, simulate the time-varying curve of the traffic flow data of the controlled road section under the optimal control measure, wherein the time-varying curve represents the ideal control state of the optimal control measure.

[0052] The role of the simulation model in this step is to simulate the time-varying curve of the traffic flow data of the controlled road section under the optimal control measures, and provide a reference for the traffic control status at each moment. In the case where the traffic flow prediction model in S110 adopts a simulation model, the simulation model in this step can reuse the above traffic flow simulation model. In the case where the traffic flow measurement model in S110 adopts other neural network models, the simulation model in this step can be a separate simulation model.

[0053] Usually, traffic flow simulation models are calibrated based on historical data, and can take into account some traffic instability factors, such as road randomness, weather randomness, and driver driving randomness, etc. However, the simulation model cannot fully take into account traffic instability factors that have not occurred and unforeseen traffic conditions. Therefore, the time-varying curve simulated in this step can only represent the ideal control state of the current control measures under foreseeable factors, and is also a kind of virtual traffic flow data.

[0054] Furthermore, the time-varying curve of traffic flow data may include a time-varying curve of each traffic flow parameter of each lane, or may include a time-varying curve of an average value of the same traffic flow parameter of multiple lanes, which is not specifically limited in this embodiment.

[0055] S140, using the optimal control measures to actually control the controlled road section, and recording the actual traffic flow data after the control in real time.

[0056] Compared with the virtual traffic flow data obtained in S130, this step actually puts the optimal control measures into operation on the expressway, and obtains the actual traffic flow data after control.

[0057] S150. If the difference between the actual traffic flow data at a certain moment and the time-varying curve exceeds the set range, the actual control duration up to the certain moment is recorded; if the certain moment does not exist, the predicted duration of the target intelligent body is used as the actual control duration.

[0058] This step compares the ideal traffic flow data with the actual traffic flow data in real time. Once it is found that the actual traffic flow data is far from the ideal traffic flow data, the actual control time at this time is recorded, and other control measures are considered to be replaced.

[0059] Optionally, if the difference between the actual traffic flow data at a certain moment and the time-varying curve exceeds the set range, and the difference has not recovered to the set range after the set time, the actual control time up to the certain moment is recorded. If the difference between the actual traffic flow data and the time-varying curve always remains within the set range, the actual control time is the predicted time of the target agent.

[0060] Exemplarily, when the average traffic flow of each lane is taken as the comparison object, if the difference between the actual average traffic flow at a certain moment and the time-varying curve of the average traffic flow exceeds the set range, and the difference has not returned to the set range after a set period of time, the actual control duration up to the certain moment is recorded; if there is no such moment, the predicted duration of the target intelligent body is used as the actual control duration.

[0061] S160. Provide feedback reward to the target intelligent agent according to the actual control time, wherein the feedback reward is used to characterize the anti-interference ability of the target intelligent agent to actual traffic factors, and the longer the actual control time is, the higher the reward value is.

[0062] This step conducts secondary training on the target agent based on the actual control time. The closer the actual control time is to the predicted time, the more compatible the agent's prediction results are with the actual traffic factors, and the better the prediction performance is, so the agent should be rewarded; on the contrary, if the actual control time is too short, it means that the agent's prediction results are seriously interfered by the actual traffic factors, and the prediction effect is unstable, so the agent should be punished.

[0063] Optionally, construct the following reward function to provide feedback reward to the target agent:

[0064] (4)

[0065] in, Represents the reward value, represents the predicted duration of the target agent, Indicates the actual control time of the target agent in this control. When the predicted time of each agent increases in equal steps, You can take the growth step of each prediction duration, such as If the time is 2 hours, 4 hours, and 6 hours respectively, = 2 hours; In addition, It can also be a set duration measurement unit, that is, all predicted durations are in Is a unit of measurement. is the reward value adjustment factor, which is used to adjust the reward value range to cover the positive and negative values. For example, assuming The value range of is [0,5], then You can take 2, so The value range is [-2,3]. is the weight adjustment factor used to adjust the prediction duration and actual control time The weight in the reward value, >1, The larger the value, the longer the actual control time. The greater the weight, the greater the value. Optionally, a can be determined based on experience or by the following formula:

[0066] (5)

[0067] Among them, i represents the index of the predicted duration, Indicates the corresponding time of all predictions The maximum value in Indicates that the absolute value can be Smallest value, is less than Optionally, we can first determine the hyperparameter x and then solve Get the value of a. is a preferred value provided in this embodiment, Can also be replaced by less than Other values ​​of .

[0068] Furthermore, in this function: (negative value), that is, when the predicted time is too short or the actual control time is too short, the target agent is punished. Take the maximum value and hour, (positive value), that is, when the predicted duration is longer and the actual control duration is equal to the predicted duration, the target agent is rewarded the most. Represents the prediction duration The impact on the reward value, the longer the prediction time, the more difficult it is to predict the control measures, and the greater the reward value for accurate control measures. Therefore, Follow increases with the increase of Represents the ratio of actual control time to predicted time The impact on the reward value is that the longer the actual control time is, the greater the reward value the control measure should receive. The solution obtained value, which can ensure that most of the actual control time accounts for The impact of is greater than the impact of the prediction duration, and as As the value of the reward increases, the difference between the two effects gradually decreases; at the same time, the increase rate of the reward value increases with The increase of the reward value slows down as The increase in the actual control time further emphasizes the importance of the proportion of actual control time, which is also consistent with the prediction of control measures. and The optimization rules (such as optimization difficulty) are consistent.

[0069] Reward function value After the calculation is completed, the Q value and model parameters can be updated as the final reward for this regulation. Optionally, the Q value can be updated as follows:

[0070] (6)

[0071] in, represents a hyperparameter, and the meanings of the other parameters are the same as those in formula (3), except that the new state variables are no longer calculated through the simulation model, but are directly derived from the traffic flow parameters on the actual highway.

[0072] It can be seen that the above update is a secondary training of the agent based on the actual control data. Through the training update after each control, the target agent can gradually make up for the difference between the simulation results and the actual traffic status under the corresponding time, and provide a more accurate optimal control strategy in subsequent predictions. For the convenience of distinction and description, r1 in formula (1) is called the first reward value, and r2 in formula (4) is called the second reward value, such as Figure 2 shown.

[0073] S170. According to the actual control time and the accumulated reward value of each agent under the actual control time, reselect the predicted time, and use the agent corresponding to the new predicted time as the new target agent, return to the operation of using the target agent to predict the optimal control measures, and start a new round of prediction.

[0074] After the agent is updated, a new target agent can be selected from the updated agents to restart a new round of prediction. Specifically, depending on the actual control time, the selection of the new target agent can include the following situations:

[0075] Case 1: The actual control duration is less than the predicted duration of the target agent (i.e., the target agent used for this control), indicating that the target agent did not perform sufficient actual control as scheduled. In this case, the next round of predicted duration can be shortened, and the predicted duration closest to the actual control duration is selected as the new predicted duration. The agent corresponding to the new predicted duration is used as the new target agent, and the process returns to S120 to start a new round of prediction of the optimal control measures using the new target agent.

[0076] Case 2: The actual control duration is equal to the predicted duration of the target agent (i.e., the target agent used for this control), indicating that the target agent has carried out sufficient actual control as scheduled. At this time, the current predicted duration can be kept unchanged, or the predicted duration can be appropriately increased. Optionally, if the predicted duration is to be increased, the agent with the highest cumulative reward value under the actual control duration can be selected from other agents with a predicted duration greater than the current target agent as the new target agent. The predicted duration corresponding to the agent is the new predicted duration, and then return to S120 to start a new round of prediction of the optimal control measures using the new target agent. Among them, the cumulative reward value is the accumulation of the reward value r2 obtained by each control calculation. The cumulative reward value reflects both the actual control effect of the control measures and the number of times the agent is selected, which can comprehensively reflect the applicability of a certain agent in actual highway control.

[0077] In addition, since the uncertain interference factors of traffic flow data are an important reason affecting the actual control time, in addition to increasing the prediction time according to the cumulative reward value, the prediction time of the next round can also be corrected according to the volatility of traffic flow data in two adjacent controls. Optionally, if the volatility of traffic flow data in this control is increased compared with the volatility of the previous control, the reselected prediction time can be increased as the final new prediction time; if the volatility of traffic flow data in this control is reduced compared with the volatility of the previous control, the reselected prediction time can be reduced as the final new prediction time. Among them, the volatility of traffic flow data can be reflected by the variance of a certain traffic flow parameter at all actual control moments in a control. The larger the variance, the greater the volatility. Exemplarily, when the average traffic flow of all lanes is used to measure the volatility, the average traffic flow at each actual control moment in a certain control is a data, and the variance of all data items is calculated, which is the volatility of the control.

[0078] In summary, the embodiment of the present invention provides a highway lane control method based on real-time traffic flow, which can achieve the following beneficial effects:

[0079] 1. Since traffic control measures take some time to improve traffic conditions, the agents in this embodiment predict the optimal control measures for a period of time in the future to provide enough time for the control measures to take effect;

[0080] 2. We have built multiple agents with different prediction durations, and reduced or increased the prediction duration for the next round based on the actual control duration of the previous round of predictions and the accumulated rewards of each agent under the control duration, thus achieving adaptive dynamic changes in the prediction duration and improving the accuracy of prediction and control;

[0081] 3. Compare the actual traffic status under control measures with the ideal control status in real time, promptly identify the period when the actual situation deviates from the ideal state and stop the control, and conduct a second feedback training for the agent used according to the actual control duration to enhance the agent's ability to resist interference from actual traffic;

[0082] 4. Provide comprehensive feedback to the agents used based on the predicted duration and the actual control duration, and accurately reflect the influence of the two in the reward feedback by setting a reasonable reward function, so that it is consistent with the action law and optimization law of each parameter in the control measure prediction, thereby gradually making up for the difference between the simulation model and the actual traffic status, and improving the accuracy and anti-interference of the entire control method;

[0083] 5. This embodiment uses the simulation model to perform initial training on each intelligent agent, and performs secondary training on each intelligent agent through the actual control duration. Through the cooperation between the intelligent agent and the simulation model, the difference between the simulation results of different durations and the actual traffic status is gradually eliminated, so that one simulation model can take into account multiple simulation durations, and has a certain fault tolerance for the calculation accuracy of the simulation model, reducing the stringent requirements on the simulation model in terms of simulation accuracy and simulation duration.

[0084] Figure 3 FIG. 1 is a schematic diagram of a highway lane control system based on real-time traffic flow provided by an embodiment of the present invention. Figure 3 As shown, the system includes: multiple intelligent agents based on reinforcement learning, a traffic flow simulation model and a control module.

[0085] Among them, each intelligent agent is used to predict the optimal lane-level control measures for the controlled road section at different time lengths in the future, and different intelligent agents have different corresponding prediction time lengths.

[0086] The control module is used to select a target intelligent agent corresponding to a predicted duration, and use the target intelligent agent to predict the optimal control measures within the predicted duration in the future based on the current traffic flow data of the controlled section.

[0087] The simulation model is used to simulate the time-varying curve of traffic flow data of the controlled road section under the optimal control measure, wherein the time-varying curve represents the ideal control state of the optimal control measure.

[0088] The control module is also used to use the optimal control measures to actually control the controlled road section, and record the actual traffic flow data after the control in real time; if the difference between the actual traffic flow data at a certain moment and the time-varying curve exceeds the set range, record the actual control time up to the certain moment; if the certain moment does not exist, use the predicted time of the target intelligent agent as the actual control time; provide feedback reward to the target intelligent agent based on the actual control time, wherein the feedback reward is used to characterize the anti-interference ability of the target intelligent agent to actual traffic factors, and the longer the actual control time, the higher the reward value; based on the actual control time and the accumulated reward value of each intelligent agent under the actual control time, reselect the predicted time, and use the intelligent agent corresponding to the new predicted time as the new target intelligent agent, return to the operation of using the target intelligent agent to predict the optimal control measures, and start a new round of prediction.

[0089] This embodiment and the above method embodiment are based on the same inventive concept, and any features in the above method embodiment are applicable to this embodiment and have the same beneficial effects, which will not be described in detail here.

[0090] Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention is shown in FIG. Figure 4 As shown, the device includes a processor 60, a memory 61, an input device 62 and an output device 63; the number of processors 60 in the device can be one or more. Figure 4 A processor 60 is taken as an example; the processor 60, the memory 61, the input device 62 and the output device 63 in the device can be connected by a bus or other means. Figure 4 The example of connecting through bus is taken in the following.

[0091] The memory 61 is a computer-readable storage medium that can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the highway lane control method based on real-time traffic flow in the embodiment of the present invention. The processor 60 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 61, that is, realizing the above-mentioned highway lane control method based on real-time traffic flow.

[0092] The memory 61 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 61 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 61 may further include a memory remotely arranged relative to the processor 60, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0093] The input device 62 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. The output device 63 may include a display device such as a display screen.

[0094] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the highway lane control method based on real-time traffic flow of any embodiment.

[0095] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0096] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0097] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0098] Computer program code for performing the operation of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.

Claims

1. A highway lane control method based on real-time traffic flow, characterized in that: include: Obtain the traffic flow simulation model of the controlled road section; According to the simulation model, multiple intelligent agents based on reinforcement learning are constructed; wherein each intelligent agent is used to predict the optimal lane-level control measures of the controlled road section in different future durations, and different intelligent agents correspond to different prediction durations; each intelligent agent uses the traffic flow data of the controlled road section as a state variable, the lane-level control measures within the corresponding prediction duration as an action variable, and the total number of vehicles passing through and the congestion index within the corresponding prediction duration as a reward function, and uses the simulation model to calculate the change of the state variable; Selecting an agent corresponding to a predicted duration as a target agent, and using the target agent to predict optimal control measures within the predicted duration in the future according to current traffic flow data of the controlled road section; Using a simulation model, simulating a time-varying curve of traffic flow data of the controlled road section under the optimal control measure, wherein the time-varying curve represents an ideal control state of the optimal control measure; Using the optimal control measures to actually control the controlled road section, and recording the actual traffic flow data after the control in real time; If the difference between the actual traffic flow data at a certain moment and the time-varying curve exceeds the set range, the actual control duration up to the certain moment is recorded; if the certain moment does not exist, the predicted duration of the target intelligent agent is used as the actual control duration; According to the actual control time, the target intelligent agent is given a feedback reward, wherein the feedback reward is used to characterize the anti-interference ability of the target intelligent agent to actual traffic factors, and the longer the actual control time, the higher the reward value; According to the actual control time and the accumulated reward value of each agent under the actual control time, the predicted time is reselected, and the agent corresponding to the new predicted time is used as the new target agent, and the operation of using the target agent to predict the optimal control measure is returned to start a new round of prediction; Among them, before using the target intelligent agent to predict the optimal control measures within the future prediction time period based on the current traffic flow data of the controlled section, it also includes: obtaining the current traffic flow data of the controlled section; generating a control measure in the action space; inputting the current traffic flow data and the one control measure into the simulation model to simulate the traffic flow data of the controlled section after the prediction time period; calculating the total number of vehicles passing through and the congestion index within the prediction time period according to the simulation results, and calculating the first reward function value according to the total number of vehicles passing through and the congestion index; according to the principle of reinforcement learning, using the first reward function value r1 to update the Q value of each control measure and the agent network parameters in the target intelligent agent.

2. The method according to claim 1, characterized in that The reselecting of the predicted duration according to the actual control duration and the accumulated reward value of each agent under the actual control duration includes: If the actual control duration is less than the predicted duration of the target agent, select the predicted duration closest to the actual control duration as the new predicted duration; If the actual control duration is equal to the predicted duration of the target agent, the predicted duration of the agent with the highest cumulative reward value under the actual control duration is selected from other agents with predicted durations longer than that of the target agent as the new predicted duration.

3. The method according to claim 2, characterized in that After getting the new predicted duration, it also includes: If the volatility of the traffic flow data in this control is higher than that in the previous control, the new prediction duration is increased to serve as the final new prediction duration; If the volatility of the traffic flow data in this control is smaller than that in the previous control, the new prediction duration will be reduced and used as the final new prediction duration.

4. The method according to claim 1, characterized in that: If the difference between the actual traffic flow data at a certain moment and the time-varying curve exceeds a set range, the actual control duration up to the certain moment is recorded, including: If the difference between the actual data of any traffic flow parameter of any lane at a certain moment and the corresponding time-varying curve exceeds the set range, and the difference has not returned to the set range after the set time, the actual control time up to the said certain moment shall be recorded.

5. The method according to claim 1, characterized in that The providing feedback reward to the target agent according to the actual control time includes: According to the following reward function, calculate the second reward function value r2: Among them, t p represents the predicted duration of the target agent, t c represents the actual control time of the optimal control measure provided by the target agent in this control, Δt represents the growth step of the prediction time between agents; δ is the reward value adjustment factor, which is used to adjust the value range of the reward value to a range that can cover positive and negative values; a is the weight adjustment factor, which is used to adjust the prediction time t p and actual control time t c The weight in the reward value, a>1; i represents the index of the prediction duration, Represents all prediction durations t pi Corresponding The maximum value in Indicates that the absolute value can be The smallest a value, x is less than Hyperparameters of The reward function value r2 is used as the final reward for this regulation, and the Q value of each control measure in the target intelligent agent and the intelligent agent network parameters are updated twice.

6. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the highway lane control method based on real-time traffic flow as described in any one of claims 1-5.

7. A highway lane control system based on real-time traffic flow, characterized in that: include: Multiple reinforcement learning-based intelligent agents, traffic flow simulation models, and control modules; Among them, each intelligent agent is used to predict the optimal lane-level control measures for the controlled road section at different time periods in the future, and different intelligent agents have different corresponding prediction time periods; The control module is used to select a target intelligent agent corresponding to a predicted duration, and use the target intelligent agent to predict the optimal control measures within the predicted duration in the future according to the current traffic flow data of the controlled road section; The simulation model is used to simulate the time-varying curve of the traffic flow data of the controlled road section under the optimal control measure, wherein the time-varying curve represents the ideal control state of the optimal control measure; The control module is also used to use the optimal control measures to actually control the controlled road section, and record the actual traffic flow data after control in real time; if the difference between the actual traffic flow data at a certain moment and the time-varying curve exceeds the set range, record the actual control time up to the certain moment; if the certain moment does not exist, use the predicted time of the target intelligent agent as the actual control time; give feedback reward to the target intelligent agent according to the actual control time, wherein the feedback reward is used to characterize the anti-interference ability of the target intelligent agent to actual traffic factors, and the longer the actual control time, the higher the reward value; according to the actual control time and the accumulated reward value of each intelligent agent under the actual control time, reselect the predicted time, and use the intelligent agent corresponding to the new predicted time as the new target intelligent agent, return to the operation of using the target intelligent agent to predict the optimal control measures, and start a new round of prediction; Among them, each intelligent agent is constructed in the following way: a traffic flow simulation model of the controlled road section is obtained; according to the simulation model, multiple intelligent agents based on reinforcement learning are constructed; each intelligent agent uses the traffic flow data of the controlled road section as a state variable, the lane-level control measures within the corresponding prediction time as an action variable, the total number of vehicles passing through and the congestion index within the corresponding prediction time as a reward function, and uses the simulation model to calculate the change of the state variable; Before using the target intelligent agent to predict the optimal control measures within the future prediction time period based on the current traffic flow data of the controlled section, the target intelligent agent is operated as follows: obtaining the current traffic flow data of the controlled section; generating a control measure in the action space; inputting the current traffic flow data and the one control measure into the simulation model to simulate the traffic flow data of the controlled section after the prediction time period; calculating the total number of vehicles passing through and the congestion index within the prediction time period based on the simulation results, and calculating the first reward function value based on the total number of vehicles passing through and the congestion index; and using the first reward function value r1 to update the Q value of each control measure and the agent network parameters in the target intelligent agent according to the principle of reinforcement learning.

Images