Traffic light control model training method, traffic light control method, and electronic device

Abstract

This disclosure provides a training method for a signal light control model, a signal light control method, and an electronic device, all of which relate to the fields of artificial intelligence, particularly deep learning and intelligent transportation. The specific implementation scheme involves: meta-training a first signal light control model based on an offline trajectory dataset for each of multiple source cities to obtain a second signal light control model; obtaining an online trajectory dataset for the target city based on the interaction between the second signal light control model and the target city's traffic system; and fine-tuning the second signal light control model based on the target city's online trajectory dataset to obtain a third signal light control model.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to the fields of deep learning and intelligent transportation. Background Art

[0002] Problems including traffic congestion not only prolong commuters’ travel time but also have an impact on the environment. Therefore, intelligent traffic light control algorithms are becoming an indispensable solution to urban congestion. Summary of the Invention

[0003] The present disclosure provides a training method for a signal light control model, a signal light control method, and an electronic device.

[0004] According to one aspect of the present disclosure, a method for training a traffic light control model is provided, comprising:

[0005] performing meta-training on the first signal light control model based on an offline trajectory dataset of each source city in a plurality of source cities to obtain a second signal light control model;

[0006] Based on the interaction between the second signal light control model and the traffic system of the target city, an online trajectory dataset of the target city is obtained;

[0007] The second traffic light control model is fine-tuned based on the online trajectory dataset of the target city to obtain the third traffic light control model.

[0008] According to another aspect of the present disclosure, a signal light control method is provided, comprising:

[0009] Using a third signal light control model, traffic characteristics and congestion indicators associated with historical time steps in the traffic system are predicted to obtain the current signal light phase; wherein the third signal light control model is trained based on the training method of the above-mentioned signal light control model;

[0010] Control the traffic lights in the traffic system based on the current signal light phase.

[0011] According to another aspect of the present disclosure, a training device for a traffic light control model is provided, comprising:

[0012] a meta-training unit, configured to perform meta-training on the first signal light control model based on an offline trajectory dataset of each source city in a plurality of source cities to obtain a second signal light control model;

[0013] an interaction unit, configured to obtain an online trajectory dataset of the target city based on interaction between the second signal light control model and the traffic system of the target city;

[0014] The fine-tuning unit is used to fine-tune the second signal light control model based on the online trajectory dataset of the target city to obtain a third signal light control model.

[0015] According to another aspect of the present disclosure, there is provided a signal light control device, comprising:

[0016] a prediction unit configured to use a third signal light control model to predict traffic characteristics and congestion indicators associated with historical time steps in the traffic system to obtain a current signal light phase; wherein the third signal light control model is trained by a training device based on the signal light control model;

[0017] A control unit is used to control a signal light in a traffic system based on a current signal light phase.

[0018] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0019] at least one processor; and

[0020] a memory communicatively connected to the at least one processor; wherein,

[0021] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any method in the embodiments of the present disclosure.

[0022] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0023] According to another aspect of the present disclosure, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the computer program implements any one of the methods according to the embodiments of the present disclosure.

[0024] The disclosed embodiment utilizes source city data for meta-training and transfers the trained second signal light control model to the target city for online fine-tuning. This can improve travel efficiency in the target city and resolve congestion in the target city with as little online interaction as possible.

[0025] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings are used to better understand the present invention and do not constitute a limitation of the present invention.

[0027] Figure 1 is a flowchart of a training method for a traffic light control model provided by an embodiment of the present disclosure;

[0028] Figure 2 is a schematic diagram of a masking rule in an application example according to an embodiment of the present disclosure;

[0029] Figure 3 is a schematic diagram of a training method for a traffic light control model according to another application example of an embodiment of the present disclosure;

[0030] Figure 4 is a schematic diagram of processing representation information in another application example according to an embodiment of the present disclosure;

[0031] Figure 5 is a flow chart of a signal light control method provided by an embodiment of the present disclosure;

[0032] Figure 6 is a schematic block diagram of a training device for a traffic light control model provided by an embodiment of the present disclosure;

[0033] Figure 7 is a schematic block diagram of a training device for a traffic light control model provided by another embodiment of the present disclosure;

[0034] Figure 8 is a schematic block diagram of a training device for a traffic light control model provided by another embodiment of the present disclosure;

[0035] Figure 9 is a schematic block diagram of a training device for a traffic light control model provided by another embodiment of the present disclosure;

[0036] Figure 10 is a schematic block diagram of a signal light control device provided by an embodiment of the present disclosure;

[0037] Figure 11 It is a block diagram of an electronic device used to implement the training method of the signal light control model / the signal light control method according to the embodiment of the present disclosure. DETAILED DESCRIPTION

[0038] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0039] In order to facilitate understanding of the training method of the traffic light control model and the traffic light control method provided by the embodiments of the present disclosure, the relevant technologies of the embodiments of the present disclosure are explained below. The following related technologies can be arbitrarily combined with the technical solutions of the embodiments of the present disclosure as optional solutions, and they all fall within the protection scope of the embodiments of the present disclosure.

[0040] In the related art, reinforcement learning (RL)-based traffic signal control (TSC) has been proposed. These methods treat traffic lights (hereinafter referred to as signals) as independent agents and the traffic system as the environment. By leveraging the structural dependencies between signals, they can achieve superior performance compared to traditional rule-based methods. However, due to the nature of learning and updating policies through trial and error, these RL methods require a large number of online exploration attempts to collect sufficient samples to learn the optimal policy. Furthermore, this trial and error approach is extremely risky in practical deployment, as it requires sacrificing congestion or accident rates to achieve good model training results. This safety issue limits their practical application in real-world scenarios. Given these issues, offline reinforcement learning (RL) has recently been considered as an alternative to traffic signal control. Algorithms such as batch reinforcement learning (BRL) and decision sequence modeling aim to learn effective policies from pre-collected trajectory datasets. These methods have demonstrated high performance while reducing the risk of online interaction. However, a major limitation of offline RL is its reliance on data collected from expert models, including records of state, action, and reward trajectories. When considering deploying traffic control methods in new cities, it is impractical to assume the availability of such a comprehensive trajectory dataset.

[0041] In summary, for methods based on online reinforcement learning, although these models have demonstrated excellent performance on multiple datasets, the deployment of such models in the real world is not feasible due to the uncertainty brought about by the trial-and-error process inherent in online reinforcement learning training. The problem with methods based on offline reinforcement learning is that an online expert model needs to be trained in the target city to collect demonstration datasets, which is not feasible for deployment in real-world scenarios. In addition, the performance of such methods is not only limited by the quality of the behavioral policies of the collected data, but is also very sensitive to the differences between the distributions of training and evaluation data. Methods based on offline reinforcement learning-online fine-tuning (Offline2Online, offline to online), as a combination of online reinforcement learning and offline reinforcement learning, although they solve the shortcomings of both, such methods rely on the trajectory data during offline training coming from the same environment as the online fine-tuning stage, and cannot achieve migration across city scenarios.

[0042] Figure 1The training method of the signal light control model provided by an embodiment of the present disclosure is shown. The method can be applied to the training method of the signal light control model, and the device can be deployed in an electronic device. The electronic device is, for example, a single or multi-machine terminal, server or other processing device. Among them, the terminal can be a user equipment (UE) such as a mobile device, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementations, the method can also be implemented by a processor calling a computer-readable instruction stored in a memory. As Figure 1 As shown, the method may include the following steps S110 to S130.

[0043] Step S110 : Based on the offline trajectory dataset of each source city in the plurality of source cities, meta-train the first signal light control model to obtain a second signal light control model.

[0044] Among them, meta-training refers to training based on meta-learning. Meta-learning is usually understood as "learning to learn", which aims to enable the model to learn previous experiences of multiple related tasks (which can be called meta-tasks), and quickly adapt to and learn new tasks with minimal data and calculations. In the embodiment of the present disclosure, the traffic light control for each source city can be regarded as a meta-task, that is, the traffic light control of different source cities is a different meta-task, and the offline trajectory data set corresponding to the source city is used for training. By training the first traffic light control model based on different meta-tasks respectively, the trained second traffic light control model can acquire the ability to learn and adapt to the new task of traffic light control in the target city, so as to facilitate the migration of the signal control strategy to the signal control task of the target city with minimal data and calculations.

[0045] Here, the source city may refer to a city for which trajectory data has been pre-collected. For example, an expert model may be used to collect trajectory data in the source city, and an offline trajectory dataset for the source city may be obtained based on the multiple trajectory data. In the disclosed embodiments, an offline trajectory dataset refers to a trajectory dataset that is pre-collected rather than obtained through online interaction and contains multiple trajectory data. This trajectory data may also be referred to as sample data or training data.

[0046] For example, the trajectory data may include characteristic information and decision information related to signal control. For example, the trajectory data may include characteristic information such as traffic characteristics and congestion indicators in the traffic system, and may also include decision information such as signal light phase.

[0047] For example, for the problem of traffic light control, each traffic light in the traffic system can be regarded as an intelligent agent. The traffic system is the environment in reinforcement learning and can be defined as a Markov Decision Process (MDP), described by a tuple (S, A, P, R, γ), where S is the state space, A represents the action space, P is the transition probability matrix, R and γ represent the reward function and the reward discount factor, respectively. The state space S includes traffic characteristics on different entry and exit sections, the action space A includes all pre-set traffic light phases, and the reward function R includes congestion indicators such as total queue length, import lane queue length, average travel time, etc. Accordingly, the trajectory dataset can contain trajectory data consisting of state s, action a, and reward r in the following format: τ = (s1, a 1, r1,s2,a 2, r 2, …,s T ,a T ,r T ), where T is the given trajectory length, ranging from 1 to the longest trajectory length. Optionally, in order to perform sequence modeling on trajectory data and make the predicted action maximize the subsequent cumulative reward, the concept of target return can also be introduced, that is, It is used to represent the cumulative reward from a specific time step t to the end of the trajectory. Correspondingly, the trajectory data can also be

[0048] In the embodiment of the present disclosure, the first, second and third of the first signal light control model, the second signal light control model and the third signal light control model are used to distinguish signal light control models in different training stages, and do not limit the structure and parameters of the signal light control model.

[0049] Step S120: obtaining an online trajectory dataset of the target city based on the interaction between the second signal light control model and the traffic system of the target city.

[0050] The target city is the city where the traffic light control model is trained and applied in the embodiments of the present disclosure. Alternatively, the target city may be a city for which trajectory data has not been collected in advance. In some embodiments, the target city may also be a city for which the traffic signal structure is unknown.

[0051] It is understandable that the online trajectory data set may include multiple trajectory data, wherein the format of the trajectory data can be set with reference to the above description, which will not be described in detail here.

[0052] In practice, a city's traffic signal structure can include information about traffic lights and connections between them, and the structures vary from city to city. In the disclosed embodiments, after meta-training using data from source cities with different traffic signal structures, the traffic light control model is transferred to a target city with a new traffic signal structure for fine-tuning and application.

[0053] For example, the traffic signal structure diagram of a city can be expressed as G = (V, E, A, X), where V is the set of traffic lights, v i represents the i-th signal light; E represents the set of edges, each edge consists of e ij =(v i ,v j ) means that each edge is determined based on two traffic lights; It is used to represent the connection relationship between traffic lights (i.e. whether traffic lights affect each other). A is the adjacency matrix of graph G, where N = |V|, a ij =1 refers to the signal light v i and v j There is a connection; is the node feature, represented by trajectory data τ, where K is the time step length of the trajectory data. Define M source cities, that is, and a target city G target Among them, the source city has a full amount of offline trajectory data, and the target city has no pre-collected offline trajectory data. In the embodiment of the disclosure, a small amount of online interactive learning strategy can be used with the environment. The goal of the cross-city traffic light control knowledge transfer in the embodiment of the disclosure is to transfer the traffic light control knowledge from G source Available trajectory data and G target A small number of online small samples are used to learn an action prediction model, namely the signal light control model, to generate the maximum G target Signal control actions for traffic efficiency.

[0054] Step S130 : fine-tuning the second signal light control model based on the online trajectory dataset of the target city to obtain a third signal light control model.

[0055] Fine-tuning refers to further adjusting a trained model using a small number of task-specific samples to better align the model output with expectations. In the disclosed embodiment, fine-tuning can be performed on the second traffic light control model obtained through meta-training, using a small amount of training using an online trajectory dataset from a target city to achieve better performance in the target city.

[0056] As previously explained, since the embodiment of the present disclosure obtains the second signal light control model through meta-training, the second signal light control model is capable of learning and adapting to the signal light control of the target city, thereby being able to migrate the signal control strategy to the signal control task of the target city with a minimal amount of data. Unlike traditional meta-learning scenarios, both the training and adaptation stages of traditional meta-learning require offline data. The special feature of the cross-city signal light control task is that only the meta-training stage is completely offline, while the adaptation stage has no ready-made offline trajectory for fine-tuning. Therefore, the present disclosure innovatively proposes a combination of offline meta-training and online fine-tuning, which can achieve performance comparable to online reinforcement learning and offline reinforcement learning methods in related technologies with a minimal number of fine-tuning iterations.

[0057] In summary, the above-mentioned method of the embodiment of the present disclosure, by using the source city data for meta-training and transferring the trained second traffic light control model to the target city for online fine-tuning, can improve the travel efficiency of the target city and solve the congestion in the target city with as little online interaction as possible.

[0058] In some embodiments, the online trajectory dataset may include multiple trajectory data. Taking the first trajectory data among the multiple trajectory data as an example, the first trajectory data may include multiple reinforcement learning triplet data respectively associated with multiple time steps.

[0059] Accordingly, the above step S120, obtaining the online trajectory dataset of the target city based on the interaction between the second signal light control model and the traffic system of the target city, includes:

[0060] In the traffic system, executing the signal light phase associated with the first time step predicted by the second signal light control model to obtain traffic characteristics and congestion indicators associated with the second time step in the traffic system;

[0061] Using the second signal light control model, traffic characteristics and congestion indicators associated with the second time step in the traffic system are predicted to obtain a signal light phase associated with the second time step in the traffic system;

[0062] Reinforcement learning triplet data associated with the second time step in the first trajectory data is obtained based on traffic characteristics, congestion indicators, and signal light phases associated with the second time step in the traffic system.

[0063] Among them, reinforcement learning triplet data can include state information, action information, and reward information in reinforcement learning. Typically, state information is represented by s, action information is represented by a, and reward information is represented by r. It can be understood that according to the above embodiment, traffic characteristics, signal light phases, and congestion indicators associated with the same time step can be used as reinforcement learning state information, action information, and reward information associated with the same time step, respectively. A trajectory data can include reinforcement learning triplet data corresponding to multiple time steps. It can be understood that any reinforcement learning triplet data associated with a specific time step in any trajectory data in the online trajectory dataset can be obtained using a method similar to the above embodiment.

[0064] In the above embodiment, the traffic light is regarded as an intelligent agent and the traffic system is regarded as the environment. By combining the online interaction method of reinforcement learning to obtain the online trajectory dataset required for the meta-learning adaptation stage, a smart traffic light control solution with feasible reinforcement learning can be provided, which is conducive to utilizing the performance of reinforcement learning to improve the performance of traffic light control.

[0065] In some embodiments, using the second signal light control model, predicting the traffic state characteristics and congestion index associated with the second time step in the traffic system to obtain the signal light phase associated with the second time step in the traffic system includes:

[0066] Using the second signal light control model, predicting traffic state characteristics and congestion indicators associated with the second time step in the traffic system, and obtaining a first action probability for each of the plurality of signal light phases;

[0067] Based on the first action probability of each signal light phase, sampling is performed in a plurality of signal light phases to obtain a signal light phase associated with a second time step in the traffic system.

[0068] Among them, the signal light phase includes information on the signal light display color of one or more traffic flows within a signal cycle (a time period), that is, based on the signal light phase of an intersection or a signal light, the driving direction of one or more traffic flows within the signal cycle can be determined. In some descriptions, the green phase can be used to refer to a specific driving direction allowed within a specific signal cycle. Therefore, the decision of signal light control can be understood as controlling the green phase of the next signal cycle or the next time period given the signal light phase of the next signal cycle. Here, the signal cycle refers to the time period in which the traffic flow obtains exactly the same signal light display color at any time, that is, a continuous time sequence of fixed signal light color display. In the embodiment of the present disclosure, a time step associated with a triplet of data can correspond to a signal cycle.

[0069] In the embodiment of the present disclosure, the information that can be predicted by the second signal light control model includes the action probability of each signal light phase in the multiple signal light phases. Here, the action probability refers to the probability of executing the signal light phase in the next time step.

[0070] In related technologies, based on the predicted action probability for each signal light phase, the signal light phase with the highest action probability is selected from multiple signal light phases as the predicted signal light phase for the next time step. However, the above embodiment does not deterministically select the signal light phase corresponding to the highest action probability, but instead samples from a categorical distribution based on the action probability. This design achieves a balance between exploration and exploitation by making high-value signal light phases more likely to be selected while still ensuring exploration of low-value signal light phases. This ensures the quality of the online trajectory dataset, thereby ensuring the effectiveness and efficiency of fine-tuning.

[0071] In some embodiments, the training method of the traffic light control model further includes:

[0072] For each trajectory data in the online trajectory dataset, the average reward of the trajectory data is calculated based on the reward values ​​corresponding to multiple congestion indicators in the trajectory data;

[0073] Based on the average reward of each trajectory data, some trajectory data are deleted from the online trajectory dataset.

[0074] Optionally, an iteratively updated replay buffer can be designed to store the online trajectory dataset. After the second signal light control model interacts with the target city's traffic system to obtain trajectory data, the trajectory data can be stored in the replay buffer as part of the online trajectory dataset. Simultaneously, the average reward can be periodically calculated for the trajectory data in the replay buffer to sort and filter the trajectory data.

[0075] For example, k% of trajectory data with the highest average reward may be retained from the online trajectory dataset, and other trajectory data other than the k% of trajectory data may be deleted, where k may be preset.

[0076] According to the above embodiment, sorting and filtering the trajectory data in the online trajectory dataset can further ensure the quality of the online trajectory dataset, thereby ensuring the effect and efficiency of fine-tuning.

[0077] In some embodiments, step S110, performing meta-training on the first signal light control model based on the offline trajectory dataset of each source city in the plurality of source cities to obtain the second signal light control model, includes:

[0078] Based on the support set in the offline trajectory dataset of each source city, the meta-task parameters of the first traffic light control model are updated respectively;

[0079] Based on the query set in the offline trajectory dataset of each source city, the first traffic light control model after the meta-task parameters are updated is evaluated to obtain the cumulative gradient across the task query set;

[0080] Based on the accumulated gradient, the general model parameters of the first traffic light control model after the meta-task parameters are updated are updated to obtain the second traffic light control model.

[0081] For example, offline trajectory datasets from different source cities are used as training datasets for different meta-tasks. Each meta-task's training dataset can include a support set and a query set. The support set is used to update task parameters, while the query set is primarily used to evaluate the model. In the above embodiment, the accumulated gradients of the query sets across all tasks can be used to train the common model parameters.

[0082] According to the above embodiment, the support set corresponding to each source city is used to update the meta-task parameters of the first traffic light control model, that is, to update the parameters of the meta-task specific to that source city. After updating each meta-task, the accumulated gradients of the cross-task query set are used to perform a cross-task update, that is, to update the common model parameters, thereby obtaining the second traffic light model.

[0083] For example, the meta-training task is The training task for each source city is a meta-task The training dataset of the meta-task includes D S Support set τ spt and D Q query set τ qry ,and (empty set). For tasks Parameters of the task Update by performing several steps of gradient descent on the support set, i.e. Among them, the left side of the equal sign is the updated parameter, the right side of the equal sign is the parameter before updating, α represents the learning rate of the meta-task, Denotes the loss function over the support set. Afterwards, the model is evaluated on the query set, where the accumulated gradients of the query set across all tasks are used to train / update the common model parameters θ: Among them, θ on the left side of the equal sign is the updated parameter, θ on the right side of the equal sign is the parameter before the update, and β represents the learning rate of the general parameter update. represents the loss function based on the cross-task query set.

[0084] According to the above embodiment, the meta-training phase is completed through two-step parameter updating, so that the obtained second traffic light parameters can acquire the ability to learn and adapt to new tasks, which is conducive to improving the online migration effect.

[0085] In some embodiments, based on the support set in the offline trajectory dataset of each source city, the meta-task parameters of the first traffic light control model are updated respectively, including:

[0086] For a first source city among the multiple source cities, using the representation processing module in the first traffic light control model, the second trajectory data included in the support set of the offline trajectory dataset of the first source city is processed to obtain a target representation that includes decision rule information and urban structure information;

[0087] Processing the target representation using a decision module in the first signal light control module to obtain a second action probability for each of the plurality of signal light phases;

[0088] determining an action prediction loss based on a second action probability for each signal light phase and a signal light phase included in the second trajectory data;

[0089] The meta-task parameters of the first signal light control module are updated based on the action prediction loss.

[0090] The above steps describe how to complete the meta-task parameter update for an offline trajectory dataset of one source city (the first source city). It is understood that a similar approach can be used to complete the meta-task parameter update for other source cities. That is, each source city can be traversed and the above steps can be performed on each source city as if it were the first source city.

[0091] According to the above embodiment, the first traffic light control model includes a representation processing module and a decision module. The representation processing module processes the trajectory data in the support set to obtain a target representation containing decision rule information and urban structure information. The decision module uses the target representation to predict the action probability of each traffic light phase. The action prediction loss can then be determined based on the action probability and the traffic light phase in the trajectory data to update the meta-task parameters.

[0092] Since the representation processing in the above embodiment takes into account both decision-making rule information and urban structure information, training can be performed based on the complex dependencies between traffic lights in the city, so that the traffic light control model can well cope with the heterogeneity between the road networks of the source city and the target city, that is, it can achieve good cross-city scenario migration for different environments.

[0093] Exemplarily, the action prediction loss may be determined based on a cross entropy loss function.

[0094] In some embodiments, the characterization processing module processes the second trajectory data in the following manner:

[0095] Using the decision rule extractor obtained through pre-training, the second trajectory data is processed to obtain a meta-knowledge representation of the second trajectory data;

[0096] Determine the road network adjacency matrix of the first source city based on the meta-knowledge representation and the predefined weight matrix;

[0097] Based on the road network adjacency matrix and the second trajectory data, a target representation is obtained.

[0098] In the processing of the above-mentioned characterization processing module, the predefined weight matrix can be a learnable weight matrix, that is, the weight matrix is ​​a part of the parameters updated after the subsequent calculation of the action prediction loss. The weight matrix is ​​used to determine the road network adjacency matrix of the city, and the road network adjacency matrix is ​​used to represent the connection relationship between traffic lights (that is, whether traffic lights affect each other). For example, the road network adjacency matrix It is used to represent the connection relationship between traffic lights (i.e. whether traffic lights affect each other), where N is the number of traffic lights in the city, and the element a in the matrix A is ij =1 refers to the signal light v i and v j There is a connection, a ij =0 means signal light v i and v j There is no connection. In particular, in some descriptions of the embodiments of the present disclosure, for the source city c, the road network adjacency matrix can be expressed as

[0099] For example, for the second trajectory data Meta-knowledge representation can be obtained through decision rule extractor Optionally, the meta-knowledge representation can be further optimized by splicing representations of different modalities, that is, the meta-knowledge representation can also be a node-level meta-knowledge representation In order to express the structural information of different city maps and reduce structural deviation, the embodiment of the present disclosure reconstructs the adjacency relationship based on meta-knowledge representation. Specifically, using To predict each pair of nodes (n i ,n j )∈G c The probability that there is a connection between them. Then, the constructed adjacency matrix can be obtained as follows: in(·) T represents the matrix transpose, is a learnable weight matrix.

[0100] According to the above embodiment, a decision rule extractor is first used to obtain a meta-knowledge representation of trajectory data. This meta-knowledge representation incorporates decision rule knowledge. Based on this meta-knowledge representation, a road network adjacency matrix is ​​determined using a learnable and optimized weight matrix to obtain city structure information. Because this city structure information, namely the road network adjacency matrix, is dependent on decision rules, the target representation derived from this road network adjacency matrix can include both decision rule information and city structure information, improving the model's cross-city transferability.

[0101] In some embodiments, obtaining a target representation based on the road network adjacency matrix and the second trajectory data includes:

[0102] Based on the road network adjacency matrix, the predefined spatial location code and the second trajectory data, an urban structure dependency representation of the second trajectory data is obtained;

[0103] Combining the meta-knowledge representation and the urban structure dependency representation, the target representation is obtained.

[0104] In the above embodiment, a predefined spatial position encoding is first introduced. The spatial position encoding is learnable, that is, the spatial position encoding is a part of the parameters that are updated after the subsequent calculation of the action prediction loss. The spatial position encoding can be expressed as Road network adjacency matrix Initialization, where N is the maximum number of traffic lights in multiple source cities.

[0105] On this basis, based on the road network adjacency matrix, spatial position encoding and second trajectory data, the urban structure dependency representation of the second trajectory data can be obtained. This representation mainly contains urban structure information based on regular knowledge, and then combines the meta-knowledge representation and the urban structure dependency representation to obtain the target representation.

[0106] Since meta-knowledge representation mainly reflects decision-making law knowledge, combining meta-knowledge representation with urban structure dependency representation to obtain target representation can further strengthen the processing of decision-making law knowledge and ensure the rationality of the model's decision-making.

[0107] In some embodiments, obtaining the urban structure-dependent representation of the second trajectory data based on the road network adjacency matrix, the predefined spatial position code, and the second trajectory data may include:

[0108] Obtaining a first coded representation of the second trajectory data based on the road network adjacency matrix, the predefined spatial position code, and the second trajectory data;

[0109] Based on the spatially directed multi-head attention mechanism with residual connection, the first encoded representation is processed to obtain a second encoded representation of the second trajectory data;

[0110] Based on a multi-layer graph convolutional network, the second trajectory data is processed to obtain a spatial aggregation representation of the second trajectory data;

[0111] Based on the second coding representation and the spatial aggregation representation, the urban structure dependence representation is obtained.

[0112] In the above embodiment, optionally, the road network adjacency matrix, the predefined spatial position code, and the second trajectory data (or information representing the second trajectory data, such as a normalized result of the second trajectory data) may be concatenated to obtain a first coded representation of the second trajectory data.

[0113] After obtaining the first encoding representation, a second encoding representation containing the potential spatial dependencies between traffic lights and a static spatial aggregation representation can be obtained through two-way processing. One of the processes is based on a spatially oriented multi-head attention mechanism with residual connections, and the other is completed by a multi-layer graph convolutional network. Specifically, the spatially oriented multi-head attention mechanism with residual connections can be used to capture the potential spatial dependencies between different traffic lights. In parallel, a multi-layer GCN (Graph Convolutional Networks) including a residual network is used to obtain a static spatial period representation. The final urban structure dependency representation is obtained by fusing the information of the two.

[0114] According to the above embodiment, the urban structure dependency representation can obtain dynamic spatiotemporal dependency relationships, overcoming the problem that the traditional spatial information transmission scheme based on GCN directly relies on the adjacency relationship between nodes and cannot effectively obtain dynamic spatiotemporal dependency relationships, thereby improving the effectiveness of urban structure information in the representation of trajectory data.

[0115] The following describes a method for obtaining the decision rule extractor for obtaining meta-knowledge representation. In some embodiments, the method for obtaining the decision rule extractor through pre-training includes:

[0116] For each trajectory data in the offline trajectory dataset of each source city, mask the trajectory data based on a preset masking rule to obtain masked data;

[0117] Predict the masked data based on the autoencoder to obtain a prediction sequence;

[0118] Update the autoencoder based on the predicted sequence;

[0119] When the autoencoder is updated to meet the preset requirements, the encoder part in the autoencoder is used as a decision rule extractor.

[0120] As can be seen, in the above embodiment, sequence modeling is applied to traffic light control. Specifically, a preset masking rule is used to mask some elements in the trajectory data to obtain masked data, which is then encoded and decoded by the autoencoder (including an encoder and a decoder) in the model, in the hope that the autoencoder can output a prediction sequence consistent with the trajectory data. The loss can be calculated based on the predicted sequence and the trajectory data, and then the autoencoder can be updated. Ultimately, the encoder part of the autoencoder can be used as a decision rule extractor.

[0121] Based on the above pre-training method, the decision rule extractor can obtain an effective representation of the input trajectory data. This representation represents the common decision rule knowledge of the source city and can effectively assist in subsequent action prediction.

[0122] In some embodiments, masking the trajectory data based on a preset masking rule includes:

[0123] Randomly select one of the traffic characteristics, signal light phase and congestion index in the trajectory data as the autoregressive modal information, and randomly determine the target time step index within the time step length range of the trajectory data;

[0124] The elements in the autoregressive modal information in the trajectory data whose time step index is greater than the target time step index are masked.

[0125] According to the above embodiment, the masking rule requires that at least one modal element in the masked data be autoregressive. This means that at least one modal element is predicted based on previous elements, while all future elements are masked. This allows sequence modeling to be applied to trajectory data sequences containing heterogeneous modal information, ensuring that the prediction process meets the requirements of actual traffic light control prediction.

[0126] In some embodiments, masking the trajectory data based on a preset masking rule further includes: randomly masking other information in the trajectory data except the autoregressive modal information.

[0127] Exemplarily, random masking may be performed based on a preset masking ratio.

[0128] According to the masking design of the above embodiment, the autoencoder can autoregressively learn the hidden cross-modal temporal correlation.

[0129] Furthermore, in some embodiments, the prediction of masked data can be performed in combination with modal coding and time step coding. Specifically, the prediction of masked data based on the autoencoder to obtain a prediction sequence can include:

[0130] Determining a modal coding sequence corresponding to the masked data according to the modality of each element in the masked data;

[0131] Determine the time step encoding sequence corresponding to the masking data according to the time step corresponding to each element in the masking data;

[0132] Obtaining input information based on the masking data, the modal code sequence, and the time step code sequence;

[0133] Input the input information into the autoencoder and obtain the predicted sequence output by the autoencoder.

[0134] For example, modal codes can be defined for the three modal information of traffic characteristics, signal light phase and congestion index in the trajectory data. In addition, according to the time step sequence, each time step index is assigned a code, so each element in the trajectory data has a corresponding modal code and time step code. Figure 2 For example, the modal code of the traffic feature can be s, the modal code of the signal light phase is a, and the modal code of the congestion index is s; the congestion index r0, traffic feature s0, signal light phase a0 and time step code associated with time step 0 are t0, the congestion index r1, traffic feature s1, signal light phase a1 and time step code associated with time step 1 are t1, and so on.

[0135] For example, after the masked data is mapped to a unified representation space, modal encoding and time step encoding can be added, so that the autoencoder can effectively distinguish and process different elements based on modal information and time step information, thereby improving the prediction effect.

[0136] As can be seen, through the above-mentioned masking and encoding designs, sequence modeling can be effectively applied to the reinforcement learning-based traffic light control algorithm, thereby training a decision rule extractor for extracting the decision rules of traffic light control. In combination with any of the above embodiments, the trajectory data input to the model can be processed into a target representation that integrates decision rule knowledge and urban structure information. This allows the model to make decisions based on decision rule knowledge and urban structure information, which can effectively achieve cross-city migration.

[0137] To facilitate understanding of the technical solution of the embodiment of the present disclosure, an application example of the embodiment of the present disclosure will be provided below in chronological order. Figure 3 This is a schematic diagram of the structure of this application example. Figure 3 As shown, in the embodiment of the present disclosure, the training framework of the traffic light control model includes four parts: trajectory data reconstruction 310, urban structure reconstruction 320, law-structure dependent action prediction 330 and online fine-tuning 340.

[0138] Regarding trajectory data reconstruction 310:

[0139] We design a meta-decision regularity extractor which is pre-trained on trajectory data of source cities and propose a masking pattern specifically designed for heterogeneous modality decision trajectories.

[0140] Specifically, we adopt a random autoregressive masking pattern to align with the offline decision sequence modeling. In particular, we enforce that at least one element of the modality in the masking sequence is autoregressive, which means that the element should be predicted based only on the previous elements, while all future elements are masked. In particular, for each input trajectory τ = (s1, a 1, r1,s2,a 2, r 2, …,s K ,a K ,r K ), s i represents the state (traffic characteristics) at time step i, a i represents the action (signal phase) at time step i, r i represents the reward (congestion indicator) at time step i. We first randomly select a mode m0∈{S,A,R}, and then select a random index for mode m0 For the elements of mode m0, from the time step The starting element is shaded, i.e. At the same time, for the other two modes m1∈{S,A,R} and m2∈{S,A,R}), the trajectory and The autoencoder is randomly masked according to a predefined masking ratio. Based on such a masking design, the autoencoder can autoregressively learn the hidden cross-modal temporal correlations.

[0141] For the decision rule extractor, we adopt a common encoder-decoder architecture, where both parts are bidirectional transformers. diversity (c is used to identify the source city), we first use the fully connected layer f s (·),f a (·),f r (·) These trajectories are mapped into a unified representation space to obtain representations where N is the number of traffic lights in the source city c and D is the hidden dimension of the encoder output representation. To enable the autoencoder to distinguish elements of different modalities at different time steps in the sequence, we superimpose a learnable modality-specific encoding and a sinusoidal timestep encoding on top of the encoding of the input trajectory. The decoder is then trained to reconstruct the original sequence, including the unmasked tokens, using the mean squared error (MSE) as the loss. Therefore, the training objective for trajectory dynamics reconstruction from self-supervised pre-training can be expressed as where τmasked is the masked trajectory sequence, is the prediction sequence. Through this method, the decision rule extractor can obtain an effective representation of the input sequence, which represents the common decision rule knowledge of the source city.

[0142] On the reconstruction of urban structure 320:

[0143] Given an input trajectory of source city c We query the pre-trained autoencoder to extract decision-rule-aware representations, i.e. where D is the hidden dimension of the encoder output representation.

[0144] Then, we construct the node-level meta-knowledge representation by splicing representations of different modalities, namely

[0145] Based on the fact that many existing works have confirmed that structured traffic light communication is a key factor in the field of TSC, in order to express the structural information of different city graphs and reduce structural deviations, we propose to reconstruct the adjacency relationship from node-level meta-knowledge. Specifically, we use To predict each pair of nodes (n i ,n j )∈G c The probability of the existence of edges between them. Then, the constructed adjacency matrix can be obtained as follows: in(·) T represents the matrix transpose, is a learnable weight matrix.

[0146] Based on this, the reconstructed road network structure diagram can be determined, and then the structural perception representation information can be obtained.

[0147] On rule-structure-dependent action prediction 330:

[0148] This part includes the processing of structure-dependent representation and the action prediction process.

[0149] The processing process of structural dependency representation can refer to the following information:

[0150] Traditional spatial information transmission methods are basically based on graph neural networks, such as graph convolutional networks (GCNs). However, graph neural networks directly rely on the adjacency relationship between nodes and cannot effectively obtain the dynamic spatiotemporal dependencies between traffic lights. Therefore, we propose to use transformers to obtain spatial perception representations.

[0151] First, we introduce a learnable spatial position encoding, which is designed specifically for each type of element. The encoding can be expressed as Road network adjacency matrix Initialization, where N is the maximum number of traffic lights in several source cities. These codes are then combined with the latent element representation Concatenated, the encoded representation can be expressed as Where f(·) represents a linear mapping and || represents a concatenation operation.

[0152] On this basis, the processing process refers to Figure 4 As shown in Figure 2, we further utilize a spatially directed multi-head attention mechanism with residual connections to capture the potential spatial dependencies between different traffic lights: where MHA(·) represents the multi-head attention operation.

[0153] In parallel, a multi-layer GCN including a residual network is used to obtain a static spatial aggregation representation. The information transfer of the GCN layer l+1 is as follows: Among them D c It's A c The diagonal matrix, W (l) is the learnable weight matrix and σ is the activation function.

[0154] The spatial aggregation representation output of GCN is defined as The final spatial dependency representation is obtained through the following gating mechanism: in Where ⊙ represents the dot product, on this basis we can adaptively represent the implicit complex dependencies between traffic lights.

[0155] The processing of the action prediction process can refer to the following information:

[0156] Meta-decision rules and representation of structural perception Then through the splicing operation are combined to generate a regularity-structure-assisted representation. This representation is then reconstructed into a sequence of the following format: in

[0157] Given the successful application of Decision Transformer in offline decision sequence modeling, we adopt a similar Transformer structure for causal action prediction. Specifically, we use the decoder architecture to autoregressively predict the action probability at time step t based on the elements from time step 1 to t. Right now

[0158] Due to the discrete nature of actions, we use cross entropy loss as the optimization objective, namely: Where P represents the number of optional actions.

[0159] About Online Fine-tuning 340:

[0160] To address the few-shot adaptation problem in the target city, we propose a training framework derived from model-agnostic meta-learning, which consists of a meta-training process in the source city and an adaptation process in the target city. However, unlike traditional meta-learning scenarios, where both training and adaptation phases require offline data, our task is unique in that only the meta-training phase is fully offline, while the adaptation phase has no existing offline trajectories for fine-tuning.

[0161] To address this problem, we first define each source task Including D S Support set τ spt and D Q query set τ qry ,and For training tasks Task parameters Update by performing several steps of gradient descent on the support set, i.e.

[0162] The model is then evaluated on the query set, where the accumulated gradients of the query set across all tasks are used to train the common model parameters θ:

[0163] Furthermore, in the fine-tuning adaptation phase, in order to ensure fine-tuning in the target city where there is no existing offline trajectory, we designed an iteratively updated replay buffer. It collects trajectories of the target city while interacting with the environment. Well trained on source city trajectories that are different from the target city in data distribution, directly apply Generating actions may lead to suboptimal trajectories and hurt model fine-tuning performance.

[0164] Therefore, we propose a method to determine The exploration strategy on the model is Boltzmann exploration. Specifically, during online fine-tuning and data collection, instead of deterministically selecting the action with the maximum action probability, we use The output action probabilities sample actions from a categorical distribution as follows: in This design achieves a balance between exploration and exploitation by making high-value actions more likely to be selected while still ensuring the exploration of low-value actions.

[0165] In addition, to ensure the quality of target trajectory data, we apply a trajectory filtering scheme to filter the target trajectory data according to the average reward of the sequence. The trajectory ranking in Based on the filtered trajectories, we further Fine-tune the sample in to update the decision model Right now

[0166] It can be seen that the method of the embodiment of the present disclosure uses meta-reinforcement learning to solve cross-city traffic light control. It is trained offline based on pre-collected data in the source city and fine-tuned online in the target city. It can provide an intelligent traffic control solution with feasible reinforcement learning implementation.

[0167] Figure 5 The signal light control method provided by an embodiment of the present disclosure is shown. The method can be applied to a signal light control device, which can be deployed in an electronic device. The electronic device is, for example, a single or multi-machine terminal, server or other processing device. The terminal can be a user device such as a mobile device, PDA, handheld device, computing device, vehicle-mounted device, wearable device, etc. In some possible implementations, the method can also be implemented by a processor calling computer-readable instructions stored in a memory. Figure 5 As shown, the method may include:

[0168] S510: Using a third signal light control model, predict traffic characteristics and congestion indicators associated with historical time steps in the traffic system to obtain a current signal light phase; wherein the third signal light control model is trained based on any of the methods in the aforementioned embodiments;

[0169] S520: Control the traffic lights in the traffic system based on the current traffic light phase.

[0170] Since the third traffic light control model is trained based on any of the methods in the aforementioned embodiments, the traffic light control method can accurately predict the traffic light phase in the target city. Controlling traffic lights based on the traffic light phase can improve travel efficiency in the target city and solve congestion in the target city.

[0171] It should be noted that the third signal light control model used in the signal light control method of the present embodiment is obtained based on the training method of the aforementioned embodiment, and the third signal light control model has the same structure as one or more of the first signal light control model, the second signal light control model, and the third signal light control model in the aforementioned embodiment. For example, the third signal light control model in the signal light control method of the present embodiment may include a characterization processing module and a decision module, wherein the processing of trajectory data in the characterization processing module and the decision module is the same or similar to that in the aforementioned embodiment. The specific implementation can be referred to the aforementioned embodiment and will not be elaborated on here.

[0172] According to an embodiment of the present disclosure, the present disclosure also provides a training device for a traffic light control model. Figure 6 A schematic block diagram of a training device for a signal light control model according to an embodiment of the present disclosure is shown. Figure 6 As shown, the device includes:

[0173] a meta-training unit 610 for performing meta-training on the first signal light control model based on an offline trajectory dataset of each source city in a plurality of source cities to obtain a second signal light control model;

[0174] An interaction unit 620 is configured to obtain an online trajectory dataset of the target city based on interaction between the second signal light control model and the traffic system of the target city;

[0175] The fine-tuning unit 630 is configured to fine-tune the second signal light control model based on the online trajectory dataset of the target city to obtain a third signal light control model.

[0176] In some embodiments, the online trajectory data set may include multiple trajectory data. Taking the first trajectory data among the multiple trajectory data as an example, the first trajectory data may include multiple reinforcement learning triplet data associated with multiple time steps. Figure 7 As shown, the interaction unit 620 includes:

[0177] The traffic execution subunit 710 is configured to execute, in the traffic system, the signal light phase associated with the first time step predicted by the second signal light control model, and obtain traffic characteristics and congestion indicators associated with the second time step in the traffic system;

[0178] The prediction subunit 720 is configured to use the second signal light control model to predict the traffic characteristics and congestion index associated with the second time step in the traffic system, and obtain the signal light phase associated with the second time step in the traffic system;

[0179] The processing subunit 730 is configured to obtain reinforcement learning triplet data associated with the second time step in the first trajectory data in the online trajectory dataset based on the traffic characteristics, congestion index, and signal light phase associated with the second time step in the traffic system.

[0180] In some embodiments, the prediction subunit 720 is further configured to:

[0181] Using the second signal light control model, predicting traffic state characteristics and congestion indicators associated with the second time step in the traffic system, and obtaining a first action probability for each of the plurality of signal light phases;

[0182] Based on the first action probability of each signal light phase, sampling is performed in a plurality of signal light phases to obtain a signal light phase associated with a second time step in the traffic system.

[0183] In some embodiments, as Figure 8 As shown, the training device for the traffic light control model further includes a deletion unit 810, which is used to:

[0184] For each trajectory data in the online trajectory dataset, the average reward of the trajectory data is calculated based on the reward values ​​corresponding to multiple congestion indicators in the trajectory data;

[0185] Based on the average reward of each trajectory data, some trajectory data are deleted in the online trajectory dataset.

[0186] In some embodiments, as Figure 9 As shown, the meta-training unit 610 includes:

[0187] A first updating subunit 910 is configured to update the meta-task parameters of the first traffic light control model based on the support set in the offline trajectory dataset of each source city;

[0188] A model evaluation subunit 920 is configured to evaluate the first traffic light control model after the meta-task parameters are updated based on the query set in the offline trajectory dataset of each source city, and obtain the cumulative gradient across the task query set;

[0189] The second updating subunit 930 is configured to update the general model parameters of the first signal light control model after the meta-task parameters are updated based on the accumulated gradient to obtain a second signal light control model.

[0190] In some embodiments, the first updating subunit 910 is further configured to:

[0191] For a first source city among the multiple source cities, using the representation processing module in the first traffic light control model, the second trajectory data included in the support set of the offline trajectory dataset of the first source city is processed to obtain a target representation that includes decision rule information and urban structure information;

[0192] Processing the target representation using a decision module in the first signal light control module to obtain a second action probability for each of the plurality of signal light phases;

[0193] determining an action prediction loss based on a second action probability for each signal light phase and a signal light phase included in the second trajectory data;

[0194] The meta-task parameters of the first signal light control module are updated based on the action prediction loss.

[0195] In some embodiments, the first updating subunit 910 is further configured to:

[0196] Using the decision rule extractor obtained through pre-training, the second trajectory data is processed to obtain a meta-knowledge representation of the second trajectory data;

[0197] Determine the road network adjacency matrix of the first source city based on the meta-knowledge representation and the predefined weight matrix;

[0198] Based on the road network adjacency matrix and the second trajectory data, a target representation is obtained.

[0199] In some embodiments, the first updating subunit 910 is further configured to:

[0200] Based on the road network adjacency matrix, the predefined spatial location code and the second trajectory data, an urban structure dependency representation of the second trajectory data is obtained;

[0201] Combining the meta-knowledge representation and the urban structure dependency representation, the target representation is obtained.

[0202] In some embodiments, the first updating subunit 910 is further configured to:

[0203] Obtaining a first coded representation of the second trajectory data based on the road network adjacency matrix, the predefined spatial position code, and the second trajectory data;

[0204] Based on the spatially directed multi-head attention mechanism with residual connection, the first encoded representation is processed to obtain a second encoded representation of the second trajectory data;

[0205] Based on a multi-layer graph convolutional network, the second trajectory data is processed to obtain a spatial aggregation representation of the second trajectory data;

[0206] Based on the second coding representation and the spatial aggregation representation, the urban structure dependence representation is obtained.

[0207] In some embodiments, the first updating subunit 910 is further configured to:

[0208] For each trajectory data in the offline trajectory dataset of each source city, mask the trajectory data based on a preset masking rule to obtain masked data;

[0209] Predict the masked data based on the autoencoder to obtain a prediction sequence;

[0210] Update the autoencoder based on the predicted sequence;

[0211] When the autoencoder is updated to meet the preset requirements, the encoder part in the autoencoder is used as a decision rule extractor.

[0212] In some embodiments, the first updating subunit 910 is further configured to:

[0213] Randomly select one of the traffic characteristics, signal light phase and congestion index in the trajectory data as the autoregressive modal information, and randomly determine the target time step index within the time step length range of the trajectory data;

[0214] The elements in the autoregressive modal information in the trajectory data whose time step index is greater than the target time step index are masked.

[0215] In some embodiments, the first updating subunit 910 is further configured to:

[0216] All information except the autoregressive modal information in the trajectory data is randomly masked.

[0217] In some embodiments, the first updating subunit 910 is further configured to:

[0218] Determining a modal coding sequence corresponding to the masked data according to the modality of each element in the masked data;

[0219] Determine the time step encoding sequence corresponding to the masking data according to the time step corresponding to each element in the masking data;

[0220] Obtaining input information based on the masking data, the modal code sequence, and the time step code sequence;

[0221] Input the input information into the autoencoder and obtain the predicted sequence output by the autoencoder.

[0222] According to an embodiment of the present disclosure, the present disclosure further provides a signal light control device, Figure 10 A schematic block diagram of a signal light control device according to an embodiment of the present disclosure is shown. Figure 10As shown, the device includes:

[0223] A prediction unit 1010 is configured to use a third signal light control model to predict traffic characteristics and congestion indicators associated with historical time steps in the traffic system to obtain a current signal light phase; wherein the third signal light control model is obtained by training based on any of the training devices in the aforementioned embodiments;

[0224] The control unit 1020 is configured to control the signal lights in the traffic system based on the current signal light phase.

[0225] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, please refer to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0226] In the technical solutions disclosed herein, the acquisition, storage, and application of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0227] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0228] Figure 11 A schematic block diagram of an example electronic device 1100 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0229] like Figure 11 As shown, the device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded from a storage unit 1108 into a random access memory (RAM) 1103. Various programs and data required for the operation of the device 1100 can also be stored in the RAM 1103. The computing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0230] Various components in device 1100 are connected to I / O interface 1105, including an input unit 1106, such as a keyboard and mouse; an output unit 1107, such as various types of displays and speakers; a storage unit 1108, such as a magnetic disk and optical disk; and a communication unit 1109, such as a network card, a modem, a wireless communication transceiver, etc. Communication unit 1109 allows device 1100 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0231] The computing unit 1101 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1101 performs the various methods and processes described above, such as the signal light control model training method / signal light control method. For example, in some embodiments, the signal light control model training method / signal light control method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 1108. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 1100 via the ROM 1102 and / or the communication unit 1109. When the computer program is loaded into the RAM 1103 and executed by the computing unit 1101, one or more steps of the signal light control model training method / signal light control method described above can be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured to execute the training method of the traffic light control model / the traffic light control method in any other appropriate manner (for example, by means of firmware).

[0232] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system comprising at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0233] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0234] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0235] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0236] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0237] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0238] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0239] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A training method for a traffic light control model, comprising: performing meta-training on the first signal light control model based on an offline trajectory dataset of each source city in a plurality of source cities to obtain a second signal light control model; obtaining an online trajectory dataset of the target city based on the interaction between the second signal light control model and the traffic system of the target city; Fine-tuning the second traffic light control model based on the online trajectory dataset of the target city to obtain a third traffic light control model; The step of performing meta-training on the first signal light control model based on the offline trajectory dataset of each source city in the plurality of source cities to obtain the second signal light control model includes: For a first source city among the multiple source cities, using a pre-trained decision rule extractor, processing second trajectory data included in a support set in an offline trajectory dataset of the first source city to obtain a meta-knowledge representation of the second trajectory data; Determine a road network adjacency matrix of the first source city based on the meta-knowledge representation and a predefined weight matrix; Based on the road network adjacency matrix and the second trajectory data, obtaining a target representation including decision rule information and urban structure information; Processing the target representation using a decision module in the first signal light control module to obtain a second action probability for each of the plurality of signal light phases; determining an action prediction loss based on the second action probability of each signal light phase and the signal light phase included in the second trajectory data; Performing a meta-task parameter update on the first signal light control module based on the action prediction loss.

2. The method according to claim 1, wherein The online trajectory dataset includes first trajectory data, wherein the first trajectory data includes a plurality of reinforcement learning triplet data respectively associated with a plurality of time steps; Obtaining an online trajectory dataset of the target city based on the interaction between the second signal light control model and the traffic system of the target city includes: executing, in the traffic system, a signal light phase associated with the first time step predicted by the second signal light control model to obtain a traffic characteristic and a congestion index associated with the second time step in the traffic system; Using the second signal light control model, predicting traffic characteristics and congestion indicators associated with the second time step in the traffic system to obtain a signal light phase associated with the second time step in the traffic system; Reinforcement learning triplet data associated with the second time step in the first trajectory data is obtained based on traffic characteristics, congestion indicators, and signal light phases associated with the second time step in the traffic system.

3. The method according to claim 2, wherein: The method of using the second signal light control model to predict traffic state characteristics and congestion indicators associated with the second time step in the traffic system to obtain a signal light phase associated with the second time step in the traffic system includes: Using the second signal light control model, predicting traffic state characteristics and congestion indicators associated with the second time step in the traffic system to obtain a first action probability for each of the plurality of signal light phases; Based on the first action probability of each signal light phase, sampling is performed in the plurality of signal light phases to obtain a signal light phase associated with the second time step in the traffic system.

4. The method according to any one of claims 1 to 3, further comprising: For each trajectory data in the online trajectory dataset, calculating an average reward for the trajectory data based on reward values ​​corresponding to multiple congestion indicators in the trajectory data; Based on the average reward of each trajectory data, some trajectory data are deleted from the online trajectory dataset.

5. The method according to any one of claims 1 to 3, wherein The step of performing meta-training on the first signal light control model based on the offline trajectory dataset of each source city in the plurality of source cities to obtain the second signal light control model further includes: Based on the query set in the offline trajectory dataset of each source city, the first traffic light control model after the meta-task parameters are updated is evaluated to obtain the cumulative gradient across the task query set; Based on the accumulated gradient, the general model parameters of the first signal light control model after the meta-task parameters are updated are updated to obtain the second signal light control model.

6. The method according to any one of claims 1 to 3, wherein The obtaining the target representation based on the road network adjacency matrix and the second trajectory data includes: Obtaining an urban structure dependency representation of the second trajectory data based on the road network adjacency matrix, a predefined spatial position code, and the second trajectory data; The target representation is obtained by combining the meta-knowledge representation and the urban structure dependency representation.

7. The method according to claim 6, wherein: The obtaining, based on the road network adjacency matrix, the predefined spatial position code, and the second trajectory data, an urban structure dependency representation of the second trajectory data includes: Obtaining a first coded representation of the second trajectory data based on the road network adjacency matrix, the predefined spatial position code, and the second trajectory data; processing the first encoded representation based on a spatially directed multi-head attention mechanism with residual connections to obtain a second encoded representation of the second trajectory data; processing the second trajectory data based on a multi-layer graph convolutional network to obtain a spatial aggregation representation of the second trajectory data; The urban structure dependency representation is obtained based on the second coding representation and the spatial aggregation representation.

8. The method according to any one of claims 1 to 3, wherein Ways to obtain a decision rule extractor through pre-training include: For each trajectory data in the offline trajectory dataset of each source city, masking the trajectory data based on a preset masking rule to obtain masked data; Predicting the masked data based on the autoencoder to obtain a prediction sequence; updating the autoencoder based on the prediction sequence; When the autoencoder is updated to meet preset requirements, the encoder part in the autoencoder is used as the decision rule extractor.

9. The method according to claim 8, wherein The masking of the trajectory data based on a preset masking rule includes: Randomly selecting one of the traffic characteristics, signal light phase, and congestion index in the trajectory data as autoregressive modal information, and randomly determining a target time step index within a time step length range of the trajectory data; Elements in the autoregressive modal information in the trajectory data whose time step index is greater than the target time step index are masked.

10. The method according to claim 9, wherein: The step of masking the trajectory data based on a preset masking rule further includes: All information except the autoregressive modal information in the trajectory data is randomly masked.

11. The method according to claim 8, wherein The step of predicting the masked data based on the autoencoder to obtain a prediction sequence includes: Determining a modal coding sequence corresponding to the masking data according to the modality of each element in the masking data; Determining a time step encoding sequence corresponding to the masking data according to the time step corresponding to each element in the masking data; obtaining input information based on the masking data, the modality code sequence, and the time step code sequence; The input information is input into the autoencoder to obtain a prediction sequence output by the autoencoder.

12. A signal light control method, comprising: Using a third signal light control model, predicting traffic characteristics and congestion indicators associated with historical time steps in the traffic system to obtain a current signal light phase; wherein the third signal light control model is trained based on the method according to any one of claims 1 to 11; Based on the current signal light phase, a signal light in the traffic system is controlled.

13. A training device for a traffic light control model, comprising: a meta-training unit, configured to perform meta-training on the first signal light control model based on an offline trajectory dataset of each source city in a plurality of source cities to obtain a second signal light control model; an interaction unit, configured to obtain an online trajectory dataset of the target city based on interaction between the second signal light control model and the traffic system of the target city; a fine-tuning unit, configured to fine-tune the second signal light control model based on the online trajectory dataset of the target city to obtain a third signal light control model; The meta-training unit includes a first updating sub-unit, wherein the first updating sub-unit is configured to: For a first source city among the multiple source cities, using a pre-trained decision rule extractor, processing second trajectory data included in a support set in an offline trajectory dataset of the first source city to obtain a meta-knowledge representation of the second trajectory data; Determine a road network adjacency matrix of the first source city based on the meta-knowledge representation and a predefined weight matrix; Based on the road network adjacency matrix and the second trajectory data, obtaining a target representation including decision rule information and urban structure information; Processing the target representation using a decision module in the first signal light control module to obtain a second action probability for each of the plurality of signal light phases; determining an action prediction loss based on the second action probability of each signal light phase and the signal light phase included in the second trajectory data; Performing a meta-task parameter update on the first signal light control module based on the action prediction loss.

14. The device according to claim 13, wherein The online trajectory dataset includes first trajectory data, wherein the first trajectory data includes a plurality of reinforcement learning triplet data respectively associated with a plurality of time steps; The interaction unit includes: a traffic execution subunit, configured to execute, in the traffic system, the signal light phase associated with the first time step predicted by the second signal light control model, and obtain traffic characteristics and congestion indicators associated with the second time step in the traffic system; a prediction subunit, configured to use the second signal light control model to predict the traffic characteristics and congestion index associated with the second time step in the traffic system, and obtain a signal light phase associated with the second time step in the traffic system; The processing subunit is configured to obtain reinforcement learning triplet data associated with the second time step in the first trajectory data based on traffic characteristics, congestion indicators, and signal light phases associated with the second time step in the traffic system.

15. The device according to claim 14, wherein The prediction subunit is further configured to: Using the second signal light control model, predicting traffic state characteristics and congestion indicators associated with the second time step in the traffic system to obtain a first action probability for each of the plurality of signal light phases; Based on the first action probability of each signal light phase, sampling is performed in the plurality of signal light phases to obtain a signal light phase associated with the second time step in the traffic system.

16. The apparatus according to any one of claims 13 to 15, further comprising a deleting unit, wherein the deleting unit is configured to: For each trajectory data in the online trajectory dataset, calculating an average reward for the trajectory data based on reward values ​​corresponding to multiple congestion indicators in the trajectory data; Based on the average reward of each trajectory data, some trajectory data are deleted from the online trajectory dataset.

17. The device according to any one of claims 13 to 15, wherein: The meta-training unit further includes: a model evaluation subunit, configured to evaluate the first traffic light control model after the meta-task parameters are updated based on the query set in the offline trajectory dataset of each source city, and obtain a cumulative gradient across the task query set; The second updating subunit is configured to update the general model parameters of the first signal light control model after the meta-task parameters are updated based on the accumulated gradient to obtain the second signal light control model.

18. The device according to any one of claims 13 to 15, wherein: The first updating subunit is further configured to: Obtaining an urban structure dependency representation of the second trajectory data based on the road network adjacency matrix, a predefined spatial position code, and the second trajectory data; The target representation is obtained by combining the meta-knowledge representation and the urban structure dependency representation.

19. The device according to claim 18, wherein The first updating subunit is further configured to: Obtaining a first coded representation of the second trajectory data based on the road network adjacency matrix, the predefined spatial position code, and the second trajectory data; processing the first encoded representation based on a spatially directed multi-head attention mechanism with residual connections to obtain a second encoded representation of the second trajectory data; processing the second trajectory data based on a multi-layer graph convolutional network to obtain a spatial aggregation representation of the second trajectory data; The urban structure dependency representation is obtained based on the second coding representation and the spatial aggregation representation.

20. The device according to any one of claims 13 to 15, wherein The first updating subunit is further configured to: For each trajectory data in the offline trajectory dataset of each source city, masking the trajectory data based on a preset masking rule to obtain masked data; Predicting the masked data based on the autoencoder to obtain a prediction sequence; updating the autoencoder based on the prediction sequence; When the autoencoder is updated to meet preset requirements, the encoder part in the autoencoder is used as the decision rule extractor.

21. The device according to claim 20, wherein The first updating subunit is further configured to: Randomly selecting one of the traffic characteristics, signal light phase, and congestion index in the trajectory data as autoregressive modal information, and randomly determining a target time step index within a time step length range of the trajectory data; Elements in the autoregressive modal information in the trajectory data whose time step index is greater than the target time step index are masked.

22. The device according to claim 21, wherein The first updating subunit is further configured to: All information except the autoregressive modal information in the trajectory data is randomly masked.

23. The apparatus according to claim 20, wherein The first updating subunit is further configured to: Determining a modal coding sequence corresponding to the masking data according to the modality of each element in the masking data; Determining a time step encoding sequence corresponding to the masking data according to the time step corresponding to each element in the masking data; obtaining input information based on the masking data, the modality code sequence, and the time step code sequence; The input information is input into the autoencoder to obtain a prediction sequence output by the autoencoder.

24. A signal light control device comprising: a prediction unit, configured to use a third signal light control model to predict traffic characteristics and congestion indicators associated with historical time steps in the traffic system to obtain a current signal light phase; wherein the third signal light control model is trained based on the apparatus according to any one of claims 13 to 23; A control unit is configured to control a signal light in the traffic system based on the current signal light phase.

25. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 12.

26. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-12.

27. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 12.

Images