Parking space association method, device and medium based on hierarchical reinforcement learning

Abstract

The present invention relates to the field of computer technology applications and specifically provides a parking stand association method, device, and medium based on hierarchical reinforcement learning, including: obtaining grid area information and associated subject information corresponding to a target geographic area; the grid area being the parking stand and the associated subject being the flight; based on the grid area information, dividing all grid areas corresponding to the grid area information into multiple mutually exclusive area groups; based on preset association constraint information, obtaining the associated subjects allowed to be associated with each mutually exclusive area group, and obtaining an initial association feasibility matrix for each mutually exclusive area group; based on the initial association feasibility matrix and preset mutual exclusion constraint information for each mutually exclusive area group, obtaining an associated operation heterogeneous graph for each mutually exclusive area group; and utilizing a hierarchical reinforcement learning model to obtain association results corresponding to each mutually exclusive area group. The present invention can improve parking stand allocation efficiency and docking rate.

Description

Technical Field

[0001] The present invention relates to the field of computer technology applications, and in particular to a parking space association method, device and medium based on hierarchical reinforcement learning. Background Art

[0002] In some application scenarios, it is necessary to associate two given sets of data to obtain the optimal association solution. For example, in the parking stand allocation scenario, it is necessary to assign a suitable parking stand to each flight. Traditional parking stand algorithms mainly include mathematical optimization algorithms and heuristic algorithms. Mathematical optimization algorithms model parking stand allocation as a 0-1 integer optimization problem, optimizing parking stand utilization under constraints such as aircraft model matching, arrival time, and aircraft conflict. Heuristic algorithms determine the solution by exploring the search space. The above two types of optimization algorithms can obtain better results in small and medium-scale scenarios, but their modeling of actual constraints (such as taxiway conflicts and overlapping parking stands) is often relatively simplified and cannot be applied to complex parking stand allocation problems. The main manifestations are: (1) For dynamically changing flights at the same airport, it is necessary to repeatedly experiment to select the appropriate solution strategy, and the solution complexity increases exponentially with the number of parking stands. (2) The parking stand allocation optimization solution of a single airport cannot be quickly migrated to other airports. Although the parking stand allocation problems of many airports are logically similar, each has its own unique rules.

[0003] In recent years, reinforcement learning has become an emerging method in aviation ground dispatching due to its ability to learn decision-making strategies from interactions. Previous research has used deep reinforcement learning to implement gate assignment, employing architectures such as actor-strategy and DQN (Deep Q-Network), achieving certain performance improvements in simulation environments. However, given the vast scale of flight and slot resources and the complex constraints involved, a single reinforcement learning strategy often converges slowly in the global state space and lacks a robust representation of multi-layered decision-making processes. Therefore, implementing hierarchical or multi-agent reinforcement learning in a multi-constrained environment, while simultaneously achieving global planning and local optimization capabilities, has become a key research direction. Summary of the Invention

[0004] In view of the above technical problems, the technical solution adopted by the present invention is:

[0005] According to a first aspect of the present invention, a method for associating parking spaces based on hierarchical reinforcement learning is provided, the method comprising the following steps:

[0006] S100, obtaining grid area information and associated subject information corresponding to the target geographical area; the grid area is a parking space, and the associated subject is a flight.

[0007] S200 , based on the grid area information, dividing all grid areas corresponding to the grid area information into a plurality of mutually exclusive area groups that do not conflict with each other.

[0008] S300 , based on preset association constraint information, obtaining association entities allowed to be associated with each mutually exclusive area group, and obtaining an initial association feasibility matrix for each mutually exclusive area group.

[0009] S400, based on the initial association feasibility matrix of each mutually exclusive region group and preset mutual exclusion constraint information, obtaining an initial association operation heterogeneous graph of each mutually exclusive region group; one node in the initial association operation heterogeneous graph corresponds to one association operation, and any two nodes that do not have a mutually exclusive relationship are connected using an undirected edge.

[0010] S500, based on the grid area information, the associated subject information, the initial associated feasibility matrix and the initial associated operation heterogeneous graph, the trained hierarchical reinforcement learning model is used to obtain the associated operation results corresponding to each mutually exclusive area group, and then obtain the associated operation results corresponding to the target geographical area.

[0011] According to a second aspect of the present invention, an electronic device is provided, comprising a processor and a memory; the processor is configured to execute the steps of the method according to the first aspect of the present invention by calling a program or instruction stored in the memory.

[0012] According to a third aspect of the present invention, there is provided a computer-readable storage medium storing a program or instructions, wherein the program or instructions enable a computer to execute the steps of the method according to the first aspect of the present invention.

[0013] The present invention has at least the following beneficial effects:

[0014] The parking stand association method based on hierarchical reinforcement learning provided by an embodiment of the present invention first obtains multiple mutually exclusive area groups based on grid area data; then, based on the grid area information and the associated subject information, obtains the associated subjects allowed to be associated for each grid area, and generates an initial association feasibility matrix based on the associated subjects allowed to be associated for all grid areas; then, generates an initial association operation heterogeneous graph of the mutually exclusive area groups, and uses a trained hierarchical reinforcement learning model to obtain the association operation results of the mutually exclusive area groups in each association operation heterogeneous graph. The present invention models the grid area association problem as a combinatorial optimization problem on a graph and predicts the association operations of the grid areas through a hierarchical reinforcement learning model. This can help improve the policy optimization performance in large-scale flight scheduling tasks, that is, it can improve the efficiency and accuracy of data association, and can also provide stronger generalization capabilities and task adaptability.

[0015] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 A flowchart of a parking space association method based on hierarchical reinforcement learning provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present invention.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this invention pertains. The terms used herein in the specification of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0020] It should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the steps as sequential processes, many of the steps can be performed in parallel, concurrently, or simultaneously. In addition, the order of the steps can be rearranged. A process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. A process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0021] The embodiment of the present invention provides a parking space association method based on hierarchical reinforcement learning, such as Figure 1 As shown, the method may include the following steps:

[0022] S100: Obtain grid area information and associated subject information corresponding to a target geographical area.

[0023] In an embodiment of the present invention, the target geographic area may be an airport. The grid areas may include a first grid area of ​​a first location type and a second grid area of ​​a second location type. The distance between the grid area of ​​the first location type and the set location is less than or equal to the set distance, and the distance between the grid area of ​​the second location type and the set location is greater than the set distance. The set location is a location where the user can directly access the docked object, such as an airport terminal. In an embodiment of the present invention, the grid areas are parking stands, with the first grid area being a close stand and the second grid area being a remote stand.

[0024] In the embodiment of the present invention, the associated subject is an object that needs to dock in the grid area, specifically a flight.

[0025] In the embodiment of the present invention, the grid area information can be expressed as PR=(PR1, PR2, ..., PR u ,……,PR M ), PR u The information of the u-th grid area, u ranges from 1 to M, and M is the number of grid areas in the target geographical area. u =(N u , L u , C u , I u 、AC u ), where N u is the ID of the u-th grid area, i.e. the position number, L u is the location type identifier of the u-th grid area. If L u =1, indicating that the uth grid area is the first grid area. If L u =0, indicating that the u-th grid area is the second grid area. u is the area model of the u-th grid area, that is, the size of the grid area. u is the area attribute value of the u-th grid area, if I u =1, indicating that the u-th grid area belongs to the first area position, I u =0, indicating that the uth grid area belongs to the second area position. The first area position is the location of the associated entity for docking to perform international tasks, such as an international flight stand, and the second area position is the location of the associated entity for docking to perform domestic tasks, such as a domestic flight stand. AC u The target field corresponding to the associated entity allowed to dock in the u-th grid area. The target field can be a two-character airline code.

[0026] In the embodiment of the present invention, the associated subject information can be expressed as S=(S1, S2, ..., S v ,……,S M ), S vis the information of the vth associated entity, where v ranges from 1 to N, and N is the number of associated entities in the target geographical area. v =(U v ,TY v , AR v , D v , R v , G v , B v , w v , AD v ), U v The ID of the vth associated entity, such as the flight number. v is the model of the vth associated entity. v is the planned arrival time of the vth associated entity, that is, the planned arrival time in the grid area, that is, the planned arrival time. v is the planned departure time of the vth associated subject, that is, the time when it plans to leave the grid area, that is, the planned departure time. v is the task identifier of the vth associated subject, if R v =1, indicating that the vth associated subject is the first task subject for performing international tasks, such as international flights. If R v =0, indicating that the vth associated subject is a second task subject for performing domestic tasks, such as domestic flights. v is the VIP identifier of the vth associated entity, if G v =1, indicating that the vth associated subject is a VIP subject, such as a VIP flight. If G v =0, indicating that the vth associated subject is not a VIP subject. v is the overnight identification of the vth associated subject, if B v =1, indicating that the vth associated subject is an overnight subject, such as an overnight flight. If B v =0, indicating that the vth associated subject is not an overnight subject, that is, it is not an overnight flight. v is the selection weight set of the vth associated subject, including the selection weight of the vth associated subject for different grid areas, that is, the preference of the flight for different parking positions. If a certain associated subject prefers to stop at a certain grid area, the greater the selection weight of the grid area is. The grid area selection weight can be expressed as a number between 0 and 5. v The target field corresponding to the vth associated entity.

[0027] S200 , based on the grid area information, dividing all grid areas corresponding to the grid area information into a plurality of mutually exclusive area groups that do not conflict with each other.

[0028] In this embodiment of the present invention, the same mutually exclusive area group is composed of conflicting grid areas, and there is no conflict between different mutually exclusive area groups, that is, there is no conflict between grid areas in different mutually exclusive area groups. In this embodiment of the present invention, the conflict between grid areas refers to sliding conflict between grid areas.

[0029] In the embodiment of the present invention, two grid areas are associated with the same runway surface and have the same position type, i.e., the close stand group contains close stands, and the far stand group contains far stands. Two grid areas being associated with the same runway surface means that the two grid areas share the same taxiway.

[0030] In an embodiment of the present invention, the existence of a conflict between two grid areas means that there may be a potential taxiing conflict between two associated entities respectively associated with the two grid areas. Taxiing conflicts may include double push-out conflicts, double slide-in conflicts and push-out-slide-in conflicts. Among them, a double push-out conflict refers to when flights parked at two adjacent parking stands have similar departure times, and a conflict may occur during the push-out process. A double slide-in conflict refers to when flights parked at two adjacent parking stands have similar arrival times, and a conflict may occur during the slide-in process. A slide-in push-out conflict refers to when flights parked at two adjacent parking stands have similar arrival times, and a conflict may occur during the slide-in process.

[0031] In an embodiment of the present invention, multiple mutually exclusive area groups may also be obtained based on the aircraft stand distribution map of the target geographical area. Specifically, grid areas belonging to the same taxiway and within a set range may be regarded as a mutually exclusive area group.

[0032] In another embodiment of the present invention, multiple mutually exclusive area groups can be obtained based on a reachability matrix of an initial conflict relationship matrix of a target geographical area, and can be obtained specifically through the following steps:

[0033] S201: Obtain an initial conflict relationship matrix CA of a target geographical area.

[0034] Where A=(P1,P2,……,P m1 ,……,P M ), P m1 The m1th row of data in CA, where the value of m1 ranges from 1 to M. m1 =(p m11 , p m12 ,……,p m1m2 ,……,p m1M ), p m1m2 is the conflict identification value between the m1th grid area and the m2th grid area. If there is a conflict between the m1th grid area and the m2th grid area, p m1m2 =1, otherwise, p m1m2 =0, the value of m2 ranges from 1 to M.

[0035] In this embodiment of the present invention, the initial conflict relationship matrix of the target geographic area represents whether a conflict exists between two adjacent grid areas. This matrix is ​​a pre-determined matrix. Whether a conflict exists between two adjacent grid areas can be determined based on their locations and prior knowledge. Specifically, whether a potential taxiing conflict exists between the two adjacent grid areas is known, i.e., CA is known.

[0036] S202, obtain the first matrix R1 = (CA) 0 ∨ (CA) 1 ∨ (CA) 2 ∨……∨ (CA) logM , ∨ represents the OR operator symbol.

[0037] S203, obtain the second matrix R2=R1ΛR1 T , Λ represents the intersection operator, R1 T is the transposed matrix of R1.

[0038] S204 , obtaining an all-1 matrix in R2 , and taking the grid area corresponding to each obtained all-1 matrix as an initial mutually exclusive area group, to obtain multiple initial mutually exclusive area groups.

[0039] S205 , based on the initial association feasibility matrix, obtaining the number of associated entities allowed to be associated with each initial mutually exclusive region group, and deleting mutually exclusive region groups with zero number of associated entities allowed to be associated from the multiple initial mutually exclusive region groups, to obtain multiple mutually exclusive region groups.

[0040] S300 , based on preset association constraint information, obtaining association entities allowed to be associated with each mutually exclusive area group, and obtaining an initial association feasibility matrix for each mutually exclusive area group.

[0041] In this embodiment of the present invention, preset association constraint information is used to constrain the model, target field, and task identifier corresponding to the associated entities permitted to be associated with a grid area to be identical to the regional model, target field, and regional attribute values ​​of the grid area. Thus, based on the preset association constraint information, matching grid areas and associated entities can be associated to obtain an initial association feasibility matrix, i.e., an initial feasible domain matrix. Those skilled in the art will appreciate that, in the initial association feasibility matrix, the same associated entity may be associated with multiple grid areas, and the same grid area may also be associated with multiple associated entities.

[0042] In an embodiment of the present invention, the initial association feasibility matrix may be a two-dimensional matrix of size N×M. The elements in the initial association feasibility matrix are composed of a first association identifier and a second association identifier. The first association identifier indicates that the corresponding grid area is allowed to be associated with the associated subject, and the second association identifier indicates that the corresponding grid area is not allowed to be associated with the associated subject. In an exemplary embodiment, the first association identifier may be 1 and the second association identifier may be 0. For example, if the element R in the k1th row and k2th column of the initial association feasibility matrix is k1k2 =1, indicating that the k1th association subject is allowed to be associated with the k2th grid area. The value of k1 ranges from 1 to N, and the value of k2 ranges from 1 to M.

[0043] S400, based on the initial association feasibility matrix of each mutually exclusive region group and preset mutual exclusion constraint information, obtaining an initial association operation heterogeneous graph of each mutually exclusive region group; one node in the initial association operation heterogeneous graph corresponds to one association operation, and any two nodes that do not have a mutually exclusive relationship are connected using an undirected edge.

[0044] In the embodiment of the present invention, the preset mutually exclusive constraint information may include a first constraint condition, a second constraint condition, and a third constraint condition.

[0045] In the embodiment of the present invention, the first constraint condition is used to constrain each associated subject to be associated with at most one grid area. The first constraint condition can be expressed as: M k=1 x ik =1, where x ik is the decision variable indicating whether to associate the i-th association subject with the k-th grid area. If x ik =1, indicating that the kth grid area is associated with the i-th associated subject. If x ik =0, indicating that the kth grid area is not associated with the ith association entity. The value of k ranges from 1 to M, and the value of i ranges from 1 to N.

[0046] In this embodiment of the present invention, the second constraint is used to constrain the existence of a minimum buffer time interval between two adjacent associated entities associated with the same grid area. That is, the time interval between the arrival time of the latter flight and the departure time of the previous flight of two adjacent flights assigned to the same parking position should be greater than the minimum buffer time interval to ensure that there is no conflict between the flights.

[0047] The second constraint can be expressed as: ik +x jk ≤2, min (t i d , t j d )-max(t ia , t j a )+t0<0.

[0048] Among them, x jk is the decision variable indicating whether to associate the jth association subject with the kth grid area. If x jk =1, indicating that the jth associated subject is associated with the kth grid area. If x jk =0, indicating that the jth associated subject is not associated with the kth grid area, t i a is the planned arrival time of the i-th associated entity, t i d is the planned departure time of the i-th associated subject, t j a and t j d are the planned arrival time and planned departure time of the j-th associated entity, t0 is the minimum buffer time interval, and the value of j ranges from 1 to N.

[0049] In this embodiment of the present invention, the third constraint is used to ensure that no collisions occur between two entities associated with two adjacent grid areas. Airport ground traffic conditions are complex and unpredictable. When flights at adjacent parking stands arrive and depart close together, collisions are more likely to occur on the same taxiway. Therefore, collisions between flights at adjacent parking stands should be avoided.

[0050] The third constraint can be expressed as: ik +x jg +p kg ≤3,|t i a -t j a |-t0>0,|t i d -t j d |-t0>0,|t i a -t j d |-t0>0,|t i d -t j a |-t0>0. Among them, x jg is the decision variable indicating whether to associate the jth association subject with the gth grid area. If x jg =1, indicating that the jth associated subject is associated with the gth grid area. If x jg=0, indicating that the jth associated subject is not associated with the gth grid area, and the value of g ranges from 1 to M. kg Identify the conflict relationship value between the k-th grid area and the g-th grid area.

[0051] In an embodiment of the present invention, if any two nodes of the initial associated operation heterogeneous graph satisfy the above-mentioned preset constraint mutually exclusive relationship, it means that the two nodes do not have a mutually exclusive relationship. If any two nodes of the initial associated operation heterogeneous graph do not satisfy any constraint condition in the above-mentioned preset constraint mutually exclusive relationship, it means that the two nodes have a mutually exclusive relationship.

[0052] When constructing the initial association operation heterogeneous graph for each mutually exclusive region group, the association operation corresponding to the association identifier value of 1 in the initial association feasibility matrix for that mutually exclusive region group is treated as a node. That is, the number of nodes in the heterogeneous graph corresponding to each mutually exclusive region group is equal to the number of first association identifiers. Next, undirected edges are used to connect two nodes that do not have a mutually exclusive relationship. The node name can be the identifier of the corresponding association operation, which can be composed of the position number of the corresponding grid region and the ID of the associated entity.

[0053] S500, based on the grid area information, the associated subject information, the initial associated feasibility matrix and the initial associated operation heterogeneous graph, the trained hierarchical reinforcement learning model is used to obtain the associated operation results corresponding to each mutually exclusive area group, and then obtain the associated operation results corresponding to the target geographical area.

[0054] The problem of aircraft stand allocation is inherently highly structured and time-dependent: the stand allocation for the current flight directly impacts the feasible domain, taxiing conflicts, and resource utilization of subsequent flights. Therefore, hierarchical reinforcement learning offers a natural advantage. By combining a top-level strategy for allocating stand clusters or blocks of spatial resources with a bottom-level strategy for determining the optimal ordering of multiple flights within a given stand, this effectively mitigates the dimensionality explosion of the action space in global scheduling and improves the overall efficiency of resource allocation.

[0055] In an embodiment of the present invention, a hierarchical reinforcement learning model includes a top-level network and a bottom-level network. The top-level network is configured to obtain, based on currently received concatenated features corresponding to all currently received mutually exclusive region groups, a mutually exclusive region group currently undergoing an association operation, and transmit the result as a target mutually exclusive region group to the bottom-level network. The bottom-level network is configured to obtain, based on currently received fused features corresponding to the target mutually exclusive region group, a result of the association operation corresponding to the target mutually exclusive region group. The current concatenated feature is a feature obtained by concatenating the current first state vectors of all mutually exclusive region groups, and the current fused feature is a feature obtained by fusing the current second state vectors of the target mutually exclusive region group.

[0056] In an embodiment of the present invention, the top-level network may be a DQN architecture, and the bottom-level network may be a TD3 architecture. Specifically, the top-level network may include a main Q network and a target Q network, and the bottom-level network may include a main policy network, a target policy network, a value network module, and a target value network module. The value network module includes a first value network and a second value network, and the target value network module includes a first target value network and a second target value network.

[0057] In this embodiment of the present invention, the top-level network is tasked with selecting a cluster of stands to be scheduled, i.e., a mutually exclusive zone group. The bottom-level network is tasked with allocating specific stands to flights within the selected cluster. The top-level network's state space consists of the first state vectors of all mutually exclusive zone groups, and its action space consists of selecting a mutually exclusive zone group as the target mutually exclusive zone group to initiate an association operation. The bottom-level network's state space consists of the second state vector of the target mutually exclusive zone group, and its action space consists of selecting an association operation from the association operations corresponding to the target mutually exclusive zone group. The immediate reward corresponding to each association operation selected by the bottom-level network satisfies the following conditions:

[0058] r t bot =α×w t +β×n t .

[0059] r t bot is the reward corresponding to the associated operation selected by the underlying network at the current time step t, α and β are weight coefficients, w t The selection weight of the associated subject for the corresponding grid area corresponding to the associated operation selected at the current time step t, n t The number of associated entities associated with the first grid region in the target mutually exclusive region group selected for the current time step t, where t ranges from 1 to T, and T is the total time step.

[0060] When the bottom network completes the association operation of the target mutually exclusive region group, the top network receives the immediate reward: r t top =(r1 bot +……+r t bot +……+r T bot ).

[0061] In the embodiment of the present invention, the current first state vector of each mutually exclusive region group includes the number of grid regions of the mutually exclusive region group, the number of associated entities, and the selection weight and value.

[0062] The selection weight and value of each mutually exclusive region group are obtained through the following steps:

[0063] For any grid area in the mutually exclusive area group, the selection weight of each associated subject in the mutually exclusive area group for the grid area is obtained, and the selection weights of all associated subjects for the grid area are added together to obtain the selection weight sum value corresponding to the grid area.

[0064] The selection weights and values ​​corresponding to all grid areas in the mutually exclusive area group are added together to obtain the selection weight and value of the mutually exclusive area group.

[0065] Specifically, in this embodiment of the present invention, the first state vector of each mutually exclusive region group satisfies the following conditions:

[0066] .

[0067] F1 u is the first state vector of the u-th mutually exclusive region group, u ranges from 1 to C, and C is the number of mutually exclusive region groups; NR u is the number of grid areas in the u-th mutually exclusive area group, NT u is the number of associated entities corresponding to the u-th mutually exclusive region group, w i1k1 is the selection weight of the i1th associated subject in the uth mutually exclusive area group for the k1th grid area, and the value of i1 ranges from 1 to NT u , k1 ranges from 1 to NR u .

[0068] In an embodiment of the present invention, the current second state vector of each mutually exclusive region group includes the grid region ID set in the mutually exclusive region group, the current association feasibility matrix, the current association operation heterogeneous graph, the current association operation mask matrix, the static feature matrix of the association subject, and the static feature matrix of the grid region.

[0069] Furthermore, in this embodiment of the present invention, the second state vector of each mutually exclusive region group satisfies the following conditions:

[0070] F2 u =(LT u , LR u , MR u , E u , X u ,ST u , SR u ).

[0071] Among them, F2 u is the second state vector of the u-th mutually exclusive region group, LT u is the ID set of the associated subject of the u-th mutually exclusive region group, LR u is the grid region ID set of the u-th mutually exclusive region group, MR uis the current feasibility association matrix of the u-th mutually exclusive region group, E u is the current associated operation heterogeneous graph of the u-th mutually exclusive region group, X u is the current association operation mask matrix of the u-th mutually exclusive region group, ST u is the static feature matrix of the associated subject of the u-th mutually exclusive region group, SR u is the static characteristic matrix of the grid area of ​​the u-th mutually exclusive area group.

[0072] In the embodiment of the present invention, the initial value of the current feasibility association matrix of each mutually exclusive region group is the corresponding initial association feasibility matrix, and the initial value of the current association operation heterogeneous graph of each mutually exclusive region group is the corresponding initial association operation heterogeneous graph.

[0073] In this embodiment of the present invention, the current association operation mask matrix for each mutually exclusive region group is a two-dimensional matrix used to represent the currently associated grid regions and associated entities within that mutually exclusive region group. If a value of 0 in a matrix element indicates that the corresponding associated entity and grid region have already been associated, or that the associated entity and grid region do not satisfy the preset mutual exclusion constraint information, the association operation corresponding to that element value will not be selected. If the value is 1, it indicates that the associated entity and grid region have not been associated. The initial value of the current association operation mask matrix is ​​the corresponding initial association feasibility matrix.

[0074] In the embodiment of the present invention, the static features of the associated subject include model, task identifier, target field, grid area occupancy time, etc., and the static features of the grid area may include area model, location type identifier, etc.

[0075] Furthermore, the fusion features of the target mutually exclusive region group are obtained through the following steps:

[0076] S110, using a graph convolutional neural network to obtain the current associated operation heterogeneous graph E of the target mutually exclusive region group T The node features of each node in E T Corresponding node feature FN T .

[0077] In an embodiment of the present invention, the graph convolutional neural network can be a two-layer standard graph convolutional neural network. The input features of the graph convolutional neural network are the initial feature matrix of the received association operation heterogeneous graph. The initial feature matrix of the association operation heterogeneous graph is formed by the initial feature vectors of all nodes. The initial feature vector of each vertex can be obtained by concatenating the static features of the node's corresponding association subject, the static features of the network region, the selection weight, the feasible association identifier, and the association operation mask identifier.

[0078] In an embodiment of the present invention, the feasible association identifier of a node is used to indicate whether the corresponding association subject is allowed to be associated with the grid area in the current state, and the association operation mask identifier of a node is used to indicate whether the corresponding association subject has been associated with the corresponding grid area in the current state.

[0079] It is known to those skilled in the art that any method for obtaining node features of each node using a graph convolutional neural network falls within the scope of protection of the present invention. For example, in one embodiment, the output feature H of the zth layer of the graph convolutional neural network is z for: ,in, is the self-loop weighted adjacency, , A is the adjacency matrix of the associated operation heterogeneous graph. yes The degree matrix, W z is the learnable parameter of the z-th layer, σ() is a nonlinear activation function, which can be a ReLU activation function, the value of z ranges from 1 to L, and L is the number of layers of the graph convolutional neural network.

[0080] S111, the current feasibility association matrix MR of the target mutually exclusive region group T Flatten it and get MR T The corresponding flattened feature MR T f , and the current association operation mask matrix X for the target mutually exclusive region group T Flatten it and get X T The corresponding flattened feature X T f .

[0081] In this embodiment of the present invention, the current association operation mask matrix is ​​a two-dimensional matrix used to represent currently associated grid regions and associated entities. If an element in the matrix is ​​1, it indicates that the corresponding associated entity has been associated with the grid region, that is, the corresponding flight has been assigned to the corresponding stand. If it is 0, it indicates that the corresponding associated entity has not been associated with the grid region. The initial value of the elements in the current association operation mask matrix is ​​0.

[0082] S112, MR T f 、FN T 、X T f 、ST T and SR T Perform splicing to obtain the splicing feature FC of the target mutually exclusive region group T Among them, ST T is the static feature matrix of the associated subject of the target mutually exclusive region group, SR Tis the static characteristic matrix of the grid region of the target mutually exclusive region group.

[0083] S113, using multilayer perceptron to analyze FC T Perform feature extraction to obtain the fusion features of the target mutually exclusive region group.

[0084] In this embodiment of the present invention, the trained hierarchical reinforcement learning model can be obtained by the following steps:

[0085] S1, obtain a sample dataset, including grid area information and associated subject information within a specified geographical area.

[0086] S2, obtain the mutually exclusive region groups corresponding to the sample data set, the initial association feasibility matrix of each mutually exclusive region group, and the initial association operation heterogeneous graph.

[0087] The specific implementation of S2 can refer to the above content.

[0088] S3, set the iteration counter N0=1.

[0089] S4, the current top state s c top Input into the main Q network to obtain the main Q value corresponding to each mutually exclusive region group. The main Q network selects the current state s based on a greedy strategy such as the ε greedy strategy c top Corresponding action a c top , as the current top-level action and sends a control instruction indicating the execution of the associated operation on the current top-level action to the bottom-level network; the current top-level state is the current first state vector of all mutually exclusive region groups in the sample data set, and the top-level action is the selected mutually exclusive region group.

[0090] In an embodiment of the present invention, the main Q value of each mutually exclusive region group can be obtained by using the existing DQN network Q value calculation method. In an exemplary embodiment, the main Q network may include a convolutional feature extraction layer, a feature flattening layer, a fully connected layer, and an output layer. The main Q value corresponding to each mutually exclusive region group is obtained by the following steps:

[0091] (1) Using the convolutional feature extraction layer to extract s c top Perform multiple convolution operations to obtain the corresponding convolution features.

[0092] In this embodiment of the present invention, three convolutional layers can be used to c top Perform three convolution operations. The convolution feature h3 obtained satisfies the following conditions:

[0093] h3=ReLU(Conv3(h2)).

[0094] Among them, ReLU() is the ReLU activation function. Conv3(h2) means that the third convolutional layer is used to perform a convolution operation on the second feature h2, h2=ReLU(Conv2(h1)), Conv2(h1) means that the second convolutional layer is used to perform a convolution operation on the first feature h1, h1=ReLU(Conv1(s c top ))、Conv1(s c top ) indicates that the first convolutional layer is used to c top Perform convolution operation.

[0095] (2) Use the feature flattening layer to flatten the convolutional features to obtain the flattened features.

[0096] (3) Use the fully connected layer to process the flattened features and obtain the corresponding processing results as fully connected features.

[0097] The fully connected feature Fz can be expressed as: Fz=ReLU(W fc f+b fc ).

[0098] Among them, W fc is the learnable weight matrix of the fully connected layer, b fc is the bias term of the fully connected layer, and f is the flattened feature.

[0099] (4) Use the output layer to linearly activate the fully connected features to obtain s c top The corresponding main Q value Q main .

[0100] In the embodiment of the present invention, Q main =W out Fz+b out .W out is the learnable weight matrix of the output layer, b out is the bias term of the output layer. It is known to those skilled in the art that based on Q main The main Q value corresponding to each mutually exclusive region group can be obtained.

[0101] S5: Based on the received control instructions, the bottom layer network performs association operations on the mutually exclusive region group corresponding to the current top layer action, including:

[0102] (1) The current underlying state s c bot Input the main strategy network, get the scores of all nodes in the associated operation heterogeneous graph corresponding to the mutually exclusive region group, and take the node with the largest score as the current underlying action ac bot , and execute the current underlying action to obtain the corresponding immediate reward r c bot and the new underlying state s bot next , get the corresponding underlying quadruple information (s c bot , a c bot , r c bot , s bot next ) and stored in the current underlying experience playback area.

[0103] The current bottom-level state is the current fused feature corresponding to the mutually exclusive region group corresponding to the current top-level action. The new bottom-level state is the state obtained after updating the current association operation heterogeneous graph and the current association operation mask matrix of the mutually exclusive region group.

[0104] In an embodiment of the present invention, the scores of all nodes in the associated operation heterogeneous graph corresponding to each mutually exclusive region group can be obtained based on existing technologies. In an exemplary embodiment, the scores of all nodes in the associated operation heterogeneous graph corresponding to each mutually exclusive region group can meet the following conditions:

[0105] PV = tanh (W a 2 ReLU(W a 1 s c bot +b a 1 ) + b a 2 ).

[0106] PV is the score of all nodes in the associated operation heterogeneous graph corresponding to each mutually exclusive region group, that is, the score set formed by the scores of all nodes. a 1 is the first learnable weight matrix of the main policy network, W a 2 is the second learnable weight matrix of the main policy network, b a 1 The first bias term of the main policy network, b a 2 is the second bias term of the main policy network, and tanh() is the hyperbolic tangent function.

[0107] After executing the current underlying action, the current association operation heterogeneous graph of the mutually exclusive region group is updated, including: 1) deleting the node corresponding to the action and the edges connecting the node from the current association operation heterogeneous graph of the mutually exclusive region group; 2) deleting nodes that are not connected to the node by edges and edges that are not connected to nodes that are not connected by edges to the node; 3) deleting nodes with association entities corresponding to the action. After executing the current underlying action, the current association operation mask matrix of the mutually exclusive region group is updated, including: setting the element value corresponding to the action in the current association operation mask matrix to 0.

[0108] The initial value of the current underlying experience playback area is a preset number of underlying experience quadruple information.

[0109] (2) Obtain a preset number of bottom-level quadruple information from the current bottom-level experience playback area as bottom-level training sample data, input the next state of each bottom-level training sample data into the target strategy network, and obtain the corresponding optimal action a bot ; and input the current state and corresponding action in each underlying quadruple information into the first value network and the second value network to obtain the corresponding first Q value and the second Q value, and input the next state and optimal action corresponding to each quadruple information into the first target value network and the second target value network to obtain the corresponding first target Q value and the second target Q value.

[0110] In an embodiment of the present invention, a method for respectively obtaining a first Q value and a second Q value using a first value network and a second value network based on a current state and a corresponding action may be a prior art method. In an exemplary embodiment, the first Q value and the second Q value satisfy the following conditions:

[0111] Q qj =W qj 2 ReLU(W qj 1 [s c bot , ONa c ]+b qj 1 ) + b qj 2 .

[0112] Among them, Q qj is the Q value of the jth value network, j is 1 or 2, that is, the first value network is the first value network, the second value network is the second value network, Q q1 is the first Q value, Q q2 is the second Q value. qj 2 is the second learnable weight matrix of the j-th value network, W qj1 is the first learnable weight matrix of the j-th value network, b qj 1 is the first bias term of the j-th value network, b qj 2 is the second bias term of the jth value network. c It is the one-hot representation of the current action corresponding to the current state.

[0113] (3) Update the parameters of the first value network and the second value network. Set the parameters of the first value network and the second value network to be updated every c1 steps, that is, every time c1 association operations are performed. c1 can be an empirical value, c1 ≥ 1.

[0114] In an embodiment of the present invention, the parameters of the first value network and the second value network are updated using a gradient descent method. The parameters of the first value network and the second value network are the same, and the loss value L1 used to update the parameters of the first value network or the second value network satisfies the following conditions:

[0115] .

[0116] Among them, Q1 f1 is the first Q value of the f1th training sample data, Q2 f1 is the second Q value of the f1th training sample data, y f1 is the target value of the f1th training sample data, the value of f1 ranges from 1 to X1, X1 is the number of training sample data obtained from the underlying experience playback area, y f1 =r f1 bot +β bot min(Q1 f1 target , Q2 f1 target ). f1 bot is the immediate reward for the f1th training sample data, β bot is the underlying discount factor, Q1 f1 target is the first target Q value of the f1th training sample data, Q2 f1 target is the second target Q value of the f1th training sample data.

[0117] (4) Update the parameters of the main policy network. The parameters of the main policy network are updated d steps after the parameters of the first value network and the second value network are updated. That is, the main policy network is updated d steps after the value network is updated. The method for updating the parameters of the main policy network can adopt the existing update method. d can be an empirical value, d ≥ 1.

[0118] In an embodiment of the present invention, a gradient ascent method is used to update the parameters of the main strategy network.

[0119] (5) Set the target policy network and target value network to be updated every c2 steps.

[0120] In the embodiment of the present invention, a learning rate τ is introduced to perform a weighted average of the old target network parameters and the new corresponding network parameters, and then assign the values ​​to the target policy network and the target value network. c2 is less than c1.

[0121] (6) If the associated operation heterogeneous graph corresponding to the mutually exclusive region group corresponding to the current top-level action is empty, execute S6; otherwise, the new bottom-level state s bot next As the current underlying state, S5 is executed.

[0122] S6, based on the associated operation performed by the bottom network on the current top action, obtain the immediate reward r corresponding to the current top action top and the new top-level state s top next , get the corresponding top-level quadruple information (s c top , a c top , r c top , s top next ) and stored in the current top-level experience playback area. The initial value of the current top-level experience playback area is a preset number of top-level experience quadruple information.

[0123] S7, sample a preset number of top-level quadruple information from the current top-level experience playback area as top-level training sample data, input the next state of each top-level training sample data into the main Q network, and obtain the corresponding optimal action a top ; and input the current state and corresponding action in each top-level quadruple information into the main Q network to obtain the corresponding main Q value, and input the next state and corresponding action corresponding to each top quadruple information into the target Q network to obtain the corresponding target Q value.

[0124] S8, update the parameters of the main Q network using gradient descent. Set the parameters of the target Q network to be updated every set c3 steps.

[0125] In this embodiment of the present invention, the loss value L used to update the parameters of the primary Q network satisfies the following conditions:

[0126] .

[0127] Among them, Q f2 Y is the main Q value of the f2th top-level training sample data, the value of f2 ranges from 1 to X2, and X2 is the number of training sample data sampled from the top-level experience playback area. f2 is the target value of the top-level training sample data of the f2th item, Y f2 =r f2 top +β top maxQ f2 target ). f2 top is the immediate reward for the top-level training sample data of the f2th item, β top is the top discount factor, maxQ f2 target is the maximum value among the target Q values.

[0128] In the embodiment of the present invention, the method for updating the parameters of the target Q network may adopt an existing updating method.

[0129] S9: If all mutually exclusive region groups have performed the association operation, obtain the total reward of the top network and execute S10; otherwise, update the current top state and use the updated top state as the current top state and execute S4.

[0130] In this embodiment of the present invention, updating the current top-level state includes setting the state vector corresponding to the current top-level action in the current top-level state to a preset state vector. The preset state vector is set so that the corresponding primary Q value reaches a minimum, ensuring that these actions are not selected. If all state vectors in the current top-level state are preset state vectors, it indicates that the association operation has been performed on all mutually exclusive region groups. Otherwise, it indicates that the association operation needs to be continued.

[0131] In this embodiment of the present invention, the total reward of the top-level network is the sum of the instant rewards corresponding to all mutually exclusive area groups.

[0132] S10, set N0=N0+1, if N0≤NC, execute S4; if N0>NC, and if the number of the maximum reward in the N0 total rewards NA≥P0, exit the current control program and use the current hierarchical reinforcement learning network as the trained hierarchical reinforcement learning network. If NA<P0, it means that the total reward is still changing and N0 needs to be updated, and execute S4; P0 is the preset number threshold, and NC is the preset number of iterations threshold.

[0133] In this embodiment of the present invention, P0 can be set based on actual needs. In this embodiment of the present invention, the specific operation of updating N0 is: increasing N0. The specific value of the increase can be set based on actual needs. For example, the minimum increase value is set to (P0-NA).

[0134] Furthermore, S500 may specifically include the following steps:

[0135] In step S501, the current top-level state corresponding to the target geographic area is input into the main Q network to obtain the main Q value corresponding to each mutually exclusive area group. The main Q network selects the action corresponding to the current top-level state as the current top-level action based on a greedy strategy, such as the ε-greedy strategy, and sends the mutually exclusive area group corresponding to the current top-level action as the target mutually exclusive area group to the bottom-level network.

[0136] In the embodiment of the present invention, the current top-level state corresponding to the target geographical area is the current splicing features of all mutually exclusive area groups corresponding to the target geographical area, and the top-level action is the selected mutually exclusive area group.

[0137] S502 : The bottom layer network obtains the association operation result corresponding to the target mutually exclusive region group based on the received target mutually exclusive region group and the current bottom layer state, and returns it as the current association operation result to the top layer network.

[0138] In the embodiment of the present invention, the current underlying state is the current fusion feature corresponding to the target mutually exclusive region group.

[0139] S503, based on the current association operation result, the current top state is updated, and the updated top state is used as the current top state. If the state vectors in the current top state are all preset state vectors, execute S504, otherwise, execute S501.

[0140] S504: Obtain the association operation result corresponding to the target geographical area based on all association operation results obtained from the underlying network, that is, obtain an allocation plan for each parking position.

[0141] S502 specifically includes:

[0142] S5021: Input the current bottom layer state into the main strategy network, obtain the scores of all nodes in the associated operation heterogeneous graph corresponding to the target mutually exclusive region group, and use the node with the maximum score as the current bottom layer action.

[0143] S5022: Execute the current bottom-level action to obtain the corresponding immediate reward and new bottom-level state. If the new bottom-level state is not empty, use the new bottom-level state as the current bottom-level state and execute S5021. Otherwise, obtain the associated operation result of the mutually exclusive region group corresponding to the current top-level action and return it to the top-level network.

[0144] The parking stand association method based on hierarchical reinforcement learning provided by an embodiment of the present invention first obtains multiple mutually exclusive area groups based on grid area data; then, based on the grid area information and the associated subject information, obtains the associated subjects allowed to be associated for each grid area, and generates an initial association feasibility matrix based on the associated subjects allowed to be associated for all grid areas; then, generates an initial association operation heterogeneous graph of the mutually exclusive area groups, and uses a trained hierarchical reinforcement learning model to obtain the association operation results of the mutually exclusive area groups in each association operation heterogeneous graph. The present invention models the grid area association problem as a combinatorial optimization problem on a graph and predicts the association operations of the grid areas through a hierarchical reinforcement learning model. This can help improve the policy optimization performance in large-scale flight scheduling tasks, that is, it can improve the efficiency and accuracy of data association, and can also provide stronger generalization capabilities and task adaptability.

[0145] An embodiment of the present invention also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are configured to execute the method described in the embodiment of the present invention.

[0146] An embodiment of the present invention further provides a computer-readable storage medium storing computer-executable instructions, wherein the computer instructions are used to execute the method described in the embodiment of the present invention.

[0147] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in the present invention can be achieved. This is not limited herein.

[0148] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A parking space association method based on hierarchical reinforcement learning, characterized in that: The method comprises the following steps: S100, obtaining grid area information and associated subject information corresponding to the target geographical area; the grid area is a parking space, and the associated subject is a flight; S200, based on the grid area information, dividing all grid areas corresponding to the grid area information into a plurality of mutually exclusive area groups that do not conflict with each other; S300, based on preset association constraint information, obtaining association entities allowed to be associated with each mutually exclusive area group, and obtaining an initial association feasibility matrix for each mutually exclusive area group; S400, based on the initial association feasibility matrix of each mutually exclusive region group and preset mutual exclusion constraint information, obtaining an initial association operation heterogeneous graph for each mutually exclusive region group; each node in the initial association operation heterogeneous graph corresponds to one association operation, and any two nodes that do not have a mutually exclusive relationship are connected using an undirected edge; S500, based on the grid area information, the associated subject information, the initial associated feasibility matrix and the initial associated operation heterogeneous graph, the trained hierarchical reinforcement learning model is used to obtain the associated operation results corresponding to each mutually exclusive area group, and then obtain the associated operation results corresponding to the target geographical area.

2. The method according to claim 1, characterized in that in, The hierarchical reinforcement learning model includes a top-level network and a bottom-level network, wherein the top-level network is used to obtain the mutually exclusive region group currently performing the association operation based on the current splicing features corresponding to all currently received mutually exclusive region groups, as the target mutually exclusive region group and send it to the bottom-level network; the bottom-level network is used to obtain the association operation result corresponding to the target mutually exclusive region group based on the current fusion features corresponding to the currently received target mutually exclusive region group, wherein the current splicing feature is the feature obtained by splicing the current first state vectors of all mutually exclusive region groups, and the current fusion feature is the feature obtained by fusing the current second state vector of the target mutually exclusive region group; wherein the current first state vector of each mutually exclusive region group includes the mutually exclusive region group. The number of grid areas, the number of associated subjects, and the selection weight and value, wherein the selection weight is the selection weight of the associated subject for the grid area; the current second state vector of each mutually exclusive area group includes the grid area ID set, the current association feasibility matrix, the current association operation heterogeneous graph, the current association operation mask matrix, the static feature matrix of the associated subject, and the static feature matrix of the grid area in the mutually exclusive area group; wherein the initial value of the current association feasibility matrix of each mutually exclusive area group is the corresponding initial association feasibility matrix, the initial value of the current association operation heterogeneous graph of each mutually exclusive area group is the corresponding initial association operation heterogeneous graph, and the initial value of the current association operation mask matrix is ​​the corresponding initial association feasibility matrix.

3. The method according to claim 2, characterized in that The selection weight and value of each mutually exclusive region group are obtained by the following steps: For any grid area in the mutually exclusive area group, obtain the selection weight of each associated subject in the mutually exclusive area group for the grid area, and add up the selection weights of all associated subjects for the grid area to obtain the selection weight sum value corresponding to the grid area; The selection weights and values ​​corresponding to all grid areas in the mutually exclusive area group are added together to obtain the selection weight and value of the mutually exclusive area group.

4. The method according to claim 3, characterized in that The fusion features of the target mutually exclusive region group are obtained through the following steps: S10, using graph convolutional neural network to obtain the current associated operation heterogeneous graph E of the target mutually exclusive region group T The node features of each node in E T Corresponding node feature FN T ; S11, the current feasibility association matrix MR of the target mutually exclusive region group T Flatten it and get MR T The corresponding flattened feature MR T f , and the current association operation mask matrix X for the target mutually exclusive region group T Flatten it and get X T The corresponding flattened feature X T f ; S12, MR T f 、FN T 、X T f 、ST T and SR T Perform splicing to obtain the splicing feature FC of the target mutually exclusive region group T Among them, ST T is the static feature matrix of the associated subject of the target mutually exclusive region group, SR T is the static characteristic matrix of the grid area of ​​the target mutually exclusive area group; S13, using multilayer perceptron to analyze FC T Perform feature extraction to obtain the fusion features of the target mutually exclusive region group.

5. The method according to claim 1, wherein The top-level network includes the main Q network and the target Q network, the bottom-level network includes the policy network, the target policy network, the value network module and the target value network module, the value network module includes the first value network and the second value network, and the target value network module includes the first target value network and the second target value network.

6. The method according to claim 5, characterized in that The grid areas include a first grid area belonging to a first location type and a second grid area belonging to a second location type; The immediate reward corresponding to each associated operation selected by the underlying network satisfies the following conditions: r t bot =α×w t +β×n t ; r t bot is the reward corresponding to the associated operation selected by the underlying network at the current time step t, α and β are weight coefficients, w t The selection weight of the associated subject for the corresponding grid area corresponding to the associated operation selected at the current time step t, n t The number of associated bodies associated with the first grid region in the target mutually exclusive region group selected for the current time step t.

7. The method according to claim 5, characterized in that S500 specifically includes the following steps: S501: Input the current top-level state corresponding to the target geographic area into the main Q network to obtain the main Q value corresponding to each mutually exclusive region group. The main Q network selects the action corresponding to the current top-level state as the current top-level action based on a greedy strategy, and sends the mutually exclusive region group corresponding to the current top-level action as the target mutually exclusive region group to the bottom-level network. The current top-level state corresponding to the target geographic area is the current concatenated features of all mutually exclusive region groups corresponding to the target geographic area, and the top-level action is the selected mutually exclusive region group. S502: The bottom layer network obtains, based on the received target mutually exclusive region group and the current bottom layer state, an association operation result corresponding to the target mutually exclusive region group as the current association operation result and returns it to the top layer network; the current bottom layer state corresponding to the target mutually exclusive region group is the current fusion feature of the target mutually exclusive region group; S503: Update the current top-level state based on the current association operation result, and use the updated top-level state as the current top-level state. If the state vectors in the current top-level state are all preset state vectors, execute S504; otherwise, execute S501. S504: Obtain an association operation result corresponding to the target geographical area based on all association operation results obtained from the underlying network.

8. An electronic device, characterized in that: including processor and memory; The processor is configured to execute the steps of the method according to any one of claims 1 to 7 by calling the program or instructions stored in the memory.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium is used to store a program or instruction, and the program or instruction enables a computer to execute the steps of the method according to any one of claims 1 to 7.

Images