Port container automatic scheduling method based on multi-agent reinforcement learning

Abstract

The invention discloses a port container automatic scheduling method based on multi-agent reinforcement learning, and the method comprises the steps: S1, building a corresponding relation between equipment and agents, and constructing a task set; s2, collecting operation state data, and constructing global and local state vectors; s3, generating a scheduling constraint vector, and cutting actions according to the resource, storage yard and path state to form a feasible action set; s4, on the basis of an improved QPLEX algorithm, constructing an individual value network containing a dump structure, and calculating an individual action value; s5, constructing a joint action value hybrid network, and mixing individual values according to the global state vector to form joint action values; s6, constructing a training sample, differentiating and aggregating instant and delayed return, and updating network parameters; and S7, during online scheduling, selecting an optimal action combination according to the combined action value, and generating and issuing a scheduling instruction. According to the invention, automatic collaborative scheduling of port container operation is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-agent reinforcement learning and port container terminal operation scheduling, and particularly relates to a port container automatic scheduling method based on multi-agent reinforcement learning. BACKGROUND

[0002] With the continuous expansion of port throughput scale and the continuous improvement of container terminal operation automation level, multiple types of devices such as quayside cranes, yard cranes, horizontal transportation devices and yard operation units are collaboratively operated in the same operation system, and the port container scheduling problem presents the characteristics of multiple devices, complex operation links, strong constraint conditions and frequent dynamic changes. In order to improve the port operation efficiency and reduce the device empty running and waiting time, various automatic scheduling methods based on rules, mathematical programming and intelligent algorithms have been proposed in the prior art, among which, with the improvement of computing power, reinforcement learning and multi-agent reinforcement learning methods are gradually introduced into the field of port scheduling for adaptive decision modeling of complex operation processes.

[0003] The existing port container scheduling technology based on reinforcement learning usually models single or a small number of devices as agents, and trains through preset state, action and reward functions to realize the optimization of operation sequence or task allocation. However, in the actual port operation scene, different types of devices have significant differences in operation capacity, operation rhythm and spatial constraints, and the existing methods often only model the yard crane or the transportation device locally, which is difficult to cover the overall collaborative relationship between the quayside crane, the yard crane, the horizontal transportation device and the yard operation unit, resulting in a lack of consistency in scheduling decisions at the system level. In addition, some existing technologies use a fixed time step to collect operation states and update the decision model, which fails to fully reflect the actual operation rhythm dominated by scheduling events such as task arrival and operation completion in port operation, and is likely to cause deviations between state description and real operation process.

[0004] Therefore, how to provide a port container automatic scheduling method based on multi-agent reinforcement learning is a problem that those skilled in the art need to solve. SUMMARY

[0005] One object of the present application is to provide a port container automatic scheduling method based on multi-agent reinforcement learning. The present application builds a multi-agent reinforcement learning scheduling framework based on scheduling event driving, models the quayside crane, the yard crane, the horizontal transportation device and the yard operation unit as agents, introduces structured scheduling constraints to clip the action space on the basis of collaborative expression of global state and local state, and realizes unified evaluation of individual action value and joint action value based on an improved QPLEX; at the same time, the stability and consistency of the scheduling strategy in complex port operation scenarios are improved through the event-level construction of immediate reward and delayed reward.

[0006] According to an embodiment of the present application, a port container automatic scheduling method based on multi-agent reinforcement learning comprises the following steps: S1, obtaining device information in a port operation system, establishing a one-to-one correspondence between the devices and agents, and constructing a container operation task set; S2, collecting port operation state data at the triggering moment of the scheduling event, constructing a global state vector, and generating a local state vector for each agent based on the global state vector; S3, generating a scheduling constraint vector based on the local state vector, and performing feasibility pruning on the action set of each agent according to the resource occupation state, yard capacity state and path state, to form a set of executable actions; S4, constructing an individual action value network with a duplex dueling structure for each agent based on an improved QPLEX algorithm, and calculating individual action values based on the local state vector, the scheduling constraint vector and the set of executable actions; S5, constructing a joint action value hybrid network based on the improved QPLEX algorithm, and performing hybrid operation on the individual action values of each agent according to the global state vector to form a joint action value; S6, constructing a training sample sequence based on the scheduling event, distinguishing and aggregating immediate rewards and delayed rewards, and updating the parameters of the individual action value network and the joint action value hybrid network based on the joint action value; S7, in the online scheduling process, selecting an action combination from the set of executable actions of each agent according to the joint action value at the triggering moment of the scheduling event, generating a scheduling instruction and issuing it to the corresponding port operation device for execution.

[0007] Optionally, the S1 comprises: S11, reading the device basic information of the quay crane, the yard crane, the horizontal transportation device and the yard operation unit from the port operation management system, wherein the device basic information at least includes the device unique identifier, the device type identifier, the device current position, the device operation capability parameter and the device workable time period; S12, classifying the device basic information according to the device type identifier, and dividing the device set corresponding to the same device type into a quay crane device set, a yard crane device set, a horizontal transportation device set and a yard operation unit set respectively; S13, assigning a unique agent identifier to each device in the quay crane device set, the yard crane device set, the horizontal transportation device set and the yard operation unit set, establishing a one-to-one correspondence between the agent identifier and the corresponding device unique identifier, and generating a mapping table of the device type identifier and the agent identifier; S14, reading container operation plan data from a port operation plan system, the container operation plan data at least including container unique identification, operation type identification, starting operation position, target operation position, planned start time and planned completion time; S15, structuring the container operation plan data with the container unique identification as index, generating a container operation task set, and assigning a task identification to each container operation task, and establishing a correlation between the task identification and the corresponding starting operation position, target operation position and time information.

[0008] Optionally, the S2 comprises: S21, collecting port operation state data from a port operation management system and a device control system at a triggered moment of a scheduling event, the port operation state data at least including current operation state of each device, current position of each device, task identification allocated to each device, remaining operation time of each device, and task queue order corresponding to each container operation task; S22, performing unified time alignment processing on the collected port operation state data, mapping state data of different sources to the same scheduling event moment to form a set of original state data corresponding to the scheduling event; S23, constructing a global state vector based on the set of original state data, the global state vector at least including a device occupation state vector, a task queue state vector, a yard block state vector and a channel state vector; S24, extracting a corresponding local state vector for each agent based on the global state vector and the correspondence between the device and the agent, the local state vector at least including operation state, current position, allocated task information, reachable yard block state and related channel state of the corresponding device; S25, combining the local state vectors of each agent in a predetermined order to form a state input sequence corresponding to each agent.

[0009] Optionally, the S3 comprises: S31, generating a scheduling constraint vector according to the constraint component construction method based on the local state vector of each agent; S32, generating a resource occupation constraint component according to the operation state, allocated task identification and remaining operation time of the corresponding device; S33, generating a yard capacity constraint component according to the reachable yard block state and yard block remaining capacity information recorded in the local state vector; S34, generating a path constraint component according to the channel state information recorded in the local state vector; S35, combine the resource occupation constraint component, the yard capacity constraint component and the path constraint component according to a preset order to form a scheduling constraint vector corresponding to the agent; S36, construct an action set for each agent, the action set including a job task selection action and a job order adjustment action; S37, based on the scheduling constraint vector, check each action in the action set of each agent item by item, eliminate actions that do not satisfy the resource occupation constraint component, the yard capacity constraint component or the path constraint component, and form a corresponding actionable action set for each agent.

[0010] Optionally, the S4 includes: S41, construct an individual action value network for each agent, the individual action value network including a shared feature extraction layer, a base value item branch and a dominant item branch, the base value item branch and the dominant item branch adopting a duplex dueling structure; S42, group the local state vector according to a preset field order to form a device state field group, a task field group, a yard field group and a channel field group, and perform a numerical dimension normalization process on each field group, the numerical dimension normalization process including performing linear scaling on a count type field, performing binaryzation on a mark type field and performing discrete grid number coding on a position type field; S43, split the scheduling constraint vector into components according to constraint sources to form a resource occupation constraint subvector, a yard capacity constraint subvector and a path constraint subvector, and sequentially splice the resource occupation constraint subvector, the yard capacity constraint subvector and the path constraint subvector to form a constraint embedding vector; S44, establish an action index set based on the actionable action set, and construct an action mask vector, the action mask vector marking actions in the actionable action set as valid marker values and marking actions outside the actionable action set as invalid marker values; S45, provide the field-grouped and encoded local state vector and the constraint embedding vector to the shared feature extraction layer to output a shared feature vector; S46, in the base value item branch, map the shared feature vector to a base value item scalar; S47, in the dominant item branch, combine the shared feature vector and an action embedding vector corresponding to the action index set to respectively generate a resource occupation dominant component, a yard capacity dominant component and a path dominant component, and perform weighted summation on the resource occupation dominant component, the yard capacity dominant component and the path dominant component according to preset weight coefficients to form a dominant item corresponding to the action; S48, performing set normalization processing on the advantage item, calculating an arithmetic mean of the advantage item within the set of actionable actions, and subtracting the arithmetic mean from the advantage item of each action within the set of actionable actions; S49, synthesizing the base value item and the advantage item after set normalization processing according to the synthesis rule of the duplex dueling structure, assigning the synthesis result corresponding to the non-actionable action to a preset invalid marker value according to the action mask vector, and outputting the individual action value of each agent on the set of actionable actions.

[0011] Optionally, the S5 comprises: S51, constructing a joint action value hybrid network, the joint action value hybrid network comprising a hybrid feature generation layer and a joint value output layer; S52, grouping the global state vector according to a preset field order to form a device operation field group, a task distribution field group, a yard state field group, and a channel state field group, and performing value dimension normalization processing on each field group; S53, inputting the global state vector after field grouping encoding into the hybrid feature generation layer to generate a hybrid weight parameter set, the hybrid weight parameter set comprising weight parameters corresponding one-to-one to individual action values of each agent; S54, performing linear weighted combination operation on individual action values corresponding to each agent according to the arrangement order of the hybrid weight parameter set to generate a joint action value intermediate result; S55, performing bias item superposition processing on the joint action value intermediate result to form a joint action value; S56, associating and storing the joint action value and the corresponding agent action combination at the triggering moment of the scheduling event.

[0012] Optionally, the S6 comprises: S61, recording the corresponding global state vector, local state vector of each agent, action index executed by each agent, and recording the global state vector corresponding to the previous scheduling event before the occurrence of the scheduling event at each triggering moment of the scheduling event; S62, performing field-level comparison calculation on the device job state marker, task queue sorting position, and resource occupation marker based on the global state vector and the local state vector corresponding to the adjacent two scheduling events to form a state change amount set corresponding to a single scheduling event; S63, performing numerical mapping processing on the device job completion marker change, task queue sorting change, and resource occupation state change in the state change amount set according to a preset reward mapping rule to generate an instant reward data; S64. According to the order of occurrence of scheduling events, the set of state change quantities corresponding to multiple consecutive scheduling events is accumulated. The accumulation process includes accumulating the changes in the number of yard blocks occupied, the changes in the channel occupancy status, and the changes in equipment waiting time for each event to generate delay report data. S65. Combine the global state vector, the local state vectors of each agent, the action index of each agent, the instantaneous reward data, the delayed reward data, and the global state vector corresponding to the next scheduling event in chronological order to construct a training sample sequence. S66. Based on the training sample sequence, update the parameters of the individual action value network and the joint action value hybrid network according to the preset training order.

[0013] Optionally, S7 includes: S71. At the time the scheduling event is triggered, obtain the global state vector corresponding to the current scheduling event and the local state vector corresponding to each agent. S72. Based on the set of possible actions of each agent, call the individual action value network to calculate the individual action value of each agent on the corresponding set of possible actions. S73. Input the individual action value of each agent into the joint action value hybrid network, and calculate the joint action value corresponding to the action combination of each agent. S74. Select the action combination with the largest value from the joint action value, and determine the action to be executed by each agent under the current scheduling event. S75. Convert the execution action into equipment control instructions, wherein the equipment control instructions consist of job task assignment instructions and job sequence adjustment instructions; S76. Send the equipment control command to the corresponding quay crane, yard crane, horizontal transport equipment and yard operation unit, and record the correlation between the executed action and the corresponding scheduling event.

[0014] The beneficial effects of this invention are: The application introduces a multi-agent reinforcement learning framework based on scheduling event driving, aiming at the problems of complex multi-type device collaborative scheduling process, strong constraint conditions and frequent state changes in the port container operation system, constructs a unified multi-agent modeling system covering shore cranes, yard cranes, horizontal transportation devices and yard operation units, establishes a one-to-one correspondence between the device and the agent at the device level, and performs time alignment processing on multi-source operation state data at the triggering moment of the scheduling event to form a collaborative expression of the global state vector and the local state vector, so that the scheduling decision process is consistent with the actual operation rhythm of the port. In the decision generation stage, the application generates a structured scheduling constraint vector by component modeling of resource occupation state, yard capacity state and channel state, and performs feasibility pruning on the action set in the action generation stage to avoid non-actionable actions entering the value evaluation process, thereby reducing the interference of invalid decisions on model training and online scheduling from the source. In the action value calculation process, the application constructs an individual action value network and a joint action value hybrid network with an improved QPLEX structure under the multi-agent value function decomposition framework, introduces a base value item and an advantage item branch with a duplex dueling structure in the individual network, and embeds constraint information such as resource occupation, yard capacity and path state into the advantage item component for weighted combination, so that the individual action value can reflect the influence of the device state and the constraint condition on the decision; at the joint decision level, the hybrid weight parameter is generated by the global state vector, the individual action value of each agent is weighted and combined to form a joint action value, and the overall benefit of the multi-device collaborative execution of the same scheduling scheme is evaluated. In the training stage, the application constructs a training sample sequence based on the scheduling event, generates immediate reward data by field-level comparison of adjacent scheduling event states, and accumulates the changes of yard occupation, channel occupation and device waiting time across multiple scheduling events to generate delayed reward data, so as to depict the immediate influence of single scheduling decision and the cumulative change of long-term running state in the training process, thereby improving the stability and consistency of the strategy in complex port operation scenarios. Through the above technical means, the application realizes the collaborative unification of state modeling, constraint processing, joint decision and training update in the port container automatic scheduling process. BRIEF DESCRIPTION OF DRAWINGS

[0015] The accompanying drawings are included to provide a further understanding of the application, and constitute a part of the specification, illustrate the application together with the embodiments of the application, and do not constitute a limitation on the application. In the drawings:

[0016] Fig. 1 is a general flowchart of a port container automatic scheduling method based on multi-agent reinforcement learning proposed by the application; Fig. 2is an individual action value network structure schematic diagram of an improved QPLEX algorithm in the application; Fig. 3 is a flow schematic diagram of constructing a training sample sequence based on a scheduling event and distinguishing immediate return and delayed return in the application; DETAILED DESCRIPTION

[0017] The application will now be described in further detail with reference to the drawings. These drawings are simplified schematic diagrams and only show the basic structure of the application in a schematic manner, and therefore only show the components related to the application.

[0018] REFERENCE Figs. 1-3 A port container automatic scheduling method based on multi-agent reinforcement learning, comprising the following steps: S1, obtaining device information in a port operation system, establishing a one-to-one correspondence between the device and the agent, and constructing a container operation task set; S2, collecting port operation state data at the triggering time of the scheduling event, constructing a global state vector, and generating a local state vector for each agent based on the global state vector; S3, generating a scheduling constraint vector based on the local state vector, and performing feasibility pruning on the action set of each agent according to the resource occupation state, the yard capacity state and the path state, to form a feasible action set; S4, constructing an individual action value network with a duplex dueling structure for each agent based on the improved QPLEX algorithm, and calculating the individual action value based on the local state vector, the scheduling constraint vector and the feasible action set; S5, constructing a joint action value hybrid network based on the improved QPLEX algorithm, and performing hybrid operation on the individual action value of each agent according to the global state vector to form a joint action value; S6, constructing a training sample sequence based on the scheduling event, distinguishing and aggregating immediate return and delayed return, and updating the parameters of the individual action value network and the joint action value hybrid network based on the joint action value; S7, in the online scheduling process, selecting an action combination from the feasible action set of each agent according to the joint action value at the triggering time of the scheduling event, generating a scheduling instruction and issuing it to the corresponding port operation device for execution.

[0019] The present application is directed to the characteristics of port container operation, such as difficulty in multi-type equipment collaborative scheduling, complex constraints and frequent dynamic changes of operation state. In the multi-agent reinforcement learning scheduling framework, systematic improvements are made. First, the scheduling event is taken as the basic triggering unit of decision and training, the multi-source operation state data is time-aligned, the collaborative expression of global state vector and local state vector is constructed, and the scheduling model is consistent with the actual operation rhythm of the port. Second, the operation constraints such as resource occupation, yard capacity and channel state are modeled by components, a scheduling constraint vector is formed, and the action set is trimmed for feasibility in the action generation stage, and the inaction is excluded from the decision space. Third, in the individual action value calculation, an improved QPLEX network with duplex dueling structure is introduced, the constraint information is embedded in the advantage component and weighted combination, the mixed weight parameter is generated by global state driving in joint decision level, and the unified evaluation of multi-agent action value is realized. Finally, the training sample sequence is constructed based on the immediate reward and delayed reward of the scheduling event, so as to depict the influence of single scheduling decision and cross-event cumulative change at the same time, thereby improving the stability and consistency of the scheduling strategy in the complex port operation scene.

[0020] In the embodiment, the S1 comprises: S11, reading the device basic information of the quay crane, the yard crane, the horizontal transportation device and the yard operation unit from the port operation management system, wherein the device basic information at least comprises the device unique identifier, the device type identifier, the device current position, the device operation capability parameter and the device workable time period; S12, classifying the device basic information according to the device type identifier, and dividing the device set corresponding to the same device type into the quay crane device set, the yard crane device set, the horizontal transportation device set and the yard operation unit set respectively; S13, assigning a unique agent identifier to each device in the quay crane device set, the yard crane device set, the horizontal transportation device set and the yard operation unit set, establishing a one-to-one correspondence between the agent identifier and the corresponding device unique identifier, and generating a mapping table of the device type identifier and the agent identifier; S14, reading the container operation plan data from the port operation plan system, wherein the container operation plan data at least comprises the container unique identifier, the operation type identifier, the starting operation position, the target operation position, the planned start time and the planned completion time; S15, taking the container unique identifier as the index, structuring the container operation plan data, generating the container operation task set, and assigning a task identifier to each container operation task, and establishing an association relationship between the task identifier and the corresponding starting operation position, target operation position and time information.

[0021] In this embodiment, the S2 comprises: S21, collecting port operation state data from the port operation management system and the equipment control system at the triggering moment of the scheduling event, the port operation state data at least including current operation state of each equipment, current position of each equipment, assigned task identification of each equipment, remaining operation time of each equipment, and task queue order corresponding to each container operation task; S22, performing unified time alignment processing on the collected port operation state data, mapping state data of different sources to the same scheduling event moment to form an original state data set corresponding to the scheduling event; S23, constructing a global state vector based on the original state data set, the global state vector at least including an equipment occupation state vector, a task queue state vector, a yard block state vector, and a channel state vector; The equipment occupation state vector is used to indicate that each equipment is in an operation state or an idle state at the scheduling event moment; the task queue state vector is used to indicate the sorting position of each operation task in the corresponding equipment task queue; the yard block state vector is used to indicate the occupation number and the remaining capacity of each yard operation unit; and the channel state vector is used to indicate the occupation mark and the passage restriction mark of the port operation channel; S24, extracting a corresponding local state vector for each agent based on the global state vector and the correspondence between the equipment and the agent, the local state vector at least including the operation state, the current position, the assigned task information, the reachable yard block state, and the related channel state of the corresponding equipment; S25, combining the local state vectors of each agent in a predetermined order to form a state input sequence corresponding to each agent one by one.

[0022] In the state construction process, the scheduling event is taken as the state update boundary, the state data from the port operation management system and the equipment control system is uniformly time-labeled, and the state data is aligned according to the scheduling event occurrence order, so that the state inconsistency problem caused by the difference in sampling frequency of different systems is eliminated. The global state vector adopts a componentized organization mode, the equipment occupation state, the task queue order, the yard block occupation number, and the channel occupation mark are respectively encoded as independent state components, and are combined in a predetermined order to form a unified state expression. On this basis, according to the correspondence between the equipment and the agent, the state components related to a single equipment are intercepted from the global state vector to constitute the local state vector of the corresponding agent, so that the local state contains the running information directly related to the scheduling decision while maintaining the stability of the dimension.

[0023] In this embodiment, the S3 comprises: S31, generating a scheduling constraint vector according to a constraint component construction mode based on the local state vector corresponding to each agent; S32, generating a resource occupation constraint component according to the working state of the corresponding device, the allocated task identifier and the remaining working time; S33, generating a yard capacity constraint component according to the reachable yard block state and the remaining capacity information of the yard block recorded in the local state vector; S34, generating a path constraint component according to the channel state information recorded in the local state vector; S35, combining the resource occupation constraint component, the yard capacity constraint component and the path constraint component in a predetermined order to form the scheduling constraint vector corresponding to the agent; S36, constructing an action set for each agent, the action set including a working task selection action and a working order adjustment action; S37, based on the scheduling constraint vector, checking each action set of the agent item by item, eliminating actions that do not satisfy the resource occupation constraint component, the yard capacity constraint component or the path constraint component, and forming a corresponding actionable action set for each agent.

[0024] In the present application, the construction of the scheduling constraint vector is based on the multi-source running information in the local state vector, and the constraint components are organized according to the constraint sources. The resource occupation constraint is generated by analyzing the current working state of the device, the task binding relationship and the remaining working time field; the yard capacity constraint is generated by reading the occupation number and capacity upper limit field of the reachable yard block; the path constraint is generated by analyzing the channel occupation mark and the passage restriction mark field. The above constraint components are spliced in a fixed order to form a scheduling constraint vector with stable structure. After the action set is generated, it is checked item by item according to the scheduling constraint vector, and the actions that do not satisfy any constraint component are eliminated, so that an actionable action set consistent with the current running state is formed before the scheduling decision.

[0025] In the present embodiment, the S4 comprises: S41, constructing an individual action value network for each agent, the individual action value network including a shared feature extraction layer, a base value branch and an advantage value branch, the base value branch and the advantage value branch adopting a duplex dueling structure; S42, grouping the local state vector according to a predetermined field order to form a device state field group, a task field group, a yard field group and a channel field group, and performing a numerical dimension consistency processing on each field group, the numerical dimension consistency processing including performing linear scaling on the count type field, performing binaryzation on the mark type field and performing discrete grid number coding on the position type field; S43, componentally split the scheduling constraint vector according to constraint sources to form resource occupation constraint sub-vector, yard capacity constraint sub-vector, path constraint sub-vector, and sequentially splice the resource occupation constraint sub-vector, yard capacity constraint sub-vector, path constraint sub-vector to form a constraint embedded vector; S44, establish an action index set based on the set of actionable actions, and construct an action mask vector, the action mask vector marking actions in the set of actionable actions as valid marker values and marking actions outside the set of actionable actions as invalid marker values; S45, provide the field-grouped encoded local state vector and the constraint embedded vector to a shared feature extraction layer, and output a shared feature vector; S46, in the base value item branch, map the shared feature vector to a base value item scalar; S47, in the advantage item branch, combine the shared feature vector with the action embedded vector corresponding to the action index set to respectively generate resource occupation advantage components, yard capacity advantage components and path advantage components, and weight sum the resource occupation advantage components, yard capacity advantage components and path advantage components according to preset weight coefficients to form action corresponding advantage items; S48, perform set normalization processing on the advantage items, calculate the arithmetic mean of the advantage items within the set of actionable actions, and subtract the arithmetic mean from the advantage items of each action within the set of actionable actions; S49, according to the synthesis rule of the duplex dueling structure, synthesize the base value item and the set normalized advantage item, and according to the action mask vector, assign the synthesis result corresponding to the unactionable action to a preset invalid marker value, and output the individual action value of each agent within the set of actionable actions.

[0026] In the present application, the individual action value network is constructed in a branched structure. The local state vector is field-grouped according to device state, task information, yard state and channel state before entering the network, and is subjected to scaling, binarization and discrete number coding processing respectively for different field types, so that the data of different sources are consistent in numerical scale and representation form. The scheduling constraint vector is split into multiple constraint sub-vectors according to constraint sources before entering the network, and is sequentially spliced to form a unified constraint embedded vector, so that the constraint information participates in value calculation in a structured form. In the advantage item branch, the advantage components corresponding to different constraint sources are calculated respectively and combined through trainable weights, so that the action value reflects the influence of resource occupation, yard capacity and path state at the same time. The unactionable actions are numerically marked through the action mask to ensure that the action value only participates in subsequent processing within the range of actionable actions.

[0027] In the present embodiment, the S5 comprises: S51, construct a joint action value hybrid network, the joint action value hybrid network includes a hybrid feature generation layer and a joint value output layer; S52, field grouping is carried out to the global state vector according to preset field order, device operation field group, task distribution field group, yard state field group and channel state field group are formed, and dimension uniformization processing is carried out to each field group; S53, the global state vector encoded by field grouping is input to the hybrid feature generation layer, a hybrid weight parameter set is generated, the hybrid weight parameter set includes weight parameters corresponding to individual action values of each agent one by one; S54, according to the arrangement order of the hybrid weight parameter set, linear weighted combination operation is carried out to individual action values corresponding to each agent, and a joint action value intermediate result is generated; S55, bias item superposition processing is carried out to the joint action value intermediate result, and a joint action value is formed; S56, the joint action value is associatedly stored with corresponding agent action combination at the triggering moment of the scheduling event.

[0028] In the application, the construction of the joint action value is generated according to the global state vector, the global state vector is field grouped and value is uniformized, so that the hybrid network directly perceives the device operation distribution, the task allocation situation, the yard load state and the channel occupation in the parameter generation stage. The hybrid feature generation layer outputs weight parameters corresponding to individual action values of each agent according to the encoded global state vector, so that the dynamic association between the weight parameters and the system running state is maintained in the joint value calculation process. Individual action values of each agent are linearly combined according to the weight parameter order, and a bias item is added to generate a joint action value, so that the joint action value simultaneously contains individual decision information and global running state information in the value composition, and the joint value calculation process has a stable data dependent structure.

[0029] In the embodiment, the S6 includes: S61, at each triggering moment of the scheduling event, the corresponding global state vector, the local state vector of each agent, the action index executed by each agent are recorded, and the global state vector corresponding to the previous scheduling event before the scheduling event is recorded; S62, based on the global state vectors and local state vectors corresponding to adjacent two scheduling events, field-level comparison calculation is carried out to device job state marks, task queue sorting positions and resource occupation marks, and a state change set corresponding to a single scheduling event is formed; S63, according to the preset return mapping rule, the state change set of the device operation completion mark change, the task queue sorting change and the resource occupation state change are carried out numerical mapping processing, and instant return data is generated; S64, according to the scheduling event occurrence order, the state change set corresponding to a plurality of continuous scheduling events is accumulated, and the accumulation processing includes event-by-event accumulation of the yard block occupation number change, the channel occupation state change and the device waiting time change, and delay return data is generated; S65, the global state vector, the local state vector of each agent, the action index of each agent, the instant return data, the delay return data and the global state vector corresponding to the next scheduling event are combined in time sequence, and a training sample sequence is constructed; S66, based on the training sample sequence, the individual action value network and the joint action value mixed network are executed parameter update according to the preset training sequence.

[0030] In the present application, the construction of the training sample takes the scheduling event as the time boundary, and the port operation state before and after each scheduling event is recorded in pairs. By comparing the state fields corresponding to adjacent scheduling events field by field, the change information of the device operation state, the task queue order and the resource occupation state is extracted, and the change information is converted into instant return data according to the preset mapping rule. In view of the characteristics that the state influence in port operation has hysteresis, a cross-event accumulation mechanism is introduced in the training sample construction process, the yard block occupation change, the channel occupation change and the device waiting time change in a plurality of continuous scheduling events are accumulated event by event, and delay return data is formed. The instant return data and the delay return data are retained in the same training sample at the same time, so that the sample contains single event change information and multi-event cumulative change information in data structure at the same time.

[0031] In the present embodiment, the S7 comprises: S71, at the triggering moment of the scheduling event, the global state vector corresponding to the current scheduling event and the local state vector corresponding to each agent are obtained; S72, based on the set of available actions of each agent, the individual action value network is called to calculate the individual action value of each agent on the corresponding set of available actions; S73, the individual action value of each agent is input into the joint action value mixed network to calculate the joint action value corresponding to each agent action combination; S74, the action combination with the maximum value in the joint action value is selected to determine the execution action of each agent at the current scheduling event; S75, the execution action is converted into a device control instruction, and the device control instruction is composed of a job task assignment instruction and a job order adjustment instruction; S76, the device control instruction is issued to the corresponding shore crane, yard crane, horizontal transportation device and yard operation unit, and the association between the execution action and the corresponding scheduling event is recorded.

[0032] In the online scheduling phase of the present application, the scheduling decision is updated with the scheduling event as the trigger node. The system synchronously acquires the global state vector and the local state vector of each agent when the scheduling event is triggered, and completes the action value calculation within the action space defined by the set of actionable actions. The individual action value is calculated by the action value network corresponding to each agent, and then uniformly input into the joint action value hybrid network for combined processing, so that the calculation process of the joint action value reflects both the individual decision result and the global running state. The determination of the action combination is completed by comparing the numerical values of the joint action values, avoiding exhaustive search of the action space. The selected action combination is further mapped into the control instruction format corresponding to the specific device operation, and a corresponding relationship with the current scheduling event is recorded, so as to ensure that the scheduling execution process and the event-driven scheduling model are consistent at the data level.

[0033] Embodiment 1 In order to verify the feasibility and effectiveness of the present application in the actual port container operation scene, the port container automatic scheduling method based on multi-agent reinforcement learning proposed by the present application is applied to the actual production environment of a large coastal automated container terminal. The annual throughput of the terminal exceeds five million standard containers, and multiple shore cranes, yard cranes, horizontal transportation devices and multiple yard operation units are simultaneously running in the operation system. The operation process involves multiple operation types such as loading, unloading, storage and transfer, and there are significant resource competition, channel conflict and yard capacity constraint problems between devices. During the peak operation period, the traditional scheduling method based on rules or local optimization is prone to problems such as increased device waiting time, local congestion of the yard and increased empty running rate of the horizontal transportation device, making it difficult to achieve stable and efficient collaborative scheduling at the system level.

[0034] In this embodiment, first, the existing port operation management system and device control system of the terminal are interfaced, the device basic information and operation plan information of the shore crane, yard crane, horizontal transportation device and yard operation unit are acquired in real time, and each device is mapped to an independent agent to form a multi-agent set covering the entire operation system. The container operation plan is sorted according to the unique container identifier to generate a set of operation tasks containing the starting operation position, target operation position and planned time information. In the actual running process, the system takes the arrival of operation tasks and the completion of device operation as the trigger condition, synchronously collects multi-source operation state data at the moment when the scheduling event occurs, and constructs a global state vector reflecting the overall running situation of the system through time alignment processing, while extracting a local state vector for each agent to describe the device state and the local operation environment it is in.

[0035] In the scheduling decision process, the system generates a scheduling constraint vector containing resource occupation, yard capacity and channel state based on the local state vector, and checks the feasibility of the action set in the action generation stage. The actions that do not meet the constraint conditions are removed from the candidate set, so as to avoid generating scheduling instructions that cannot be executed. On this basis, the improved QPLEX structure of the application is used to construct an individual action value network for each agent. By introducing a base value item branch and an advantage item branch using a duplex dueling structure, and embedding different constraint sources into the advantage item component for weighted combination, the individual action value can reflect the comprehensive influence of device state and constraint condition. At the same time, the system constructs a joint action value hybrid network driven by the global state vector to generate a hybrid weight parameter, which is used to weight and combine the individual action values of each agent to obtain the joint action value, which is used to evaluate the overall benefit of multiple devices executing the same scheduling scheme. In the online scheduling stage, the system selects an action combination according to the joint action value when each scheduling event is triggered, and converts it into specific device control instructions for execution.

[0036] To verify the beneficial effects of the application, in the comparison test of the next thirty days, the method of the application is compared with the original rule-driven scheduling method of the terminal, and the changes of the key operation indicators are counted. The terminal operation load remains at a similar level during the test period, and the daily average operation box volume fluctuates by no more than 5%. The experimental results are shown in Table 1:

[0037] Table 1 Comparison of operation effects of the method of the application and the original scheduling method in the actual port scene

[0038] As can be seen from the data in Table 1, the method of the application has produced stable and consistent improvement effects on multiple key operation indicators in the port container automatic scheduling scene. In the device level, the average waiting time of the quay crane and the yard crane is reduced by about 29%, indicating that the synchronization of multi-device collaborative scheduling is significantly improved; the empty running rate of horizontal transportation equipment is reduced by 25.6%, indicating that the scheduling decision is more reasonable in the path and task allocation level. In the yard operation aspect, the high occupancy rate period ratio is reduced by 30.8%, reflecting that the yard capacity constraint is effectively controlled in the scheduling process. At the same time, the single box average operation completion time is shortened by 14.3%, and the daily average completed operation box volume is improved by about 11% under the condition of basically consistent operation load, verifying the improvement of the overall throughput capacity of the system. It is worth noting that the proportion of non-executable scheduling instructions is reduced from 4.6% to 0.8%, indicating that the structured constraint and action pruning mechanism significantly reduces the generation of invalid scheduling instructions, further embodying the stability and implementability of the method of the application in the actual port operation environment.

[0039] The above merely describes preferred specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art, according to the technical solution and inventive concept of the present application, makes equivalent replacement or change within the technical range disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A method for automatic scheduling of port containers based on multi-agent reinforcement learning, characterized in that, Includes the following steps: S1. Obtain equipment information from the port operation system, establish a one-to-one correspondence between equipment and intelligent agents, and construct a set of container operation tasks; S2. Collect port operation status data at the time of the scheduling event trigger, construct a global state vector based on the container operation task set, and generate local state vectors for each agent based on the global state vector. S3. Generate scheduling constraint vectors based on the local state vectors, and perform feasibility trimming on the action sets of each agent according to the resource occupancy status, storage capacity status and path status to form a set of actionable actions. S4. Based on the improved QPLEX algorithm, construct an individual action value network for each agent using a duplex dueling structure, and calculate the individual action value based on the local state vector, scheduling constraint vector, and set of actionable actions. S5. Construct a joint action value hybrid network based on the improved QPLEX algorithm, and perform hybrid calculation on the individual action values ​​of each agent according to the global state vector to form a joint action value; S6. Construct a training sample sequence based on the scheduling event, distinguish and aggregate instant rewards and delayed rewards, and update the parameters of the individual action value network and the joint action value hybrid network based on the joint action value. S7. During the online scheduling process, at the moment the scheduling event is triggered, a combination of actions is selected from the set of actionable actions of each agent based on the value of the joint action, a scheduling instruction is generated and sent to the corresponding port operation equipment for execution.

2. The automatic port container scheduling method based on multi-agent reinforcement learning according to claim 1, characterized in that, S1 includes: S11. Read the basic equipment information of quay cranes, yard cranes, horizontal transport equipment and yard operation units from the port operation management system. The basic equipment information includes at least the equipment unique identifier, equipment type identifier, equipment current location, equipment operation capacity parameters and equipment operation time period. S12. Classify the basic information of the equipment according to the equipment type identifier, and divide the equipment sets corresponding to the same equipment type into quay crane equipment sets, yard crane equipment sets, horizontal transport equipment sets, and yard operation unit sets. S13. Assign a unique agent identifier to each piece of equipment in the set of quay crane equipment, set of yard crane equipment, set of horizontal transport equipment, and set of yard operation units, establish a one-to-one correspondence between the agent identifier and the corresponding unique equipment identifier, and generate a mapping table between equipment type identifier and agent identifier. S14. Read container operation plan data from the port operation plan system. The container operation plan data includes at least the container unique identifier, operation type identifier, starting operation location, target operation location, planned start time, and planned completion time. S15. Using the unique identifier of the container as an index, the container operation plan data is structured to generate a set of container operation tasks, and a task identifier is assigned to each container operation task. The task identifier is then associated with the corresponding starting operation location, target operation location, and time information.

3. The automatic port container scheduling method based on multi-agent reinforcement learning according to claim 1, characterized in that, S2 includes: S21. At the time the scheduling event is triggered, port operation status data is collected from the port operation management system and the equipment control system. The port operation status data includes at least the current operation status of each piece of equipment, the current position of each piece of equipment, the task identifier assigned to each piece of equipment, the remaining operation time of each piece of equipment, and the task queue order corresponding to each container operation task. S22. Perform unified time alignment processing on the collected port operation status data, map status data from different sources to the same scheduling event time, and form the original status data set corresponding to the scheduling event. S23. Construct a global state vector based on the original state data set and the container operation task set. The global state vector includes an equipment occupancy state vector, a task queue state vector, a yard block state vector, and a passage state vector. S24. Based on the global state vector and the correspondence between devices and agents, extract the corresponding local state vector for each agent. The local state vector includes at least the working status of the corresponding device, current location, assigned task information, reachable yard block status and related channel status. S25. Combine the local state vectors corresponding to each agent in a preset order to form a state input sequence that corresponds one-to-one with each agent.

4. The automatic port container scheduling method based on multi-agent reinforcement learning according to claim 1, characterized in that, S3 includes: S31. Based on the local state vectors corresponding to each agent, generate scheduling constraint vectors according to the constraint component construction method; S32. Generate resource occupancy constraint components based on the corresponding equipment's operating status, assigned task identifiers, and remaining operating time; S33. Generate yard capacity constraint components based on the reachable yard block status and remaining capacity information of the yard block recorded in the local state vector. S34. Generate path constraint components based on the channel state information recorded in the local state vector; S35. The resource occupancy constraint component, the storage yard capacity constraint component, and the path constraint component are combined in a preset order to form a scheduling constraint vector for the corresponding intelligent agent. S36. Construct an action set for each intelligent agent, the action set including task selection action and task order adjustment action; S37. Based on the scheduling constraint vector, each action set of each agent is checked item by item, and actions that do not meet the resource occupancy constraint component, storage yard capacity constraint component or path constraint component are eliminated to form the action set corresponding to each agent.

5. The automatic port container scheduling method based on multi-agent reinforcement learning according to claim 1, characterized in that, S4 includes: S41. Construct an individual action value network for each agent. The individual action value network includes a shared feature extraction layer, a base value branch, and a dominant term branch. The base value branch and the dominant term branch adopt a duplex dueling structure. S42. Group the local state vector according to a preset field order to form equipment state field group, task field group, yard field group, and channel field group, and perform numerical dimension unification processing on each field group. The numerical dimension unification processing includes performing linear scaling on count fields, binarizing on marker fields, and performing discrete grid numbering encoding on location fields. S43. The scheduling constraint vector is split into components according to the constraint source to form resource occupancy constraint sub-vector, yard capacity constraint sub-vector, and path constraint sub-vector, and the resource occupancy constraint sub-vector, yard capacity constraint constraint sub-vector, and path constraint sub-vector are sequentially concatenated to form a constraint embedding vector. S44. Based on the set of possible actions, establish an action index set and construct an action mask vector. The action mask vector marks actions within the set of possible actions as valid values ​​and actions outside the set of possible actions as invalid values. S45. Provide the local state vector, which has been grouped and encoded by the field, and the constraint embedding vector to the shared feature extraction layer, and output the shared feature vector; S46. In the basic value term branch, the shared feature vector is mapped to a basic value term scalar; S47. In the advantage item branch, the shared feature vector is combined with the action embedding vector corresponding to the action index set to generate resource occupation advantage component, yard capacity advantage component and path advantage component respectively. The resource occupation advantage component, yard capacity advantage component and path advantage component are weighted and summed according to preset weight coefficients to form the advantage item corresponding to the action. S48. Perform set normalization on the dominance items, calculate the arithmetic mean of the dominance items within the range of possible actions, and subtract the arithmetic mean from the dominance item of each action within the range of possible actions. S49. Based on the synthesis rules of the duplex dueling structure, the base value term is synthesized with the dominant term after set normalization, and the synthesis result corresponding to the non-actionable action is assigned a preset invalid flag value according to the action mask vector, and the individual action value of each agent in the actionable action set is output.

6. The automatic port container scheduling method based on multi-agent reinforcement learning according to claim 1, characterized in that, S5 includes: S51. Construct a joint action value hybrid network, wherein the joint action value hybrid network includes a hybrid feature generation layer and a joint value output layer; S52. Group the global state vector according to the preset field order to form the equipment operation field group, task distribution field group, yard status field group and channel status field group, and perform numerical dimension consistency processing on each field group. S53. Input the global state vector, which is grouped and encoded by the field, into the hybrid feature generation layer to generate a hybrid weight parameter set, wherein the hybrid weight parameter set contains weight parameters that correspond one-to-one with the individual action value of each agent. S54. According to the arrangement order of the mixed weight parameter set, perform linear weighted combination operation on the individual action value corresponding to each agent to generate an intermediate result of joint action value; S55. Perform bias term addition processing on the intermediate results of the joint action value to form the joint action value; S56. At the time the scheduling event is triggered, the value of the joint action is associated with and stored in conjunction with the corresponding agent action combination.

7. The automatic port container scheduling method based on multi-agent reinforcement learning according to claim 1, characterized in that, S6 includes: S61. At the moment each scheduling event is triggered, record the corresponding global state vector, the local state vector of each agent, the action index executed by each agent, and record the global state vector corresponding to the scheduling event preceding the scheduling event. S62. Based on the global and local state vectors corresponding to two adjacent scheduling events, perform field-level comparison calculations on the equipment operation status marker, task queue sorting position, and resource occupation marker to form a set of state change quantities corresponding to a single scheduling event. S63. Based on the preset report mapping rules, perform numerical mapping processing on the changes in device operation completion markers, task queue sorting, and resource occupancy status in the set of state changes to generate real-time report data. S64. According to the order of occurrence of scheduling events, the set of state change quantities corresponding to multiple consecutive scheduling events is accumulated. The accumulation process includes accumulating the changes in the number of yard blocks occupied, the changes in the channel occupancy status, and the changes in equipment waiting time on an event-by-event basis to generate delay report data. S65. Combine the global state vector, the local state vectors of each agent, the action index of each agent, the instantaneous reward data, the delayed reward data, and the global state vector corresponding to the next scheduling event in chronological order to construct a training sample sequence. S66. Based on the training sample sequence, update the parameters of the individual action value network and the joint action value hybrid network according to the preset training order.

8. The automatic port container scheduling method based on multi-agent reinforcement learning according to claim 1, characterized in that, S7 includes: S71. At the time the scheduling event is triggered, obtain the global state vector corresponding to the current scheduling event and the local state vector corresponding to each agent. S72. Based on the set of possible actions of each agent, call the individual action value network to calculate the individual action value of each agent on the corresponding set of possible actions. S73. Input the individual action value of each agent into the joint action value hybrid network, and calculate the joint action value corresponding to the action combination of each agent. S74. Select the action combination with the largest value from the joint action value, and determine the action to be executed by each agent under the current scheduling event. S75. Convert the execution action into equipment control instructions, wherein the equipment control instructions consist of job task assignment instructions and job sequence adjustment instructions; S76. Send the equipment control command to the corresponding quay crane, yard crane, horizontal transport equipment and yard operation unit, and record the correlation between the executed action and the corresponding scheduling event.

Images