Confrontation training method for global resource scheduling of port equipment

Abstract

The invention relates to a confrontation training method for global resource scheduling of port equipment, which comprises the following steps: acquiring an off-line learning training sample by using historical data based on operation of ships, bridge cranes, automatic guided vehicles and rail cranes, and adopting a speech-commentator-generator model as a training model, and an adversarial sample generated based on the generator and a real sample are discriminated to form a regular term constraint of the Actor network, so that the Actor network has certain anti-noise interference and anti-attack capabilities, and the robustness of the Actor network is improved. The trained Actor network has good generalization performance in dynamic scheduling of port equipment, and real-time scheduling of the equipment can be obtained by deploying and using state information collected in real time in a port environment.

Description

Technical Field

[0001] The present invention relates to a global equipment real-time scheduling algorithm for global equipment of a port, and in particular to a robust equipment scheduling algorithm under noise interference. Background Art

[0002] Due to the dynamic and complex nature of port equipment scheduling, it is challenging to schedule global equipment serving ships in real time. In addition, due to factors such as equipment failure, weather conditions, and dynamic environment, the environmental state fluctuates greatly, which may lead to the performance degradation of offline reinforcement learning models.

[0003] To achieve dynamic allocation of port equipment resources in a noisy and highly random environment, current methods include:

[0004] (1) Equipment scheduling based on traditional algorithms: mathematical modeling of the port environment is performed, and heuristic algorithms are used to schedule equipment. Due to the complexity of algorithm modeling, it is difficult to achieve global optimization in a short period of time, and the correlation between equipment makes it difficult to deploy real-time scheduling for multiple equipment.

[0005] (2) Simulation-based port equipment dispatching process: By building a simulation platform, a twin port operating environment is constructed in the simulation environment to simulate the equipment dispatching process. However, this method is difficult to model and analyze sudden weather conditions or equipment failures.

[0006] (3) Dynamic equipment scheduling based on offline reinforcement learning: Due to the safety and reliability requirements of the port equipment operating environment, it is impossible to interact with the environment in real time, resulting in insufficient exploration capabilities. In addition, the unmanned terminal environment faces random noise interference or extreme weather conditions, which makes global equipment scheduling difficult. Existing offline reinforcement learning cannot meet the requirements of real-time port equipment scheduling. Summary of the invention

[0007] In order to solve the above-mentioned problems existing in the prior art, the purpose of this case is to propose an adversarial training method for global resource scheduling of port equipment. In the method, an adversarial network is combined with reinforcement learning for training. The trained model is based on the current state of port equipment and can obtain the port equipment scheduling action.

[0008] In order to achieve the above technical objectives, this case proposes an adversarial training method for global resource scheduling of port equipment, the steps of which include: obtaining training samples for offline learning based on historical operation data of ships, bridge cranes, automatic guided vehicles, and rail cranes, each training sample includes the current observation state and the corresponding execution action; constructing a speech-critic (Actor-Critic)-generator model as a training model, in which the Actor network also serves as a discriminator and constitutes an adversarial network with the generator; training the training model based on the training samples, and updating the parameters of the Critic network, the Actor network, and the generator using the gradient descent method respectively; wherein the step of updating the parameters of the Actor network includes: for each training sample, using the speech-critic part for training and learning, and obtaining the Actor loss L for the current observation state and the corresponding execution action. actor ; Use the generator to generate adversarial fluctuations based on the current observation state and random noise, and then adjust the interference intensity based on the adversarial fluctuations to obtain an updated observation state; Use the Actor network to perform feature judgment on the updated observation state and the current observation state to obtain the discrimination loss L dis ; Calculate the total loss of the Actor network L2 = L actor +βL dis , β is the weight of the discriminant loss; based on the total loss of the Actor network, the Actor network parameters are updated.

[0009] In one implementation of the above technical solution, the discrimination loss is calculated as:

[0010] L dis =(π(s t +εg) pa -π(s t ) pa ) 2 +KL(π(s t +εg) fea ,π(s t ) fea )

[0011] In the formula, g(z,s t ) is the generator based on the current observation state s t and the adversarial fluctuations generated by random noise z, ε is the adjustment weight of the adversarial fluctuations, s t +εg is the current observation state after counteracting interference, π(s t +εg) pa The action obtained by using the Actor network to counteract the current observation state after interference, π(s t ) pa is the current observation state s tThe action obtained using the Actor network, π(s t +εg) fea The feature encoding result obtained by using the Actor network for the current observation state after countermeasure interference, π(s t ) fea is the current observation state s t The feature encoding result obtained using the Actor network, KL is the calculation divergence loss.

[0012] In one implementation of the above technical solution, the generator loss is L gen =-(π(s t +εg(z,s t )) pa -π(s t ) pa ) 2 , g(z,s t ) is the generator based on the current observation state s t and the adversarial fluctuations generated by random noise z, ε is the adjustment weight of the adversarial fluctuations, π(s t +εg(z,s t )) pa The action obtained by using the Actor network to counteract the current observation state after interference, π(s t ) pa is the current observation state s t Actions obtained using the Actor network.

[0013] In one implementation of the above technical solution, the training sample acquisition step includes: constructing data records (Did, P, Sid, Ts, Te, Dy) based on historical data, Did is the number of each operating equipment, P is the location of the operating equipment, Sid represents the ship number, Ts is the operation start time, Te is the operation end time, and Dy is the equipment type; deleting records with empty values, records with cancelled operation plans, and records with operation time exceeding a set threshold; according to the operation start time T S and the job end time T E The time slice is divided into time slices, and the length of the time slice is a set value. The equipment status, operation tasks, number of remaining containers, and whether the ship has completed the operation of each quay crane, automatic guided vehicle, and rail crane at each time slice are counted. Based on the statistical data of each time slice, a training sample (s t , a t ,s t+1 , r t , γ).

[0014] In one implementation of the above technical solution, the time slice is 10 seconds.

[0015] In one implementation of the above technical solution, the training learning is calculated using the following reward value r: And judge whether the task is completed at each time interval. When the task is completed, if the total execution time of the task is longer than the planned completion time, calculate the penalty value f: The reward value minus the penalty value is used as the reward value for the current time interval.

[0016] In order to achieve the above technical objectives, this case proposes a global resource scheduling method for port equipment, the steps of which include: based on the current status of the port equipment, using a trained Actor network to predict the scheduling action, so that each device executes the scheduling action; the Actor network is trained using any of the above methods.

[0017] To achieve the above technical objectives, according to the above method and technical scheme, this case proposes a global resource scheduling system for port equipment, the system includes a training sample construction module and a training module; the training sample construction module is configured to obtain training samples for offline learning based on the historical operation data of ships, bridge cranes, automatic guided vehicles, and rail cranes, and each training sample includes a current observation state and a corresponding execution action; the training model unit uses the speech-critic (Actor-Critic)-generator model as a training model, wherein the Actor network is also used as a discriminator and constitutes an adversarial network with the generator; the training model unit is configured to train the training model based on the training samples, and update the parameters of the Critic network, the Actor network and the generator using the gradient descent method respectively; wherein the step of updating the parameters of the Actor network includes: for each training sample, using the speech-critic part for training and learning, and obtaining the Actor loss L for the current observation state and the corresponding execution action. actor ; Use the generator to generate adversarial fluctuations based on the current observation state and random noise, and then adjust the interference intensity based on the adversarial fluctuations to obtain an updated observation state; Use the Actor network to perform feature judgment on the updated observation state and the current observation state to obtain the discrimination loss L dis ; Calculate the total loss of the Actor network L2 = L actor +βL dis , β is the weight of the discriminant loss; based on the total loss of the Actor network, the Actor network parameters are updated.

[0018] Compared with the prior art, the beneficial effects of this solution are as follows: This solution provides an end-to-end method, which cleverly uses the Actor network as part of the speech critic model for training and learning, and also as a discriminator in the adversarial network to form an adversarial network with the generator. The Actor network, as a discriminator, forms a regular term constraint for the Actor network by judging and training the adversarial samples with small fluctuations generated by the generator and the real samples, so that the Actor network has a certain ability to resist noise interference and counterattacks, improves the robustness of the Actor network reinforcement learning model in a noisy environment, and enhances the generalization performance of the dynamic scheduling of port equipment. It solves the problem of simulation difficulties caused by the large number of variables in the real operating environment of the port, and finally realizes the requirements of robust equipment scheduling under noise interference, dynamic environment, random failures and weather conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0020] Figure 1 , one Schematic diagram of robust offline reinforcement learning for adversarial training in one implementation.

[0021] Figure 2 , one Schematic diagram comparing the mean and standard deviation of rewards obtained by the method of using only the Critic-Actor model and the method of introducing adversarial training of the generator into the Critic-Actor model under noise of different intensities in the implementation mode.

[0022] Figure 3 , one Schematic diagram of distributed execution after centralized deployment in an implementation manner. DETAILED DESCRIPTION

[0023] Methods based on traditional resource allocation algorithms have the problem of difficulty in achieving robust equipment scheduling under dynamic environments and random failures; methods based on port environment simulation algorithms have the problem of many variables in the operating environment making simulation difficult; and existing offline reinforcement learning algorithms cannot meet the requirements of real-time port equipment scheduling.

[0024] In response to the requirements for robust equipment scheduling under noise interference, dynamic environment, random failures and weather conditions, this solution proposes a global resource scheduling method for port equipment. The following will combine the accompanying drawings to clearly and completely describe how the technical solution of this case is implemented. Obviously, the described implementation method is only a part of the implementation method of this case, not all of the implementation methods. Based on the implementation methods in this case, all other implementation methods obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0025] (I) Training data preparation

[0026] First, collect historical scheduling data of each quay crane, automated guided vehicle (AGV) and rail crane for at least half a year, and construct data records (Did, P, Sid, Ts, Te, Dy). Did is the number of each operating equipment. P is the position of the operating equipment, represented by x and y coordinates in a two-dimensional coordinate system. Sid is the ship number, indicating the ship to which the loaded cargo belongs. Ts is the start time of the operation. Te is the end time of the operation. Dy is the equipment type, including gantry crane, AGV, and rail crane.

[0027] Secondly, the constructed data records are preprocessed, and records with empty values ​​are deleted, data with cancelled job plans are deleted, and records with too long job time are deleted to obtain a clean and complete data set. One way to judge whether the job time is too long is to set the difference between the start time and the end time in an entry.

[0028] Next, according to the start time T S and end time T E The time slice is divided according to the precise value of , which is accurate to 1 second. All operating equipment of the same type in a time slice does not exceed the maximum number of equipment and does not exceed the maximum transit time. Exemplarily, according to the production scheduling requirements of the port, the time slice is set to 10 seconds, that is, the timestamp is a 10-second interval. The equipment status, operation tasks, number of remaining containers, and whether the ship has completed the operation of the bridge crane, automatic guided vehicle, and rail crane at each quay crane in each time slice are counted. A training sample is obtained based on the statistical data of each time slice to generate a standard offline reinforcement learning data set.

[0029] In one implementation, a standard offline reinforcement learning dataset is constructed, including the following fields:

[0030] observations, next observations, actions, rewards, terminals. observations indicates the current observation state, next_observations indicates the observation state at the next moment, actions indicates the action corresponding to the current observation state, rewards indicates the immediate reward, and terminals indicates whether the current ship has completed the operation. These fields are stored in the form of numpy.array.

[0031] Next, a unified observation state space and action space are constructed for all devices in the current port, and reward setting rules are designed, completion status is defined, etc. The specific implementation is as follows.

[0032] (1) Observation state space vector representation S

[0033] S = (ship_id, operation task, qiaodiao_id, agv_id, guidaodiao_id, number of remaining containers, current status of the ship). Among them, ship_id is the ship index, the operation tasks are loading / unloading, qiaodiao_id is the bridge crane index, agv_id is the automatic guided vehicle index, and guidaodiao_id is the rail crane index.

[0034] (2) The action space vector A is expressed as:

[0035] A = (operation task, qiaodiao_id, agv_id, guidaodiao_id), when the equipment is currently scheduled, the action space value is 1, otherwise it is 0. The operation tasks are loading / unloading.

[0036] (3) Reward space

[0037] This case uses dense rewards.

[0038] Reward value r calculation:

[0039] At the same time, in the current time interval, the percentage of tasks that have been completed by the equipment is obtained, and the percentage of the running tasks is the total number of containers operated as of the current time / the number of containers on the current ship. The planned completion time of the running task is further obtained, and the exemplary planned completion time is 10 minutes. When the task is completed, if the total execution time of the task is less than or equal to the planned completion time, no penalty is imposed. If the total execution time of the task is greater than the planned completion time, the penalty value f is calculated:

[0040] At this time, the updated reward value is equal to the reward value r-penalty value f.

[0041] (4)terminal

[0042] The value of terminal is determined by the loading task and the unloading task. Terminal will be True only after both tasks are completed.

[0043] (II) Training Model

[0044] The training sample is represented as a five-tuple of state transitions (s t , a t ,s t+1 , r t ,γ). Among them, s t ∈S represents the current observation state, a t ∈A means taking action based on the current observation state, s t+1 represents the state at the next moment, r t represents the immediate reward, γ represents the reward discount factor, γ∈[0, 1], which is a hyperparameter.

[0045] A speech-critic (Actor-Critic)-generator model is constructed as a training model, in which the Actor network also serves as a discriminator and forms an adversarial network with the generator. The training model is trained based on training samples.

[0046] First, for each training sample, the speech-critic part is used for training and learning. The Actor network predicts the scheduling action to be executed and the next observation state based on the current observation state, and the Critic network evaluates the current observation state of the Actor network and the scheduling action to be executed.

[0047] Evaluation Set the value function Q according to the Bellman operator φ (s t , a t )(φ represents the parameter), the learning goal y can be expressed as: r represents the reward value of a single step, a t+1 Indicates the action at the next moment. Bellman loss L Q By L Q =(Q(s t , a t )-y) 2 calculate.

[0048] After training the Critic network, train the Actor network π θ (θ represents the parameter), Actor will return the prediction result and the feature encoding of the input, using π(s t ) pa represents the feedback of the Actor network on the current observation, π(s t )fea Represents the feature encoding result of the current observation by the Actor network.

[0049] The Actor network cooperates with the Critic network for intensive training, and on the other hand, it serves as a discriminator and generator. ( Represents parameters) to form an adversarial network for adversarial training.

[0050] In reinforcement training, the Actor loss L is constructed for the current intrinsic observation state and its corresponding action actor :L actor =λQ(s t ,π(s t ) pa )-(π(s t ) pa -a t ) 2 , λ represents the weighting parameter, α is set to 2.5 by default, and N is the batch size (BachSize).

[0051] In adversarial training, the generator generates adversarial fluctuations of a certain intensity, uses the fluctuations to influence the observed state, and achieves robust learning of the agent by adjusting the noise intensity. The discriminator is used to distinguish the observed state affected by the adversarial fluctuations from the real observed state.

[0052] Specifically, the generator is used to generate adversarial fluctuations g(z, s) based on the current observation state and random noise. t ), z represents random noise, and for example, z ≈ N (0, 1), N represents Gaussian distribution. t represents the true internal observation state, which remains unchanged when disturbed by noise. Then the internal observation is disturbed to obtain s t +∈g(z,s t ), ∈ represents the adjustment weight of the anti-fluctuation, see Figure 1 . The intensity range of the adversarial fluctuations is changed by gradually increasing ∈.

[0053] In one embodiment, the generator is constructed separately with the same architecture as the discriminator, so that the discriminator and the actor share weights. The discriminator calculates the loss for adversarial fluctuations and clean state observations respectively and optimizes using a gradient descent algorithm.

[0054] For the features of fluctuating observations and intrinsic true observations, KL divergence is used as a constraint to reduce the feature deviation caused by the agent facing the noisy observation state. The KL divergence loss can be expressed as:

[0055] Actor generates an action π(s) for the current observation state after the disturbance t +∈g(z,s t )), in order to generate stable action feedback, this Actor also acts as a discriminator, and the overall loss of the discriminator is:

[0056] L dis =(π(s t +∈g(z,s t )) pa -π(s t ) pa ) 2 +KL(π(s t +∈g(z,s t )) fea ,π(s t ) fea )

[0057] The loss of the entire Actor network can be expressed as L2=L actor +βL dis , β represents the weighting of the discriminative loss, L dis It can be regarded as a regular term to constrain the Actor network, so that the Actor has a certain ability to resist noise interference and counter attacks. Based on the loss L2 of the entire Actor network, the parameters of the Actor network are updated using the gradient descent method.

[0058] The generator is to maximize the difference between noisy observation state samples and clean observation state samples, and its loss can be expressed as:

[0059] L gen =-(π(s t +∈g(z,s t )) pa -π(s t ) pa ) 2

[0060] Since the gradient descent algorithm is used, a negative sign is added in front of the MSE (mean square error) loss, indicating that the optimization goal for the two samples is to maximize the distance.

[0061] At this point, the training of the Speaker-Critic-Generator model is completed. The trained Actor network has good generalization performance in the dynamic scheduling of port equipment. It is deployed in the port environment and can obtain real-time scheduling of equipment using real-time collected status information.

[0062] (III) Verification

[0063] The trained agent is tested in a clean observation environment until the loading and unloading task of a ship is completed, indicating that the task of the ship is completed. At the same time, noise of different intensities is used to interfere with the trained Actor network to verify its anti-interference ability to different noise intensities, and the mean and standard deviation of the reward under different adversarial fluctuation intensities are compared and analyzed. Figure 2 .exist Figure 2 In the figure, the original method is a method that only uses the Critic-Actor model, and adversarial training is a method that introduces a generator into the Critic-Actor model. The vertical line in the figure is the change in reward value under each noise perturbation intensity, and the curve formed by connecting the midpoints is the mean curve.

[0064] from Figure 2 It can be seen that the Actor network after adversarial training has better robustness and generally performs better than the original algorithm when facing noise, showing its excellent generalization performance.

[0065] In one embodiment, the adjustment weight of the counteracting fluctuation used is 0~∈.

[0066] (IV) Application

[0067] The model is deployed in the port environment, and the reinforcement learning model is deployed in the center. The central server is used to collect environmental information in real time (including the current operating equipment Did, the operating object ship number Sid, the equipment type Dy and other data) as input, and then the deterministic action π(s) is predicted through the Actor network. t ), thereby scheduling the functions of the equipment in real time. Commands are issued to each device to perform container scheduling tasks in a distributed manner.

[0068] Furthermore, the collected datasets can be used to fine-tune the model for offline reinforcement learning methods.

[0069] In one implementation, the model trained in an offline scenario uses a centralized node deployment method to obtain the dispatching device Did in real time to perform container transshipment work, such as Figure 3 As shown. For each ship's state observation s at the current time t t As input, the trained Actor model is used to predict the Did of the operating equipment at time t+1, thereby achieving the goal of global scheduling of port equipment.

[0070] (V) Summary

[0071] It can be seen from the above implementation process that this solution, based on the development needs of intelligence and robustness of unmanned terminals, proposes adversarial training robust offline reinforcement learning for global resource scheduling of port equipment, realizes real-time scheduling of port equipment, can handle the loading and unloading tasks of multiple ships at the same time, and has a certain ability to resist noise, meeting the needs of real-time global scheduling of equipment for intelligent production in the field of smart terminals, and is a more effective and scientific technical means.

[0072] It should be noted that this solution does not limit the network structure, and can be used for feature extraction using MLP, LSTM, CNN, and other architectures. There is no limit on the deployment method of the reinforcement learning model, and it can be deployed in a single operation scenario or centrally deployed in a global device scheduling scenario with only minor changes.

[0073] Through the description of the above implementation methods, the technical personnel in the relevant field can clearly understand that according to the method disclosed in the present invention, it can be implemented as a corresponding global resource scheduling system for port equipment. Exemplarily, the system includes a training sample construction module and a training module; the training sample construction module is configured to obtain training samples for offline learning based on the historical operation data of ships, bridge cranes, automatic guided vehicles, and rail cranes, and each training sample includes a current observation state and a corresponding execution action; the training model unit uses the speech-critic (Actor-Critic)-generator model as a training model, wherein the Actor network is also used as a discriminator and constitutes an adversarial network with the generator; the training model unit is configured to train the training model based on the training samples, and respectively update the parameters of the Critic network, the Actor network and the generator using the gradient descent method; wherein the step of updating the parameters of the Actor network includes: for each training sample, use the speech-critic part for training and learning, and obtain the Actor loss L for the current observation state and the corresponding execution action. actor ; Use the generator to generate adversarial fluctuations based on the current observation state and random noise, and then adjust the interference intensity based on the adversarial fluctuations to obtain an updated observation state; Use the Actor network to perform feature judgment on the updated observation state and the current observation state to obtain the discrimination loss L dis ; Calculate the total loss of the Actor network L2 = L actor +βL dis , β is the weight of the discriminant loss; based on the total loss of the Actor network, the Actor network parameters are updated.

[0074] Through the description of the above implementation modes, those skilled in the art can clearly understand that the method and system of the present disclosure can be implemented by means of software plus necessary general hardware, and of course can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions performed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structures used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits, etc. However, for the present disclosure, software program implementation is a better implementation mode in most cases.

[0075] Although the embodiments of the present invention are described above in conjunction with the accompanying drawings, the present invention is not limited to the above specific embodiments and application fields, and the above specific embodiments are only illustrative and instructive, rather than restrictive. A person of ordinary skill in the art can also make many forms under the guidance of this specification and without departing from the scope of protection of the claims of the present invention, all of which belong to the protection of the present invention.

Claims

1. An adversarial training method for global resource scheduling of port equipment, It is characterized in that Methods include: Based on the historical data of ship, bridge crane, automatic guided vehicle and rail crane operation, training samples are obtained for offline learning. Each training sample includes the current observation state and the corresponding execution action. Construct a speech-critic (Actor-Critic)-generator model as a training model, in which the Actor network also acts as a discriminator and forms an adversarial network with the generator; The training model is trained based on the training samples, and the parameters of the Critic network, the Actor network and the generator are updated using the gradient descent method respectively; wherein the step of updating the parameters of the Actor network includes: For each training sample, the speech-critic part is used for training and learning to obtain the Actor loss L for the current observation state and the corresponding execution action actor ; Using a generator to generate adversarial fluctuations based on the current observation state and random noise, and then adjusting the interference intensity based on the adversarial fluctuations to obtain an updated observation state; Use the Actor network to perform feature judgment on the updated observation state and the current observation state to obtain the discrimination loss L dis ; Calculate the total loss L of the Actor network 2 =L actor +βL dis , β is the weight of the discriminant loss; Based on the total loss of the Actor network, the Actor network parameters are updated.

2. The method according to claim 1, It is characterized in that The discriminative loss is calculated as: L dis =(π(s t +εg(z,s t )) pa -π(s t ) pa ) 2 +KL(π(s t +εg(z,s t )) fea ,π(s t ) fea ) In the formula, g(z,s t ) is the generator based on the current observation state s t and the adversarial fluctuations generated by random noise z, ε is the adjustment weight of the adversarial fluctuations, π(s t +εg(z,s t )) pa The action obtained by using the Actor network to counteract the current observation state after interference, π(s t ) pa is the current observation state s t The action obtained using the Actor network, π(s t +εg(z,s t )) fea The feature encoding result obtained by using the Actor network for the current observation state after countermeasure interference, π(s t ) fea is the current observation state s t The feature encoding result obtained using the Actor network, KL is the calculation divergence loss.

3. The method according to claim 1, Features: The generator loss L gen for: L gen =-(π(s t +εg(z,s t )) pa -π(s t ) pa ) 2 Where: g(z,s t ) is the generator based on the current observation state s t and the adversarial fluctuations generated by random noise z, ε is the adjustment weight of the adversarial fluctuations, π(s t +εg(z,s t )) pa The action obtained by using the Actor network to counteract the current observation state after interference, π(s t ) pa is the current observation state s t Actions obtained using the Actor network.

4. The method according to claim 1, It is characterized in that The training sample acquisition step comprises: Construct data records (Did, P, Sid, Ts, Te, Dy) based on historical data, where Did is the number of each operating equipment, P is the location of the operating equipment, Sid represents the ship number, Ts is the start time of the operation, Te is the end time of the operation, and Dy is the equipment type; Delete records with empty values, records of job plan cancellations, and records whose job time exceeds the set threshold; According to the job start time T S and the job end time T E Time slices are performed with the length of the time slice being a set value. The equipment status, operation tasks, number of remaining containers, and whether the ship has completed the operation of the gantry crane, automatic guided vehicle, and rail crane at each quay crane in each time slice are counted. A training sample is obtained based on the statistical data of each time slice.

5. The method according to claim 4, It is characterized in that The time slice is 10 seconds.

6. The method according to claim 1, It is characterized in that The training learning is calculated using the following reward value r: And judge whether the task is completed at each time interval. When the task is completed, if the total execution time of the task is longer than the planned completion time, calculate the penalty value f: The reward value minus the penalty value is used as the reward value for the current time interval.

7. A global resource scheduling method for port equipment. It is characterized in that the steps include: Based on the current status of port equipment, the trained Actor network is used to predict the scheduling action, so that each device executes the scheduling action; The Actor network is obtained by training using any one of the methods in claims 1 to 6.

8. A global resource scheduling system for port equipment. Features: The system includes a training sample construction module and a training module; The training sample construction module is configured to obtain training samples for offline learning based on the historical operation data of ships, bridge cranes, automatic guided vehicles, and rail cranes, each training sample including a current observation state and a corresponding execution action; The training model unit uses the speech-critic (Actor-Critic)-generator model as the training model, wherein the Actor network also serves as a discriminator and constitutes an adversarial network with the generator; the training model unit is configured to train the training model based on the training samples, and respectively update the parameters of the Critic network, the Actor network and the generator using the gradient descent method; wherein the step of updating the parameters of the Actor network includes: for each training sample, using the speech-critic part for training and learning, and obtaining the Actor loss L for the current observation state and the corresponding execution action actor ; Using a generator to generate adversarial fluctuations based on the current observation state and random noise, and then adjusting the interference intensity based on the adversarial fluctuations to obtain an updated observation state; Use the Actor network to perform feature judgment on the updated observation state and the current observation state to obtain the discrimination loss L dis ; Calculate the total loss L of the Actor network 2 =L actor +βL dis , β is the weight of the discriminant loss; Based on the total loss of the Actor network, the Actor network parameters are updated.

9. The system according to claim 8, It is characterized in that The discriminative loss is calculated as: L dis =(π(s t +εg(z,s t )) pa -π(s t ) pa ) 2 +KL(π(s t +εg(z,s t )) fea ,π(s t ) fea ) In the formula, g(z,s t ) is the generator based on the current observation state s t and the adversarial fluctuations generated by random noise z, ε is the adjustment weight of the adversarial fluctuations, s t +εg is the current observation state after counteracting interference, π(s t +εg) pa The action obtained by using the Actor network to counteract the current observation state after interference, π(s t ) pa is the current observation state s t The action obtained using the Actor network, π(s t +εg) fea The feature encoding result obtained by using the Actor network for the current observation state after countermeasure interference, π(s t ) fea is the current observation state s t The feature encoding result obtained using the Actor network, KL is the calculation divergence loss.

10. The system according to claim 8, Features: The training sample acquisition step comprises: Construct data records (Did, P, Sid, Ts, Te, Dy) based on historical data, where Did is the number of each operating equipment, P is the location of the operating equipment, Sid represents the ship number, Ts is the start time of the operation, Te is the end time of the operation, and Dy is the equipment type; Delete records with empty values, records of job plan cancellations, and records whose job time exceeds the set threshold; According to the job start time T S and the job end time T E Time slices are performed with the length of the time slice being a set value. The equipment status, operation tasks, number of remaining containers, and whether the ship has completed the operation of the gantry crane, automatic guided vehicle, and rail crane at each quay crane in each time slice are counted. A training sample is obtained based on the statistical data of each time slice.

Images