Reinforcement learning model training method, and unloading machine control method and device based on reinforcement learning

Abstract

This application discloses a reinforcement learning model training method, a reinforcement learning-based ship unloader control method and device, relating to the field of control technology. The reinforcement learning model training method includes: constructing an initial reinforcement learning model, comprising a graph neural network for representing the current state of the ship unloader through feature aggregation, a policy network for outputting the current control action based on the current state of the ship unloader, and a value network for estimating the long-term expected reward obtained by the ship unloader in the current state by following the current control action; and performing single-machine basic policy learning, dual-machine cooperative collision avoidance learning, and multi-machine cooperative collision avoidance learning at different training stages to obtain a target reinforcement learning model, which is used to determine the control action for each ship unloader in the system. Using the method of this application, the problems of the central scheduling system can be solved and the efficiency of multi-machine cooperative operation can be improved.

Description

Technical Field

[0001] This application relates to the field of control technology, and in particular to a reinforcement learning model training method, a reinforcement learning-based ship unloader control method and device. Background Technology

[0002] With the continuous expansion of global port logistics, bulk cargo terminals (such as those handling coal, ore, and grain) are increasingly demanding higher efficiency and automation levels in loading and unloading. As a core piece of loading and unloading equipment at terminals, the efficiency of ship unloaders directly impacts the overall throughput capacity of the terminal. In the loading and unloading of large bulk carriers, multiple ship unloaders are typically used simultaneously to improve efficiency.

[0003] However, in existing bulk cargo terminal operations, multiple ship unloaders typically rely on a central dispatch system for path planning and conflict coordination. Each ship unloader, as a passive execution unit, lacks autonomous decision-making capabilities. This centralized dispatch architecture suffers from problems such as high response latency, high system failure risk, and poor scalability. In addition, traditional methods often employ fixed rules or static optimization algorithms for collision avoidance of ship unloaders, which are difficult to adapt to dynamic operating environments and result in low collaborative efficiency. Summary of the Invention

[0004] This application provides a reinforcement learning model training method, a reinforcement learning-based ship unloader control method and apparatus, to solve or at least partially solve the defects or deficiencies in related technologies.

[0005] Firstly, this application provides a reinforcement learning model training method, the reinforcement learning model training method comprising: Construct an initial reinforcement learning model; the initial reinforcement learning model includes a graph neural network for characterizing the current state of the ship unloader through feature aggregation, a policy network for outputting the current control action for the current state of the ship unloader, and a value network for estimating the long-term expected reward obtained by the ship unloader in the current state by following the current control action. For the first training phase, a first sample set is obtained, and the parameters of the policy network and the value network are trained based on the first sample set in the first training phase so that the initial reinforcement learning model can perform single-machine basic policy learning. For the second training phase, a second sample set is obtained. In the second training phase, the policy network parameters obtained from the first training phase are inherited. The parameters of the policy network and the value network are trained based on the second sample set so that the initial reinforcement learning model can perform dual-machine collaborative collision avoidance learning. The graph neural network does not participate in the training in the second training phase. For the third training phase, a third sample set is obtained. In the third training phase, the policy network parameters obtained from the second training phase are inherited. Based on the third sample set, the parameters of the graph neural network, the policy network, and the value network are trained so that the initial reinforcement learning model can perform multi-machine cooperative collision avoidance learning to obtain a target reinforcement learning model. The target reinforcement learning model is used for each unloader in the unloader system and determines the control actions for the unloader based on the current state of the unloader. The unloader system includes multiple unloaders.

[0006] Secondly, this application provides a reinforcement learning-based control method for a ship unloader, applied to a ship unloader system. The ship unloader system includes multiple ship unloaders, each of which is equipped with a target reinforcement learning model. The target reinforcement learning model is trained using the reinforcement learning model training method described in the first aspect. The reinforcement learning-based control method for the ship unloader includes: The system collects current status data of the first unloader in the unloader system and receives current neighbor information of neighbor unloaders within the communication range of the first unloader in the unloader system; the first unloader is any unloader in the unloader system. Using the first unloader as the root node and the neighboring unloaders as neighboring nodes, the initial node characteristics of the root node are determined based on the current state data of the first unloader, and the initial node characteristics of the neighboring nodes are determined based on the neighboring machine information. The root node and the neighboring nodes are then connected to construct a local observation map of the first unloader. The initial node features of all nodes in the local observation graph are aggregated by the graph neural network deployed in the target reinforcement learning model of the first ship unloader to obtain the node aggregated features of the first ship unloader. Based on the node aggregation characteristics of the first ship unloader, the current control action for the first ship unloader is determined by the policy network deployed in the target reinforcement learning model of the first ship unloader. Control the first ship unloader to perform the control action.

[0007] Thirdly, this application provides a reinforcement learning model training device, the reinforcement learning model training device comprising: The model building module is used to build an initial reinforcement learning model. The initial reinforcement learning model includes a graph neural network for representing the current state of the ship unloader through feature aggregation, a policy network for outputting the current control action for the current state of the ship unloader, and a value network for estimating the long-term expected reward obtained by the ship unloader in the current state by following the current control action. The first training module is used to acquire a first sample set for the first training phase, and train the parameters of the policy network and the value network based on the first sample set in the first training phase, so that the initial reinforcement learning model can perform single-machine basic policy learning. The second training module is used to obtain a second sample set for the second training phase, inherit the policy network parameters trained in the first training phase in the second training phase, and train the parameters of the policy network and the value network based on the second sample set so that the initial reinforcement learning model can perform dual-machine collaborative collision avoidance learning; the graph neural network does not participate in the training in the second training phase. The third training module is used to obtain a third sample set for the third training stage. In the third training stage, the policy network parameters obtained from the second training stage are inherited. The parameters of the graph neural network, the policy network, and the value network are trained based on the third sample set so that the initial reinforcement learning model can perform multi-machine cooperative collision avoidance learning to obtain a target reinforcement learning model. The target reinforcement learning model is used to deploy each unloader in the unloader system and to determine the control actions for the deployed unloader based on the current state of the deployed unloader. The unloader system includes multiple unloaders.

[0008] Fourthly, this application provides a reinforcement learning-based unloading machine control device, applied to an unloading machine in a unloading machine system. The unloading machine system includes multiple unloading machines, each of which is equipped with a target reinforcement learning model. The target reinforcement learning model is trained using the reinforcement learning model training method described in the first aspect. The reinforcement learning-based unloading machine control device includes: The data acquisition module is used to collect the current status data of the first unloader in the unloader system, and to receive the current neighbor information of neighbor unloaders located within the communication range of the first unloader in the unloader system; the first unloader is any unloader in the unloader system. The graph construction module is used to take the first unloader as the root node and the neighbor unloader as the neighbor node, determine the initial node characteristics of the root node based on the current state data of the first unloader, determine the initial node characteristics of the neighbor node based on the neighbor machine information, and connect the root node and the neighbor node to construct a local observation graph of the first unloader. The feature aggregation module is used to aggregate the initial node features of all nodes in the local observation graph through the graph neural network deployed in the target reinforcement learning model of the first ship unloader, so as to obtain the node aggregated features of the first ship unloader. An action output module is used to determine the current control action for the first ship unloader based on the node aggregation features of the first ship unloader and through the policy network deployed in the target reinforcement learning model of the first ship unloader. The control execution module is used to control the first ship unloader to perform the control actions.

[0009] This application provides a reinforcement learning model training method, a reinforcement learning-based ship unloader control method and device. First, an initial reinforcement learning model is constructed, including a graph neural network, a policy network, and a value network. The graph neural network is used to represent the current state of the ship unloader through feature aggregation, the policy network is used to output the current control action based on the current state of the ship unloader, and the value network is used to estimate the long-term expected reward obtained by the ship unloader in the current state by following the current control action. Then, in different training stages, the initial reinforcement learning model is sequentially subjected to single-machine basic policy learning, dual-machine cooperative collision avoidance learning, and multi-machine cooperative collision avoidance learning to train the target reinforcement learning model. This target reinforcement learning model can be deployed on each unloader in the unloader system to determine control actions for each unloader based on its current state. This makes each unloader an independent intelligent agent capable of making behavioral decisions, enabling rapid response at the local level. Compared to a central dispatch system, this reduces response latency and avoids the risk of system-wide failure due to central dispatch system malfunctions. Furthermore, the multi-agent architecture of the unloader system addresses the poor scalability issues caused by the performance bottleneck of the central dispatch system. Additionally, the unloaders in this application can output behavioral decisions adapted to dynamic operating environments based on their real-time states through the target reinforcement learning model. Moreover, the target reinforcement learning model, after undergoing dual-machine collaborative collision avoidance learning and multi-machine collaborative collision avoidance learning, can adapt to flexible and varied multi-machine collaborative scenarios, improving the efficiency of multi-machine collaborative operations from a system perspective and achieving efficient multi-machine cooperative operations. Attached Figure Description

[0010] Figure 1 A flowchart illustrating a reinforcement learning model training method provided in an embodiment of this application; Figure 2 A flowchart illustrating a reinforcement learning-based control method for a ship unloader, as provided in an embodiment of this application; Figure 3 This is a schematic diagram of the functional modules of a reinforcement learning model training device provided in an embodiment of this application; Figure 4 This is a schematic diagram of the functional modules of a ship unloader control device based on reinforcement learning, provided in one embodiment of this application. Detailed Implementation

[0011] The present application will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0012] In one exemplary embodiment, such as Figure 1 As shown, a reinforcement learning model training method is provided. This method is executed by a computer device, specifically a terminal or server, or both. The terminal can be, but is not limited to, various desktop computers, laptops, and tablets. The server can be a standalone server, a server cluster consisting of multiple servers, or a cloud server.

[0013] In the embodiments of this application, such as Figure 1 As shown, the reinforcement learning model training method includes the following steps 101 to 105. Wherein: Step 101: Construct an initial reinforcement learning model. The initial reinforcement learning model includes a graph neural network for representing the current state of the unloader through feature aggregation, a policy network for outputting the current control action for the current state of the unloader, and a value network for estimating the long-term expected reward obtained by the unloader in the current state by following the current control action.

[0014] In this step, the state space, action space, and reward function are first defined. The state space represents the operating space of the ship unloader, the action space represents its policy space, and the reward function is used for policy updates, guiding the reinforcement learning model's learning. The reward function scores the current control action (i.e., the current policy), determining the reward value of the current control action, thereby quantifying the consequence of the current control action. In this embodiment, the reward function is a multi-objective reward function.

[0015] In this step, an initial reinforcement learning model can be constructed, which includes a graph neural network, a policy network, and a value network. The output layer of the graph neural network is connected to the input layer of the policy network and the input layer of the value network. The dimension of the input layer of the graph neural network is the dimension of the state space mentioned above, and the dimension of the output layer of the policy network is the dimension of the action space mentioned above.

[0016] The graph neural network is used to represent the current state of the ship unloader through feature aggregation, thereby strengthening the state representation of the ship unloader; the policy network is used to output the current control action for the current state of the ship unloader, thereby providing the ship unloader with a control policy adapted to the current state; the value network is used to estimate the long-term expected reward obtained by the ship unloader in the current state by following the current control policy. The long-term expected reward is used to measure the performance of the policy network, so that the policy network is optimized in the direction of increasing the long-term expected reward. The long-term expected reward is the cumulative result of the reward value from the present to the future.

[0017] Optionally, the state space may include the following information: the current position information of the ship unloader, the current speed information of the ship unloader, the current grab information of the ship unloader, the current predicted trajectory of the ship unloader for multiple future time steps, the current task target information of the ship unloader, and the current environmental state information of the ship unloader.

[0018] Optionally, the action space may include the following information: the trolley displacement increment command of the ship unloader along the track direction, the trolley displacement increment command of the ship unloader along the direction perpendicular to the track direction, the grab lifting speed command of the ship unloader, and the grab grab or release control command of the ship unloader.

[0019] Optionally, the multi-objective reward function R t =w1·R task +w2·R collision +w3·R energy +w4·R coop Where w1 is the task completion term (R) in the multi-objective reward function. task The corresponding weight coefficients, w2 is the collision avoidance safety term (R) in the multi-objective reward function. collision The corresponding weight coefficient, w3 is the energy consumption term (R) in the multi-objective reward function. energy The corresponding weight coefficients, w4 is the cooperative behavior term (R) in the multi-objective reward function. coop The corresponding weight coefficients, w1+w2+w3+w4=1, represent the weighted summation of multiple components in the multi-objective reward function. Optionally, the meaning and quantification method of these multiple components are as follows: Task completion rate (R) task When the unloader successfully completes a grab and delivers the material to the target hopper, it is counted as a valid grab and assigned a fixed positive reward, R. task =+1. Otherwise R task =0.

[0020] Collision avoidance safety items (R collision This is a penalty item. If at any time t the distance d between the two unloaders... t Less than the preset safe distance threshold d safe (e.g., 5 meters), then a collision is considered to occur, and R collision A large negative constant (such as -10) can be chosen. If d t ≥d safe Then R collision =0. This design aims to strictly prohibit unsafe behavior.

[0021] Energy consumption term (R) energyThis is a penalty item to encourage energy conservation. Its value is negative, and the calculation formula is Renergy=-η·E, where E is the total energy consumption of the unloader's motor system within a decision cycle (in kW·s), and η is the energy penalty coefficient (e.g., 1.0), used to adjust the weight of energy consumption in the total reward.

[0022] Collaborative Behavior Items (R) coop This is a positive reward used to quantify and guide collaboration between ship unloaders. Its value is based on a comprehensive quantification of multiple rules, which may include, for example: —Proactive yielding: If unloader A slows down or stops to make way for unloader B, which is performing a higher priority task, unloader A receives a yielding reward r. yield (e.g., +0.5).

[0023] —Path Adjustment Priority: Assign a global priority to each task (e.g., emergency repair tasks are prioritized over regular loading and unloading). If the ship unloader adjusts its path to prioritize a higher-priority task, it will receive a priority reward r. priority (e.g., +0.3).

[0024] —Task Continuation: If, after unloader A completes its current task, its position or status allows unloader B to seamlessly continue the next stage of the task (such as cleaning the cargo hold), then unloader A receives a continuation bonus r. succession (e.g., +0.2).

[0025] R coop The value of is the weighted sum of the individual rewards; for example, R coop =β1·r yield +β2·r priority +β3·r succession Weight β i It can be adjusted according to the operation mode, and is usually between [0,1].

[0026] In an alternative embodiment, an attention mechanism can be introduced into the graph neural network to perform feature aggregation. Specifically, the graph neural network can employ a Graph Attention Network (GAT) mechanism for feature aggregation. The attention mechanism dynamically generates differentiated weight coefficients for the root node and its neighboring nodes by calculating the correlation between nodes (in this embodiment, a ship unloader is considered as a node), and then performs weighted fusion, thereby selectively focusing on key interaction information.

[0027] Specifically, the input to the graph neural network includes initial node features representing the current state of the unloader and initial node features representing the current states of neighboring unloaders. The graph neural network can be used to generate attention weights for the unloader and its neighboring unloaders based on an attention mechanism. Then, based on these attention weights, a weighted sum is performed on the initial node features representing the current state of the unloader and the initial node features representing the current states of its neighboring unloaders to obtain the aggregated node features representing the current state of the unloader.

[0028] Among them, the neighboring ship unloader is the ship unloader located within the communication range of the ship unloader.

[0029] Optionally, the initial features of the nodes representing the current state of the ship unloader (i.e., the state space) may include the following information: the current position information of the ship unloader, the current speed information of the ship unloader, the current grab information of the ship unloader, the current predicted trajectory of the ship unloader for multiple future time steps (which can be predicted by a dynamic model), the current task target information of the ship unloader, and the current environmental state information of the ship unloader.

[0030] In this embodiment, each ship unloader can be regarded as an independent intelligent agent, and selective information interaction can be achieved through the local observation map of each ship unloader.

[0031] First, each ship unloader is treated as an independent intelligent agent, and a local observation graph is constructed with the ship unloader itself as the root node. Other ship unloaders within the communication range of the ship unloader are regarded as neighbor nodes, and the neighbor nodes are connected to the root node through edges.

[0032] A graph neural network is used to aggregate the features of all nodes (including the root node and its neighboring nodes) in the local observation graph. The weight coefficients of each node are generated by combining the attention mechanism. The weighted fusion is used to update the root node features. The updated root node features are used as the input features of the policy network to achieve selective information interaction.

[0033] The feature aggregation process of a graph neural network is as follows: For the root node i in the local observation graph N(i), the node aggregation feature h of the root node i is... i The formula is obtained by performing attention-weighted summation on the initial features of all nodes j (j∈N(i), where N(i) includes the root node i itself) in the local observation graph. .

[0034] in: —h jLet N(i) represent the initial node features of node j in the local observation graph N(i). The initial node features of each node can be determined based on the node's state data (such as position, velocity, etc.) through normalization and fusion operations.

[0035] —W is a trainable weight matrix used to weight h j Perform a linear transformation.

[0036] —α ij It is the attention coefficient between root node i and node j.

[0037] —σ(·) represents a nonlinear activation function (such as ELU).

[0038] In the above formula, the attention coefficient α ij α represents the importance of information about node j to root node i. ij It is calculated using the following formula: .

[0039] in: —— This indicates a feature splicing operation.

[0040] —a is a trainable attention vector.

[0041] —LeakyReLU is the activation function.

[0042] —— Let be the initial node feature of any node k in the local observation graph N(i).

[0043] —Attention coefficient α ij After softmax normalization, .

[0044] Optionally, the policy network is specifically used to output the current control action for the ship unloader based on the node aggregation features representing the current state of the ship unloader. Optionally, the control action (i.e., the action space) may include the following information: the ship unloader's trolley displacement increment command along the track direction, the ship unloader's trolley displacement increment command along the direction perpendicular to the track direction, the ship unloader's grab bucket lifting speed command, and the ship unloader's grab bucket grabbing or releasing control command.

[0045] Optionally, for stable training, the value network can adopt a dual-network structure, including an evaluation network (or online value network) and a target network (or target value network). The parameters of the evaluation network are updated in real time, while the target network is a lagged copy of its parameters, which is periodically (e.g., every 100 updates) synchronized with the parameters of the evaluation network to compute a more stable temporal difference (TD) target value.

[0046] In an optional example of the initial reinforcement learning model, the policy network can be a three-layer fully connected network: an input layer, hidden layers, and an output layer. The input layer has a dimension of 128 (after state normalization), the hidden layers use 256-dimensional and 128-dimensional neurons activated by ReLU, respectively, and the output layer has the dimension of the action space (e.g., 4-dimensional continuous actions). The output of the policy network is finally mapped to the [-1, 1] interval using the tanh activation function. The graph neural network can use a two-layer GAT structure, each layer containing 8 attention heads with a feature dimension of 64. The attention weights are normalized using softmax and then used for weighted aggregation of neighbor features. The value network shares the first two layers with the policy network (or, for a dual-network value network, the evaluation network shares the first two layers with the policy network), and the last layer outputs the long-term expected reward (or state value estimate).

[0047] Step 102: Obtain different sample sets for different training stages of the initial reinforcement learning model; different sample sets are used to enable the initial reinforcement learning model to learn different capabilities in stages.

[0048] The initial reinforcement learning model described above can be trained in stages, with different capabilities learned in each stage. In this step, different sample sets are obtained for each training stage of the initial reinforcement learning model. The first sample set is used for the first training stage, the second sample set for the second training stage, and the third sample set for the third training stage. Each sample set will be described in detail in the section on each training stage below.

[0049] Optionally, before the initial reinforcement learning model learns different capabilities in stages, the historical operation data of the unloader generated by traditional planning algorithms or manual operation can be used as pre-training data (i.e., state-action sequences) to perform supervised pre-training on the initial reinforcement learning model, thereby initializing the policy network parameters so that its initial policy is close to the feasible solution region, providing a high-quality starting point for subsequent staged training, and significantly accelerating the model convergence process.

[0050] Step 103: In the first training phase, the parameters of the policy network and the value network are trained based on the first sample set so that the initial reinforcement learning model can perform single-machine basic policy learning.

[0051] Optionally, the phased training of the initial reinforcement learning model can be simulation training, i.e., training in a simulation environment. Alternatively, the initial reinforcement learning model can be trained offline in a Unity-based simulation training scenario using the Proximal Policy Optimization (PPO) algorithm. The training process employs a progressive, multi-stage training strategy, with each stage differing in training scenario, training data, reward function emphasis, and network parameter update strategy. A virtual unloader can be set up in the simulation training environment, and the initial reinforcement learning model can be deployed within the virtual unloader.

[0052] The first training phase is used to enable the initial reinforcement learning model to learn basic policies on a single machine, so that the ship unloader can master basic operational actions and task execution capabilities.

[0053] The simulation training scenario in the first training phase may include: a simulation scenario in which a single virtual ship unloader performs a single task (optionally, it may be a task targeting a static task objective, such as a task targeting a single cargo stack); in the simulation training scenario in the first training phase, the initial reinforcement learning model is deployed on the single virtual ship unloader.

[0054] The training data for the first training phase is the first sample set. The samples in the first sample set are derived from the interaction between the ship unloader and the environment, and are used to enable the initial reinforcement learning model to learn basic actions such as moving towards the target, accurately grasping and releasing.

[0055] The reward function in the first training phase includes a weighted sum of the following terms: a task completion term representing the virtual ship unloader's task performance, and an energy consumption term representing the energy consumption of the virtual ship unloader performing its tasks. In the first training phase, the reward function uses a multi-objective reward function R. t =w1·R task +w2·R collision +w3·R energy +w4·R coop The modification involves setting w1 and w3 to their respective preset values ​​(e.g., w1=0.4, w3=0.2), and setting w2 and w4 to 0. The reward function for the first training phase can then be expressed as R. t1 =w1·R task +w3·R energy .

[0056] In the first training phase, the initial reinforcement learning model can inherit the pre-trained policy network parameters, and the policy network and value network are trained based on these parameters. It should be noted that the graph neural network does not participate in the training during this first phase.

[0057] The loss function for the first training phase can be the PPO loss function, which is used to enable the initial reinforcement learning model to learn to complete the basic job cycle.

[0058] PPO loss function L PPO (θ) consists of three parts: policy loss, value function error, and entropy reward, as shown in the following formula: .

[0059] in: —θ: Policy network parameters.

[0060] — : indicates the expectation of the sampled batch of data.

[0061] — The loss is for the clipped strategy. The importance sampling ratio, This indicates the current policy network (the current policy network parameters are...). In state s t Take control action a t The probability of using the latest updated policy parameter is used to represent the probability of using the latest updated policy parameter. The generated strategy πθ” controls action a t The probability assessment reflects the impact of the policy update on control action a. t The degree of preference, This represents the old policy network (i.e., the policy network before the last iteration update, with parameters of the old policy network). In state s t Take control action a t The probability is used as a measure of the relationship between the old and new policies for the same state-action pair (s). t ,a t The benchmark for the probability difference of ) ε is the estimated value of the advantage function, and ε is the clip range hyperparameter (usually 0.1 or 0.2) used to limit the step size of policy updates and ensure training stability.

[0062] — Value function loss (mean squared error). V θ (s t ) for the value network pair of states s t The value estimate, The target value (usually calculated using methods such as GAE).

[0063] —S[πθ](s t ) represents the policy πθ in state s t The entropy below is used to encourage exploration.

[0064] —c1 and c2 are preset coefficients used to adjust the weights of value function loss and entropy reward in the total loss, respectively.

[0065] Step 104: In the second training phase, inherit the policy network parameters obtained from the first training phase, and train the parameters of the policy network and value network based on the second sample set, so that the initial reinforcement learning model can perform dual-machine collaborative collision avoidance learning.

[0066] The second training phase is used to enable the initial reinforcement learning model to learn dual-machine cooperative collision avoidance, that is, to learn cooperative strategies in a dual-machine interactive collision avoidance scenario, so that the initial reinforcement learning model can learn dynamic collision avoidance and simple cooperation.

[0067] The simulation training scenario in the second training phase may include: two virtual ship unloaders performing their respective tasks (optionally, a dynamically updated task objective can be set for each virtual ship unloader to conform to the task objective in the real scenario) and the working areas of the two virtual ship unloaders overlap; in the simulation training scenario of the second training phase, the initial reinforcement learning model is deployed in each of the two virtual ship unloaders.

[0068] The training data for the second training phase is the second sample set. The samples in the second sample set are derived from the experience (state-action pairs) generated by the interaction between the two machines, especially the state-action pairs when the two machines are at close range.

[0069] The reward function in the second training phase includes a weighted sum of the following terms: a task completion term representing the virtual ship unloader's task completion status; an energy consumption term representing the energy consumption of the virtual ship unloader in performing its tasks; a collision avoidance safety term representing the collision risk between the two virtual ship unloaders; and a cooperative behavior term representing the cooperation between the two virtual ship unloaders. In the second training phase, the reward function employs a multi-objective reward function R. t =w1·R task +w2·R collision +w3·R energy +w4·R coop The training process is modified so that w1 and w3 remain consistent with those in the first training phase, while w2 and w4 are set to their respective preset values, such as w2=0.3 and w4=0.1. The reward function for the second training phase can be expressed as R. t2 =w1·R task +w2·R collision +w3·R energy +w4'·R coop For ease of distinction, the weight of the collision avoidance safety term in the second training phase is denoted as w4', and the weight of the collision avoidance safety term in the subsequent third training phase is denoted as w4".

[0070] In the second training phase, the initial reinforcement learning model can inherit the policy network parameters obtained in the first training phase, and the policy network and value network can be trained based on these parameters. It should be noted that the graph neural network does not participate in the training in the second training phase.

[0071] The loss function in the second training phase is adjusted based on the PPO loss function described above, where the advantage function ( The calculation will fully consider the impact of the presence of another ship unloader on the long-term expected return, guiding the policy network to learn to actively avoid and cooperate simply with the two machines.

[0072] Step 105: In the third training phase, the policy network parameters obtained from the second training phase are inherited, and the parameters of the graph neural network, policy network, and value network are trained based on the third sample set, so that the initial reinforcement learning model can perform multi-machine cooperative collision avoidance learning to obtain the target reinforcement learning model. The target reinforcement learning model is used for each unloader in the unloader system, and the control actions for the unloader are determined according to the current state of the unloader. The unloader system includes multiple unloaders.

[0073] The third training phase is used to enable the initial reinforcement learning model to perform multi-machine (three or more machines, including three machines) collaborative collision avoidance learning. That is, to learn multi-machine collaborative strategies in complex operation scenarios consisting of three or more unloading machines, so that the initial reinforcement learning model can learn to achieve global path collaborative planning and autonomous conflict resolution in a dynamic environment.

[0074] The simulation training scenario in the third training phase may include: three or more virtual ship unloaders performing their respective tasks, with at least two of the virtual ship unloaders having overlapping operating areas; in the simulation training scenario of the third training phase, the initial reinforcement learning model is deployed in each of the three or more virtual ship unloaders.

[0075] The training data for the third training phase is the third sample set. The samples in the third sample set cover the competition and cooperation relationships among multiple machines, reflecting the actions (state-action pairs) of the unloader under multi-machine interaction.

[0076] The reward function in the third training phase includes a weighted sum of the following terms: a task completion term representing the virtual ship unloader's task completion status; an energy consumption term representing the energy consumption of the virtual ship unloader in performing its tasks; a collision avoidance safety term representing the collision risk between the virtual ship unloader and any other virtual ship unloader; and a cooperative behavior term representing the cooperation among three or more virtual ship unloaders. The weight of the cooperative behavior term in the third training phase is greater than that in the second training phase. In the third training phase, the reward function uses a multi-objective reward function R. t =w1·Rtask +w2·R collision +w3·R energy +w4·R coop To modify the system, increase the value of w4 (i.e., w4” is greater than w4', for example, w4” = 0.4), and adjust w1, w2, and w3 accordingly to ensure that w1, w2, and w3 are all non-zero and w1 + w2 + w3 + w4 = 1. Then adjust R... coop The quantitative rules are strengthened by adding or modifying existing rules for multi-machine scenarios (for example, the proactive yielding rule can be modified to: if unloader A yields its path to an unloader performing a higher priority task by slowing down or stopping, unloader A receives a yielding reward r). yield (e.g., +0.1), for example, if ship unloader A yields its path to 3 ship unloaders this time, then ship unloader A will receive a yielding reward r. yield =0.3), the reward function for the third training phase can be expressed as R t3 =w1·R task +w2·R collision +w3·R energy +w4”·R coop .

[0077] In the third training phase, the initial reinforcement learning model can inherit the policy network parameters obtained from the third training phase, and on this basis, the graph neural network, policy network, and value network are trained.

[0078] The graph neural network participates in the training in the third training stage, making decisions by aggregating features through nodes. The loss function in the third training stage still adopts the loss function of the second training stage, but due to the activation of the graph neural network, the loss function in the third training stage will be based on this global state that includes collaborative perception for model evaluation.

[0079] After training through the above three stages, a target reinforcement learning model can be obtained. Each ship unloader in the ship unloader system (including multiple ship unloaders) is deployed with this target reinforcement learning model to determine the current control action of the ship unloader.

[0080] The reinforcement learning model training method provided in this application first constructs an initial reinforcement learning model, including a graph neural network, a policy network, and a value network. The graph neural network is used to represent the current state of the unloader through feature aggregation; the policy network outputs the current control action based on the current state of the unloader; and the value network estimates the long-term expected reward obtained by the unloader in the current state by following the current control action. Then, in different training stages, the initial reinforcement learning model is sequentially trained through single-machine basic policy learning, dual-machine cooperative collision avoidance learning, and multi-machine cooperative collision avoidance learning to obtain the target reinforcement learning model. This target reinforcement learning model can be deployed on each unloader in the unloader system to determine the control action for each deployed unloader based on its current state. This makes each unloader in the system an independent intelligent agent capable of making behavioral decisions, achieving rapid response at the local level. Compared to a scheme where a central scheduling system makes decisions, this reduces response latency and avoids the risk of system-wide failure due to a central scheduling system malfunction. Furthermore, the multi-agent architecture of the unloader system can solve the problem of poor scalability caused by the performance bottleneck of the central scheduling system. In addition, the ship unloader in this application can output behavioral decisions adapted to the dynamic working environment based on the real-time status of the ship unloader through a target reinforcement learning model. At the same time, the target reinforcement learning model, which has undergone dual-machine collaborative collision avoidance learning and multi-machine collaborative collision avoidance learning, can adapt to flexible and ever-changing multi-machine collaborative scenarios, improve the efficiency of multi-machine collaborative operation from a system perspective, and realize efficient multi-machine cooperative operation.

[0081] In one exemplary embodiment, such as Figure 2 As shown, a reinforcement learning-based control method for a ship unloader is provided, which is executed by the ship unloader.

[0082] In this embodiment, the reinforcement learning-based unloader control method is applied to the unloader in a ship unloader system. The ship unloader system includes multiple unloaders, and each unloader in the system has a target reinforcement learning model deployed in it. This target reinforcement learning model is trained using the reinforcement learning model training method described above. Figure 2 As shown, the reinforcement learning-based ship unloader control method includes the following steps 201 to 205. Wherein: Step 201: Collect the current status data of the first unloader in the unloader system, and receive the current neighbor information of neighbor unloaders within the communication range of the first unloader in the unloader system; the first unloader is any unloader in the unloader system.

[0083] For example, the current status data of the first ship unloader may include: the current position coordinates (x, y, z) of the first ship unloader, and the current speed (V) of the first ship unloader. x V yThe following information is included: the current grab status of the first unloader (open, lifting, etc.); the predicted trajectory of the first unloader for multiple future time steps (e.g., the predicted position coordinates of the first unloader in the next 5 time steps, for a total of 5 predicted position coordinates); the current task target information of the first unloader (e.g., the center coordinates of the current target cargo pile); and the current environmental status information of the first unloader (e.g., the remaining material volume of the current target cargo pile, and / or the hold offset caused by changes in ship attitude when the first unloader performs the current task).

[0084] For example, the first ship unloader can calculate its own Cartesian coordinates in real time through a GNSS (Global Navigation Satellite System) / BeiDou and inertial navigation fusion positioning module, periodically acquire cargo stack point cloud data through a laser 3D scanner, measure ship attitude changes through a ship attitude sensor group, and broadcast and share its own information through a 5G private network or Wi-Fi 6 communication link.

[0085] Within the edge computing unit of the first ship unloader, the multi-source heterogeneous real-time sensing data (i.e., the current state data of the first ship unloader) undergoes preprocessing steps such as time synchronization, coordinate system unification, formatting, and normalization to ultimately form the state vector required for the input dimension of the target reinforcement learning model. This preprocessing process may include the following steps: Time synchronization and alignment: Stamp all state data with a unified timestamp to ensure that state information is based on the same time base.

[0086] Coordinate System 1: All coordinate and attitude data based on the coordinate system are transformed to a three-dimensional Cartesian coordinate system with a fixed point in the working environment as the origin.

[0087] Information extraction and formatting: Extract material distribution heat map information (such as the remaining material volume of the cargo pile) from cargo pile point cloud data; calculate the offset of the hold from ship attitude data; extract information such as the coordinates, speed, and predicted trajectory of the ship and its neighboring ships from positioning data.

[0088] Normalization: Normalize all physical quantities according to a preset range. For example, coordinates are normalized to [0,1] according to the track length, and velocity is normalized to the maximum velocity.

[0089] State vector assembly: The processed data is assembled into a one-dimensional state vector according to the defined order of the state space, which is then used as the input to the target reinforcement learning model.

[0090] The following is a specific example of the fusion processing result: Suppose that at a certain time t, for the first unloader A: Self-state: Normalized coordinates are [0.45, 0.60, 0.30], normalized velocity is [0.02, 0.01], normalized grab bucket state is "open" (coded as 1), normalized result of the predicted trajectory of the first unloader A in the next 3 time steps.

[0091] Task objective: The normalized center coordinates of the current target cargo pile are [0.50, 0.65, 0.32].

[0092] Environmental conditions: The normalized value of the remaining material volume of the current target cargo pile extracted from the heat map is 0.7; the normalized value of the hold offset caused by the ship's attitude is [0.01, -0.005], where the roll angle φ is 0.01 and the pitch angle θ is -0.005.

[0093] Neighbor machine status (broadcast from other unloaders): Received status data packet (i.e. neighbor machine information) broadcast by the second unloader B, with normalized coordinates [0.48, 0.58, 0.35], normalized velocity [0.015, 0.005], and normalized result of the predicted trajectory of the second unloader B for the next 3 steps.

[0094] The above data is concatenated to obtain the state vector s of the first ship unloader A. t .

[0095] Step 202: Take the first unloader as the root node and the neighboring unloaders as the neighboring nodes. Determine the initial node characteristics of the root node based on the current state data of the first unloader, determine the initial node characteristics of the neighboring nodes based on the neighboring machine information, and connect the root node and the neighboring nodes to construct a local observation map of the first unloader.

[0096] In this step, a local observation graph can be constructed with the first unloader as the root node. The nodes and initial features of these nodes in the local observation graph will serve as the basis for feature aggregation in the graph neural network of the target reinforcement learning model.

[0097] Step 203: The initial node features of all nodes in the local observation graph are aggregated by the graph neural network deployed in the target reinforcement learning model of the first ship unloader to obtain the node aggregated features of the first ship unloader.

[0098] In this step, the initial node features of all nodes in the local observation graph can be aggregated by the graph neural network deployed in the target reinforcement learning model of the first ship unloader to obtain the node aggregated features of the first ship unloader. The feature aggregation process can refer to the feature aggregation process of the graph neural network in the reinforcement learning model training method, and will not be repeated here.

[0099] Step 204: Based on the node aggregation features of the first ship unloader, determine the current control action for the first ship unloader through the policy network deployed in the target reinforcement learning model of the first ship unloader.

[0100] The policy network in the objective reinforcement learning model receives the node aggregation features s from the first ship unloader. t ', Output the current control action a for the first unloader t .

[0101] The value network in the objective reinforcement learning model receives the node aggregation features s from the first ship unloader. t ', Output the long-term expected return r that the first ship unloader can obtain by performing the current control action in the current state. t .

[0102] Among them, the control action a output by the policy network t (That is, a one-dimensional action vector) is the normalized value. To drive the actual physical device, the output of the policy network needs to be converted into an engineering control quantity. The conversion process includes inverse normalization, amplitude limiting, and smoothing filtering. Denormalization: The normalized action values ​​output by the policy network, ranging from [-1, 1], are linearly mapped to the actual physical range of each actuator to restore the true control command. For example, the trolley displacement increment command. The actual displacement increment is obtained after inverse normalization. m.

[0103] Limiting processing: The denormalized control commands are input into the limiting module to ensure that their absolute values ​​do not exceed the maximum allowable instantaneous changes of each actuator, in order to prevent overshoot, protect equipment, and ensure operational safety. All action commands must undergo limiting processing.

[0104] Smoothing filtering: To suppress control command jitter caused by fluctuations in the strategy network output or sensor noise, the limited control command u is... in (t) is fed into a first-order low-pass filter for smoothing, resulting in the final execution instruction u sent to the PLC. out (t). Smoothing filtering is performed after amplitude limiting and before the instruction is sent to the PLC. The filter formula is: u out (t)=α·u out (t-1)+(1-α)·u in (t).

[0105] Among them, u out (t) represents the filtered output command at time t, u in(t) is the input command after limiting at time t. Optionally, the smoothing coefficient α is used to balance the response speed and smoothness. Optionally, α is 0.8.

[0106] Step 205: Control the first unloader to perform control actions.

[0107] In this step, control commands for the first ship unloader can be generated based on the engineering control quantities obtained from the output transformation of the policy network, and the first ship unloader can be controlled to execute the control commands, thereby enabling the first ship unloader to realize the control action a output by the policy network. t .

[0108] In an optional embodiment, the current state data of the first unloader includes a first predicted trajectory of the first unloader for multiple future time steps, and the current neighbor information of the neighbor unloaders includes a second predicted trajectory of the neighbor unloaders for multiple future time steps. The reinforcement learning-based unloader control method may further include the following steps: If a trajectory point less than a preset safe distance threshold is detected between the first predicted trajectory and the second predicted trajectory, the first ship unloader is triggered to replan its control actions by using the target reinforcement learning model deployed on the first ship unloader and constrained by the predicted trajectories of neighboring ship unloaders.

[0109] In this embodiment, each ship unloader in the ship unloader system can operate at a fixed period T. c (100ms to 500ms, e.g., 200ms) Broadcast its own status data packet to other unloaders within communication range. The status data packet can be in a predefined binary format and includes the following fields: 1. Timestamp: The moment when the status data packet was generated.

[0110] 2. Machine status: current position coordinates, current speed, grab status, etc.

[0111] 3. Predicted trajectory: The future trajectory T predicted based on the dynamic model in the current state. p There are 3 (e.g., 3, 5, etc.) time steps (with a time step interval of T). c The sequence of coordinates of the trajectory points .

[0112] The first unloader A can perform spatial cross-detection between the received predicted trajectory of the second unloader B and its own predicted trajectory. If a distance d exists between the trajectory points... t Less than the preset safe distance threshold d safe The intersection of the trajectory points triggers the local target reinforcement learning model of the first unloader A to regenerate the control action.

[0113] Specifically, after receiving the status data packet from the second ship unloader B, the first ship unloader A can execute the following dynamic collision avoidance procedure: Trajectory cross-detection: Detects the predicted trajectories from received neighbors. The predicted trajectory of the first ship unloader A itself Perform time-step spatial distance calculations.

[0114] Risk assessment: If there exists any future time step t∈[1,T] p The Euclidean distance between the two trajectory points is satisfied. Less than the preset safe distance threshold d safe (d) safe If the distance is 5.0m, then it is determined that there is a potential collision risk between the first unloader A and the second unloader B.

[0115] Control Action Replanning: The first unloader A (i.e., the risk detector) is triggered to replan its path, and its local target reinforcement learning model regenerates the control actions. During replanning, the predicted trajectory of the second unloader B is used as a dynamic obstacle constraint to replan the control actions for the first unloader. That is, the control target reinforcement learning model replans the control actions in a direction that moves the first unloader A away from the predicted trajectory of the second unloader B. The second unloader B maintains its original control actions within this cycle to avoid decision oscillations.

[0116] Command execution: The control actions regenerated by the first ship unloader A are output by the strategy network and then undergo a series of post-processing steps, including inverse normalization, amplitude limiting (to ensure that the command is within the maximum capability range of the physical actuator), and first-order low-pass digital filtering, to be converted into analog or pulse commands that can directly drive the servo motors. Finally, the commands are sent to the programmable logic controller (PLC) of the first ship unloader A for execution, controlling the movement of the trolley, carriage, and hoisting mechanism of the first ship unloader A.

[0117] In an optional embodiment, the reinforcement learning-based ship unloader control method may further include the following steps: The material distribution of the task performed by the first ship unloader is monitored. If the rate of change of the material distribution exceeds the preset rate of change threshold, the first ship unloader is triggered to replan its control actions by using the rate of change of the material distribution as a constraint through the target reinforcement learning model deployed on the first ship unloader. Alternatively, the reinforcement learning-based ship unloader control method also includes: monitoring the ship attitude of the task performed by the first ship unloader; if the ship attitude deviation exceeds a preset deviation threshold, the first ship unloader is triggered to replan its control actions by using the ship attitude deviation as a constraint through the target reinforcement learning model deployed on the first ship unloader.

[0118] In this embodiment, during the operation of the ship unloader, the environmental perception system of the ship unloader periodically updates the perception data to provide real-time information: Material distribution perception: Optionally, a laser 3D scanner deployed on the unloader gantry can perform a panoramic scan of the ship's cargo hold at fixed intervals (e.g., 30 seconds) to generate current material distribution point cloud data. This material distribution point cloud data is then processed to update the global material distribution heat map.

[0119] Ship attitude sensing: Inclination sensors and draft sensors installed on the hull jointly measure the ship's attitude deviation (i.e., the cabin offset mentioned above) at fixed intervals (e.g., 10 seconds). The ship's attitude deviation is defined as the Euclidean norm of the current ship attitude (including roll angle φ and pitch angle θ) relative to the initial attitude (φ0, θ0) at the start of the operation, that is: .

[0120] The deviation of a ship's attitude comprehensively reflects the changes in its roll and pitch, and the unit of the deviation is degrees (°).

[0121] In this embodiment, control action replanning can be triggered by one or more of the following events: Event 1 (Material Distribution Change): If the rate of change of the current material distribution heatmap compared with the previous period (calculated by the area ratio of regions where the pixel gray value changes by more than 50%) exceeds a preset threshold (e.g., 15%), the target reinforcement learning model can be triggered to replan the control actions of the first ship unloader with the rate of change of material distribution as a constraint. That is, the control target reinforcement learning model replans the control actions in the direction that reduces the rate of change of material distribution.

[0122] Event 2 (Ship Attitude Deviation): When the ship's attitude deviation Δ attitude If the value exceeds a preset threshold (e.g., 0.5°), the target reinforcement learning model can be triggered to replan the control actions of the first unloader with the ship attitude offset as a constraint. That is, the control target reinforcement learning model replans the control actions in the direction that reduces the ship attitude offset.

[0123] In an optional embodiment, when the first unloader detects an anomaly in the neighboring unloader, the first unloader can switch to a safe operation mode. In the safe operation mode, the first unloader can adjust its own operation strategy according to the neighboring machine information received last time and the preset safety rules, so as to achieve degraded fault-tolerant operation of the system.

[0124] For example, if a ship unloader fails to send a status data packet or fails data verification for three consecutive communication cycles, the other ship unloaders can determine that it has entered a disconnected state and automatically switch to a safe operation mode. In the safe operation mode, the system adds spatial avoidance constraints to the area where the original predicted trajectory of the disconnected ship unloader was located in the local policy based on the neighbor information of the last received disconnected ship unloader, and performs deceleration, pause, or detour operations to prevent collisions.

[0125] For example, in safe operating mode, the ship unloader can execute one or more of the following preset safety rules: Immediately cease moving towards the area where the missing ship unloader is predicted to be located; When the current speed exceeds 1 m / s (adjustable parameter), reduce the speed to 0.3 m / s. 2 Acceleration (adjustable parameter) decelerates to 1 m / s (adjustable parameter); Maintain a safe distance of at least 5 meters (adjustable parameter) from the last position of the lost ship unloader; If the original predicted trajectory crosses the area where the predicted trajectory of the missing ship unloader is located, then it should deviate laterally by at least 3 meters (adjustable parameter) along the track to bypass it; Send a fault alarm to the central monitoring platform and wait for manual confirmation or restoration of communication.

[0126] Optionally, during ship unloader operations, empirical data of the ship unloader can be collected, and all ship unloader empirical data (s) can be collected. t ,a t ,r t ,s t+1 ) can be uniformly stored in a shared experience replay pool, where s t This represents the state of the ship unloader at time t (after feature aggregation), a t This represents the control action of the ship unloader at time t, r t s represents the reward value of the ship unloader at time t (calculated using a reward function). t+1 This represents the state of the ship unloader at time t+1 (after feature aggregation).

[0127] Once sufficient data has been accumulated (e.g., 10,000 records), initiate the online fine-tuning process: use a low learning rate (e.g., 1×10⁻⁶). -5 Incremental training is performed on the deployed target reinforcement learning model, with the training objective being the PPO loss function. Fine-tuning can be conducted at night or during off-peak hours to avoid impacting normal operations. Once the updated model is verified to be error-free, it replaces the original model, thus forming a closed-loop iterative optimization mechanism of "deployment-collection-optimization-redeployment". Incremental training can randomly sample batches of data from the ship unloader's experience data, rather than sampling in chronological order, to break the temporal correlation between data and improve training stability.

[0128] In addition, detailed operation logs of each ship unloader can be continuously recorded, including task cycle time, motor energy consumption, actual operating trajectory, collision avoidance event records, etc. This data can be stored in the cloud or edge server for subsequent offline iterative optimization and online fine-tuning of the model.

[0129] In this embodiment, all ship unloaders run the trained target reinforcement learning model on the local edge computing unit, forming a decentralized self-organizing collaborative network with high robustness, strong adaptability and autonomous decision-making ability, effectively solving the problems of response delay, single point of failure and poor dynamic adaptability under the traditional centralized scheduling architecture.

[0130] The reinforcement learning-based control method for ship unloaders provided in this application first deploys a target reinforcement learning model trained using the aforementioned reinforcement learning model training method in each ship unloader of the system. During operation, the ship unloader collects its own current state data and receives neighbor information from neighboring ship unloaders within its communication range. A local observation graph is constructed with the ship unloader as the root node and neighboring ship unloaders as neighbor nodes. Then, the graph neural network in the target reinforcement learning model aggregates the features of all nodes in the local observation graph. The aggregated node features are input into the policy network in the target reinforcement learning model to obtain the current control action for the ship unloader. This makes each ship unloader in the system an independent intelligent agent capable of making behavioral decisions, enabling rapid responses to its own behavioral decisions. Compared to the scheme of making decisions through a central dispatch system, this reduces response latency and avoids the risk of system-wide failure due to a central dispatch system malfunction. In addition, the multi-agent architecture of the ship unloader system can solve the problem of poor scalability caused by the performance bottleneck of the central dispatch system. In addition, the ship unloader in this application can output behavioral decisions adapted to the dynamic working environment based on the real-time status of the ship unloader through a target reinforcement learning model. At the same time, the target reinforcement learning model, which has undergone dual-machine collaborative collision avoidance learning and multi-machine collaborative collision avoidance learning, can adapt to flexible and ever-changing multi-machine collaborative scenarios, improve the efficiency of multi-machine collaborative operation from a system perspective, and realize efficient multi-machine cooperative operation.

[0131] This application also provides a reinforcement learning model training apparatus for implementing the reinforcement learning model training method described above. The solution provided by this reinforcement learning model training apparatus is similar to the implementation scheme described in the reinforcement learning model training method above. Therefore, the specific limitations in one or more embodiments of the reinforcement learning model training apparatus provided below can be found in the limitations of the reinforcement learning model training method described above, and will not be repeated here.

[0132] In one exemplary embodiment, such as Figure 3As shown, a reinforcement learning model training device is provided, which includes: The model building module is used to build an initial reinforcement learning model. The initial reinforcement learning model includes a graph neural network for representing the current state of the ship unloader through feature aggregation, a policy network for outputting the current control action for the current state of the ship unloader, and a value network for estimating the long-term expected reward obtained by the ship unloader in the current state by following the current control action. The first training module is used to acquire a first sample set for the first training phase, and train the parameters of the policy network and the value network based on the first sample set in the first training phase, so that the initial reinforcement learning model can perform single-machine basic policy learning. The second training module is used to obtain a second sample set for the second training phase, inherit the policy network parameters trained in the first training phase in the second training phase, and train the parameters of the policy network and the value network based on the second sample set so that the initial reinforcement learning model can perform dual-machine collaborative collision avoidance learning; the graph neural network does not participate in the training in the second training phase. The third training module is used to obtain a third sample set for the third training stage. In the third training stage, the policy network parameters obtained from the second training stage are inherited. The parameters of the graph neural network, the policy network, and the value network are trained based on the third sample set so that the initial reinforcement learning model can perform multi-machine cooperative collision avoidance learning to obtain a target reinforcement learning model. The target reinforcement learning model is used to deploy each unloader in the unloader system and to determine the control actions for the deployed unloader based on the current state of the deployed unloader. The unloader system includes multiple unloaders.

[0133] Optionally, in some embodiments of this application, the simulation training scenario of the first training phase includes: a simulation scenario in which a single virtual ship unloader performs a single task; in the simulation training scenario of the first training phase, the initial reinforcement learning model is deployed on the single virtual ship unloader. The simulation training scenario of the second training phase includes: two virtual ship unloaders performing their respective tasks and the working areas of the two virtual ship unloaders overlapping; in the simulation training scenario of the second training phase, the initial reinforcement learning model is deployed in each of the two virtual ship unloaders. The simulation training scenario of the third training phase includes: three or more virtual ship unloaders performing their respective tasks, and the working areas of at least two of the virtual ship unloaders overlapping; in the simulation training scenario of the third training phase, the initial reinforcement learning model is deployed in each of the three or more virtual ship unloaders.

[0134] Optionally, in some embodiments of this application, the reward function of the first training phase includes a weighted sum of the following items: a task completion degree item to characterize the completion status of the virtual ship unloader's task, and an energy consumption item to characterize the energy consumption of the virtual ship unloader in performing the task. The reward function for the second training phase includes a weighted sum of the following items: a task completion rate item representing the completion status of the virtual ship unloader's task, an energy consumption item representing the energy consumption of the virtual ship unloader in performing the task, a collision avoidance safety item representing the collision risk between the two virtual ship unloaders, and a cooperative behavior item representing the cooperation between the two virtual ship unloaders. The reward function of the third training phase includes a weighted sum of the following items: a task completion rate item representing the completion status of the virtual ship unloader's tasks; an energy consumption item representing the energy consumption of the virtual ship unloader in performing its tasks; a collision avoidance safety item representing the collision risk between the virtual ship unloader and any other virtual ship unloader; and a cooperative behavior item representing the cooperation among the three or more virtual ship unloaders. The weight of the cooperative behavior item in the third training phase is greater than the weight of the cooperative behavior item in the second training phase.

[0135] Optionally, in some embodiments of this application, the graph neural network is specifically used to generate the attention weights of the unloader and the attention weights of the neighboring unloaders based on the attention mechanism, and to perform a weighted summation of the initial node features representing the current state of the unloader and the initial node features representing the current state of the neighboring unloaders based on the attention weights of the unloader and the attention weights of the neighboring unloaders to obtain the node aggregation features representing the current state of the unloader; The neighboring ship unloader is a ship unloader located within the communication range of the ship unloader; The initial features of the node representing the current state of the ship unloader include the following information: the current position information of the ship unloader, the current speed information of the ship unloader, the current grab information of the ship unloader, the current predicted trajectory of the ship unloader for multiple future time steps, the current task target information of the ship unloader, and the current environmental state information of the ship unloader.

[0136] Optionally, in some embodiments of this application, the policy network is specifically used to output the current control action for the unloader based on the node aggregation features representing the current state of the unloader. The control actions include the following information: the trolley displacement increment command of the ship unloader along the track direction, the trolley displacement increment command of the ship unloader along the direction perpendicular to the track direction, the grab bucket lifting speed command of the ship unloader, and the grab bucket grabbing or releasing control command of the ship unloader.

[0137] This application also provides a reinforcement learning-based ship unloader control device for implementing the reinforcement learning-based ship unloader control method described above. The solution provided by this reinforcement learning-based ship unloader control device is similar to the implementation scheme described in the reinforcement learning-based ship unloader control method. Therefore, the specific limitations in one or more embodiments of the reinforcement learning-based ship unloader control device provided below can be found in the limitations of the reinforcement learning-based ship unloader control method described above, and will not be repeated here.

[0138] In one exemplary embodiment, such as Figure 4 As shown, a reinforcement learning-based ship unloader control device is provided, applied to a ship unloader system. The ship unloader system includes multiple ship unloaders, each of which is equipped with a target reinforcement learning model. The target reinforcement learning model is trained using the reinforcement learning model training method described in the first aspect. The reinforcement learning-based ship unloader control device includes: The data acquisition module is used to collect the current status data of the first unloader in the unloader system, and to receive the current neighbor information of neighbor unloaders located within the communication range of the first unloader in the unloader system; the first unloader is any unloader in the unloader system. The graph construction module is used to take the first unloader as the root node and the neighbor unloader as the neighbor node, determine the initial node characteristics of the root node based on the current state data of the first unloader, determine the initial node characteristics of the neighbor node based on the neighbor machine information, and connect the root node and the neighbor node to construct a local observation graph of the first unloader. The feature aggregation module is used to aggregate the initial node features of all nodes in the local observation graph through the graph neural network deployed in the target reinforcement learning model of the first ship unloader, so as to obtain the node aggregated features of the first ship unloader. An action output module is used to determine the current control action for the first ship unloader based on the node aggregation features of the first ship unloader and through the policy network deployed in the target reinforcement learning model of the first ship unloader. The control execution module is used to control the first ship unloader to perform the control actions.

[0139] Optionally, in some embodiments of this application, the current state data of the first ship unloader includes the first predicted trajectory of the first ship unloader for multiple future time steps, and the current neighbor information of the neighbor ship unloader includes the second predicted trajectory of the neighbor ship unloader for multiple future time steps. The reinforcement learning-based ship unloader control device is also used for: If a trajectory point less than a preset safe distance threshold is detected between the first predicted trajectory and the second predicted trajectory, the first ship unloader is triggered to replan its control actions by using the target reinforcement learning model deployed on the first ship unloader and constrained by the predicted trajectory of the neighboring ship unloader.

[0140] Optionally, in some embodiments of this application, the reinforcement learning-based ship unloader control device is further configured to: monitor the material distribution of the task performed by the first ship unloader; if the rate of change of the material distribution exceeds a preset rate of change threshold, trigger the first ship unloader to replan the control action of the first ship unloader by means of the target reinforcement learning model deployed on the first ship unloader with the rate of change of the material distribution as a constraint. Alternatively, the reinforcement learning-based ship unloader control device is further configured to: monitor the ship attitude of the task performed by the first ship unloader; if the offset of the ship attitude exceeds a preset offset threshold, trigger the first ship unloader to replan its control actions by means of the target reinforcement learning model deployed on the first ship unloader, with the offset of the ship attitude as a constraint.

[0141] In one exemplary embodiment, a ship unloading machine is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0142] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0143] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0144] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0145] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0146] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0147] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0148] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A reinforcement learning model training method, characterized in that, The reinforcement learning model training method includes: Construct an initial reinforcement learning model; the initial reinforcement learning model includes a graph neural network for characterizing the current state of the ship unloader through feature aggregation, a policy network for outputting the current control action for the current state of the ship unloader, and a value network for estimating the long-term expected reward obtained by the ship unloader in the current state by following the current control action. For the first training phase, a first sample set is obtained, and the parameters of the policy network and the value network are trained based on the first sample set in the first training phase so that the initial reinforcement learning model can perform single-machine basic policy learning. For the second training phase, a second sample set is obtained. In the second training phase, the policy network parameters obtained from the first training phase are inherited. The parameters of the policy network and the value network are trained based on the second sample set so that the initial reinforcement learning model can perform dual-machine collaborative collision avoidance learning. The graph neural network does not participate in the training in the second training phase. For the third training phase, a third sample set is obtained. In the third training phase, the policy network parameters obtained from the second training phase are inherited. Based on the third sample set, the parameters of the graph neural network, the policy network, and the value network are trained so that the initial reinforcement learning model can perform multi-machine cooperative collision avoidance learning to obtain a target reinforcement learning model. The target reinforcement learning model is used for each unloader in the unloader system and determines the control actions for the unloader based on the current state of the unloader. The unloader system includes multiple unloaders.

2. The reinforcement learning model training method according to claim 1, characterized in that, The simulation training scenario in the first training phase includes: a simulation scenario in which a single virtual ship unloader performs a single task; in the simulation training scenario of the first training phase, the initial reinforcement learning model is deployed on the single virtual ship unloader. The simulation training scenario of the second training phase includes: two virtual ship unloaders performing their respective tasks and the working areas of the two virtual ship unloaders overlapping; in the simulation training scenario of the second training phase, the initial reinforcement learning model is deployed in each of the two virtual ship unloaders. The simulation training scenario of the third training phase includes: three or more virtual ship unloaders performing their respective tasks, and the working areas of at least two of the virtual ship unloaders overlapping; in the simulation training scenario of the third training phase, the initial reinforcement learning model is deployed in each of the three or more virtual ship unloaders.

3. The reinforcement learning model training method according to claim 2, characterized in that, The reward function of the first training phase includes a weighted sum of the following items: a task completion rate item representing the completion status of the virtual ship unloader's task, and an energy consumption item representing the energy consumption of the virtual ship unloader in performing the task. The reward function for the second training phase includes a weighted sum of the following items: a task completion rate item representing the completion status of the virtual ship unloader's task, an energy consumption item representing the energy consumption of the virtual ship unloader in performing the task, a collision avoidance safety item representing the collision risk between the two virtual ship unloaders, and a cooperative behavior item representing the cooperation between the two virtual ship unloaders. The reward function of the third training phase includes a weighted sum of the following items: a task completion rate item representing the completion status of the virtual ship unloader's tasks; an energy consumption item representing the energy consumption of the virtual ship unloader in performing its tasks; a collision avoidance safety item representing the collision risk between the virtual ship unloader and any other virtual ship unloader; and a cooperative behavior item representing the cooperation among the three or more virtual ship unloaders. The weight of the cooperative behavior item in the third training phase is greater than the weight of the cooperative behavior item in the second training phase.

4. The reinforcement learning model training method according to claim 1, characterized in that, The graph neural network is specifically used to generate the attention weights of the unloading machine and the attention weights of the neighboring unloading machines based on the attention mechanism. Based on the attention weights of the unloading machine and the attention weights of the neighboring unloading machines, the initial node features representing the current state of the unloading machine and the initial node features representing the current state of the neighboring unloading machines are weighted and summed to obtain the node aggregation features representing the current state of the unloading machine. The neighboring ship unloader is a ship unloader located within the communication range of the ship unloader; The initial features of the node representing the current state of the ship unloader include the following information: the current position information of the ship unloader, the current speed information of the ship unloader, the current grab information of the ship unloader, the current predicted trajectory of the ship unloader for multiple future time steps, the current task target information of the ship unloader, and the current environmental state information of the ship unloader.

5. The reinforcement learning model training method according to claim 1, characterized in that, The policy network is specifically used to output the current control action for the ship unloader based on the node aggregation features representing the current state of the ship unloader. The control actions include the following information: the trolley displacement increment command of the ship unloader along the track direction, the trolley displacement increment command of the ship unloader along the direction perpendicular to the track direction, the grab bucket lifting speed command of the ship unloader, and the grab bucket grabbing or releasing control command of the ship unloader.

6. A reinforcement learning-based control method for a ship unloader, characterized in that, A ship unloader system is used in a ship unloader system, the ship unloader system comprising multiple ship unloaders, each of which is equipped with a target reinforcement learning model, the target reinforcement learning model being trained using the reinforcement learning model training method described in any one of claims 1 to 5, and the reinforcement learning-based ship unloader control method comprising: The system collects current status data of the first unloader in the unloader system and receives current neighbor information of neighbor unloaders within the communication range of the first unloader in the unloader system; the first unloader is any unloader in the unloader system. Using the first unloader as the root node and the neighboring unloaders as neighboring nodes, the initial node characteristics of the root node are determined based on the current state data of the first unloader, and the initial node characteristics of the neighboring nodes are determined based on the neighboring machine information. The root node and the neighboring nodes are then connected to construct a local observation map of the first unloader. The initial node features of all nodes in the local observation graph are aggregated by the graph neural network deployed in the target reinforcement learning model of the first ship unloader to obtain the node aggregated features of the first ship unloader. Based on the node aggregation characteristics of the first ship unloader, the current control action for the first ship unloader is determined by the policy network deployed in the target reinforcement learning model of the first ship unloader. Control the first ship unloader to perform the control action.

7. The reinforcement learning-based ship unloader control method according to claim 6, characterized in that, The current status data of the first ship unloader includes the first predicted trajectory of the first ship unloader for multiple future time steps, and the current neighbor information of the neighbor ship unloader includes the second predicted trajectory of the neighbor ship unloader for multiple future time steps. The reinforcement learning-based ship unloader control method also includes: If a trajectory point less than a preset safe distance threshold is detected between the first predicted trajectory and the second predicted trajectory, the first ship unloader is triggered to replan its control actions by using the target reinforcement learning model deployed on the first ship unloader and constrained by the predicted trajectory of the neighboring ship unloader.

8. The reinforcement learning-based ship unloader control method according to claim 6, characterized in that, The reinforcement learning-based ship unloader control method further includes: monitoring the material distribution of the task performed by the first ship unloader; if the rate of change of the material distribution exceeds a preset rate of change threshold, the first ship unloader is triggered to replan its control actions by using the target reinforcement learning model deployed on the first ship unloader with the rate of change of the material distribution as a constraint. Alternatively, the reinforcement learning-based ship unloader control method further includes: monitoring the ship attitude of the task performed by the first ship unloader; if the offset of the ship attitude exceeds a preset offset threshold, triggering the first ship unloader to replan its control actions by means of the target reinforcement learning model deployed on the first ship unloader, with the offset of the ship attitude as a constraint.

9. A reinforcement learning model training device, characterized in that, The reinforcement learning model training device includes: The model building module is used to build an initial reinforcement learning model. The initial reinforcement learning model includes a graph neural network for representing the current state of the ship unloader through feature aggregation, a policy network for outputting the current control action for the current state of the ship unloader, and a value network for estimating the long-term expected reward obtained by the ship unloader in the current state by following the current control action. The first training module is used to acquire a first sample set for the first training phase, and train the parameters of the policy network and the value network based on the first sample set in the first training phase, so that the initial reinforcement learning model can perform single-machine basic policy learning. The second training module is used to obtain a second sample set for the second training phase, inherit the policy network parameters trained in the first training phase in the second training phase, and train the parameters of the policy network and the value network based on the second sample set so that the initial reinforcement learning model can perform dual-machine collaborative collision avoidance learning; the graph neural network does not participate in the training in the second training phase. The third training module is used to obtain a third sample set for the third training stage. In the third training stage, the policy network parameters obtained from the second training stage are inherited. The parameters of the graph neural network, the policy network, and the value network are trained based on the third sample set so that the initial reinforcement learning model can perform multi-machine cooperative collision avoidance learning to obtain a target reinforcement learning model. The target reinforcement learning model is used to deploy each unloader in the unloader system and to determine the control actions for the deployed unloader based on the current state of the deployed unloader. The unloader system includes multiple unloaders.

10. A ship unloader control device based on reinforcement learning, characterized in that, A ship unloader system is used in a ship unloader system, the ship unloader system comprising multiple ship unloaders, each of which is equipped with a target reinforcement learning model, the target reinforcement learning model being trained using the reinforcement learning model training method according to any one of claims 1 to 5, and the reinforcement learning-based ship unloader control device comprising: The data acquisition module is used to collect the current status data of the first unloader in the unloader system, and to receive the current neighbor information of neighbor unloaders located within the communication range of the first unloader in the unloader system; the first unloader is any unloader in the unloader system. The graph construction module is used to take the first unloader as the root node and the neighbor unloader as the neighbor node, determine the initial node characteristics of the root node based on the current state data of the first unloader, determine the initial node characteristics of the neighbor node based on the neighbor machine information, and connect the root node and the neighbor node to construct a local observation graph of the first unloader. The feature aggregation module is used to aggregate the initial node features of all nodes in the local observation graph through the graph neural network deployed in the target reinforcement learning model of the first ship unloader, so as to obtain the node aggregated features of the first ship unloader. An action output module is used to determine the current control action for the first ship unloader based on the node aggregation features of the first ship unloader and through the policy network deployed in the target reinforcement learning model of the first ship unloader. The control execution module is used to control the first ship unloader to perform the control actions.

Images