Container bay recommendation method, system, device and medium based on reinforcement learning

Abstract

The application relates to the technical field of container scheduling, in particular to a container bay recommendation method, system, device and medium based on reinforcement learning, which comprises the following steps: obtaining yard state data and to-be-entered container attribute data and encoding the same into a joint state vector; inputting the joint state vector into a trained bay recommendation model, calculating the action value of all available bays, selecting the bay with the highest action value as the recommended bay, and outputting the position information of the recommended bay; performing an actual bay selection operation and recording the position information of the actually selected bay; generating a feedback signal according to the actually selected bay; storing the joint state vector, the actually selected bay, the feedback signal and the yard state data after the bay selection into an experience replay pool, and updating the parameters of the bay recommendation model through sampling training data. The application can improve the yard space utilization rate and operation efficiency, reduce the bay conflict and turnover moving cost, realize multi-target collaborative optimization, and has strong environmental adaptability.

Description

Technical Field

[0001] This application relates to the field of container scheduling technology, specifically to a container location recommendation method, system, device, and medium based on reinforcement learning. Background Technology

[0002] As a core node in the global logistics chain, container terminals handle a large volume of container loading, unloading, storage, and handling. With the surge in international trade volume, the scale and complexity of terminal operations continue to rise, and traditional manual scheduling methods are no longer sufficient to meet the demands of high throughput, especially in terms of optimizing yard space, reducing repetitive container handling, and improving operational efficiency, where significant bottlenecks exist.

[0003] In existing technologies, automated container yards need to determine the optimal location (zone, bay, row, layer) for newly arrived containers through container placement recommendations to optimize yard space, reduce redundant container handling, and improve operational efficiency. The core objective of container placement recommendations is to optimize container placement to maximize yard utilization and reduce future container handling costs and time. Existing automated container placement recommendation methods primarily employ heuristic or rule-based algorithms, using historical data to match stacking rules under fixed scenarios to recommend container locations. While these methods are effective in standardized operations, their accuracy and adaptability significantly decrease when faced with dynamic changes in yard conditions, diverse container attributes, and high-dimensional state spaces.

[0004] The limitations of existing container location recommendation methods are mainly reflected in the following aspects: they rely on manually preset rules, making it impossible to adapt to changes in site layout in real time; they are inefficient when handling complex constraints (such as mixed storage of containers from multiple ship types and of different sizes); and they are difficult to optimize strategies through self-learning, leading to wasted yard space and increased handling costs. In addition, existing systems lack a feedback mechanism for real-time interaction with dispatchers, making it impossible to dynamically adjust recommendation strategies according to actual business needs. Summary of the Invention

[0005] To address the technical problems of existing container location recommendation methods, such as lack of dynamic adaptability, low efficiency when handling complex constraints, difficulty in self-learning, and lack of feedback mechanisms, this application provides a container location recommendation method, system, equipment, and medium based on reinforcement learning. This method can improve yard space utilization and operational efficiency, reduce container location conflicts and container relocation costs, achieve multi-objective collaborative optimization, and has strong environmental adaptability.

[0006] Firstly, this application provides a container location recommendation method based on reinforcement learning, comprising the following steps:

[0007] S1. Obtain yard status data and container attribute data to be delivered;

[0008] Among them, the yard status data includes the location and availability of each area, shell, and barn in the yard, as well as the location and attribute data of the stored containers;

[0009] S2. Encode the yard status data and the attribute data of containers waiting to enter the yard into a joint state vector;

[0010] S3. Input the joint state vector into the trained bin recommendation model and calculate the action value of all available bins. The bin recommendation model is a reinforcement learning algorithm model.

[0011] Select the box with the highest action value as the recommended box, and output the location information of the recommended box, including the combination of the area, shell, and row where the recommended box is located;

[0012] S4. Perform the actual container location selection operation and record the location information of the actual selected container location;

[0013] S5. Generate a feedback signal based on the actual selected container location, and associate the feedback signal with the quality assessment indicators of the container location selection;

[0014] S6. Store the joint state vector, actual selected container location, feedback signal, and yard state data after container location selection into the experience playback pool, and update the parameters of the container location recommendation model by sampling training data.

[0015] It should be further noted that in step S1, the yard status data also includes the type of automated equipment in each yard area, the height limit of each bay, and the operation conflict markers of adjacent container positions in each row.

[0016] It should be further noted that in step S1, the attribute data of the stored containers includes the cargo category, size, loading and unloading priority, and storage time of the containers.

[0017] The attribute data of containers waiting to enter the terminal includes the cargo category, size, weight, destination port and transportation priority of the containers waiting to enter the terminal.

[0018] It should be further noted that the encoding rules for the joint state vector in step S2 include:

[0019] The yard status data is encoded into a two-dimensional yard status matrix. The yard status matrix includes yard status sub-matrices with the same number of yard areas. Each yard status sub-matrice corresponds to a yard area. The rows and columns of the yard status sub-matrices correspond to the bays and rows of the yard area, respectively. Each element in the yard status sub-matrices represents the available status, the type of container already stored, and the layer height limit of the corresponding location.

[0020] The attribute data of containers waiting to enter the yard are converted into feature vectors through one-hot encoding or embedding encoding, and then concatenated with the yard state matrix to form a joint state vector.

[0021] It should be further noted that in step S3, the bin location recommendation model is a deep Q-network model. The input of the bin location recommendation model is the joint state vector s, and the output is the Q value of each available bin location.

[0022] The bin placement recommendation model includes an online network and a target network, with the target network parameters being synchronized with the online network parameters periodically.

[0023] It should be further noted that in step S3, the recommended bins are selected using an ε-greedy strategy, and the exploration probability ε of the ε-greedy strategy decays exponentially with the number of training rounds.

[0024] It should be further noted that in the ε-greedy strategy, the initial value of the exploration probability ε is 0.8-0.95, and the decay rate is 90%-98% of the original value every 50 rounds.

[0025] It should be further noted that in step S5, the formula for calculating the feedback signal R is:

[0026]

[0027] In the formula, S match The matching coefficient is 1 when the actual selected container location matches the recommended container location, and 0 otherwise.

[0028] N move To estimate the number of moves, calculations are based on the number of operational path conflicts between the target container location and associated containers;

[0029] R waste The space wastage rate is expressed as:

[0030]

[0031] N empty Count the number of currently available positions for the target.

[0032] N max The target is to determine the theoretical maximum number of stacking levels;

[0033] N current The target is to determine the number of containers already stored.

[0034] α, β, and γ are weighting coefficients that satisfy α + β + γ = 1.

[0035] It should be further explained that N move The calculation rules include: when there are containers in the row where the target container slot is located that conflict with the vessel name or bill of lading number of the container waiting to enter the yard, for each conflicting container, N move Increase by 1;

[0036] R wasteSatisfy the following condition: If the number of empty positions in the target row is less than 10% of the theoretical maximum heap size, then R is forcibly set. waste =1.

[0037] Secondly, the present invention provides a container space recommendation system based on reinforcement learning, for implementing the above-mentioned container space recommendation method, comprising:

[0038] The data acquisition module is used to acquire yard status data and attribute data of containers waiting to enter the yard;

[0039] The data encoding module is used to encode yard status data and container attribute data to be delivered into a joint state vector;

[0040] The bin recommendation module is used to input the joint state vector into the trained bin recommendation model, calculate the action value of all available bins, select the bin with the highest action value as the recommended bin, and output the location information of the recommended bin.

[0041] The container location recording module is used to record the location information of the actually selected container location when the actual selected container location is inconsistent with the recommended container location.

[0042] The feedback signal generation module is used to generate a feedback signal based on the actual selected container position.

[0043] The experience replay and model update module is used to store the joint state vector, actual selected container locations, feedback signals, and yard state data after container location selection into the experience replay pool, and update the parameters of the container location recommendation model by sampling training data.

[0044] Thirdly, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described reinforcement learning-based container placement recommendation method.

[0045] Fourthly, the present invention provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described reinforcement learning-based container location recommendation method.

[0046] As can be seen from the above technical solutions, this application has the following advantages:

[0047] 1. This invention acquires yard status data and container attribute data to be delivered and constructs a joint state vector, enabling the container location recommendation model to fully perceive the dynamic changes and operational constraints of the yard, providing accurate input for subsequent container location selection; the global value evaluation mechanism based on the pre-trained container location recommendation model can quickly screen out the container location scheme with the best overall benefits, overcoming the problems of low efficiency and limited optimization space in traditional manual planning, realizing accurate representation of the global state of the yard, and avoiding suboptimal decision-making problems caused by the lack of local information.

[0048] 2. This invention uses a reinforcement learning model to evaluate the action value of all available box locations, and can quickly generate the optimal recommended solution that comprehensively considers factors such as space utilization and operational conflicts, overcoming the limitations of traditional methods that can only handle limited constraints and cannot find global optimization.

[0049] 3. After performing the actual container location selection operation, this invention records the location information of the selected container location, generates a feedback signal based on the selected container location, associates the feedback signal with the quality evaluation index of container location selection, and feeds it back to the model training process, forming a human-machine collaborative optimization mechanism, which effectively makes up for the shortcomings of pure algorithm recommendation and the disconnect between on-site operation experience.

[0050] 4. The mechanism of binding the experience replay pool and feedback signal to quality evaluation indicators in this invention enables the model to iteratively optimize around the core objectives of reducing container turnover and reducing space waste during the training process. Compared with the problem of lacking continuous learning ability in the prior art, this method achieves progressive enhancement of the recommendation strategy through a closed-loop data-driven parameter update strategy, which significantly improves the long-term stability and adaptability of yard resource scheduling. Attached Figure Description

[0051] To more clearly illustrate the technical solution of this application, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 This is a flowchart of a container location recommendation method based on reinforcement learning in one embodiment of the present invention.

[0053] Figure 2 This is a schematic diagram of the layout structure of a container yard in one embodiment of the present invention.

[0054] Figure 3 This is a schematic diagram of the calculation logic and feedback mechanism of the container placement recommendation model in one embodiment of the present invention.

[0055] Figure 4This is a schematic block diagram of a container space recommendation system based on reinforcement learning in one embodiment of the present invention.

[0056] Figure 5 This is a schematic diagram of the hardware structure of an electronic device in one embodiment of the present invention. Detailed Implementation

[0057] To make the purpose, features, and advantages of this application more apparent and understandable, specific embodiments and accompanying drawings will be used to clearly and completely describe the technical solution protected by this application. Obviously, the embodiments described below are only some embodiments of this application, and not all embodiments. Based on the embodiments in this patent, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this patent.

[0058] This application relates to a reinforcement learning-based container space recommendation method primarily targeting the container scheduling technology field. By acquiring yard state data and the attribute data of containers awaiting entry and constructing a joint state vector, the space recommendation model can comprehensively perceive dynamic changes in the yard and operational constraints, providing accurate input for subsequent space selection. A global value evaluation mechanism based on the pre-trained space recommendation model can quickly select the space scheme with the best overall benefits, overcoming the low efficiency and limited optimization space of traditional manual planning. This achieves accurate representation of the yard's global state and avoids suboptimal decision-making problems caused by missing local information. Furthermore, by using a reinforcement learning model to evaluate the action value of all available space slots, it can quickly generate the optimal recommendation scheme that comprehensively considers factors such as space utilization and operational conflicts, overcoming... This method overcomes the limitations of traditional methods, which can only handle finite constraints and cannot perform global optimization. After performing the actual container location selection operation, it records the location information of the selected container location, generates a feedback signal based on the selected location, associates the feedback signal with the quality evaluation index of container location selection, and feeds it back to the model training process, forming a human-machine collaborative optimization mechanism. This effectively compensates for the shortcomings of pure algorithm recommendation being disconnected from on-site operational experience. The mechanism of binding the experience replay pool with the quality evaluation index of the feedback signal enables the model to iteratively optimize around the core objectives of reducing container turnover and reducing space waste during the training process. Compared with the problem of existing technologies lacking continuous learning ability, this method achieves progressive enhancement of the recommendation strategy through a closed-loop data-driven parameter update strategy, significantly improving the long-term stability and adaptability of yard resource scheduling.

[0059] The container space recommendation method based on reinforcement learning involved in this application mainly addresses the technical problems of existing container space recommendation methods, such as lack of dynamic adaptability, low efficiency when handling complex constraints, difficulty in autonomous learning, and lack of feedback mechanism.

[0060] The following describes in detail the reinforcement learning-based container location recommendation method of this application. Specific details, such as particular system architectures and techniques, are presented for illustrative purposes and not for limitation, to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application can also be implemented in other embodiments without these specific details.

[0061] In the reinforcement learning-based container placement recommendation method disclosed in this application, the term "comprising" indicates the presence of the described feature, whole, step, operation, element, and / or component, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or sets thereof. The terms "comprising," "including," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0062] To facilitate a clear description of the technical solutions of this application, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that the terms "first" and "second" do not necessarily imply that they are different.

[0063] The terms "one embodiment" or "some embodiments" used in this application mean that one or more embodiments of this application include the specific features, structures, or characteristics described in that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this application do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0064] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0065] The container location recommendation method based on reinforcement learning provided in this embodiment of the invention is executed by a computer device, and correspondingly, the container location recommendation system based on reinforcement learning runs on the computer device.

[0066] Figure 1 This is a flowchart of a container location recommendation method based on reinforcement learning, according to an embodiment of the present invention. Figure 1The implementing entity can be a container placement recommendation system based on reinforcement learning. Depending on different requirements, the order of the steps in this flowchart can be changed, and some can be omitted.

[0067] like Figure 1 As shown, the reinforcement learning-based container location recommendation method includes:

[0068] Step S1: Obtain yard status data and container attribute data to be delivered;

[0069] The yard status data includes the location and availability of each area, bay, and row in the yard, as well as the location and attribute data of the stored containers.

[0070] Comprehensive data collection of the yard's physical layout, equipment distribution, and container business attributes can fully reflect the current occupancy and layout of the yard. Combined with the attributes of containers waiting to enter the yard, this provides comprehensive basic data support for subsequent accurate container location recommendations, thereby making container location recommendation decisions that better meet actual needs and improving the accuracy and rationality of the recommendations.

[0071] Figure 2 This is a schematic diagram of the layout structure of a container yard in one embodiment of the present invention. In the container yard, the stacking location of containers consists of four parts: zone, bay, row, and stack height. Precise yard location numbers can be generated based on the specific location of the containers in the zone, bay, and row, such as:

[0072] The container's yard number is AA001025, indicating that the container is located in the AA zone of the yard, in bay number 001, in row number 02, and at a height of 5 layers (i.e., the height of the container stack).

[0073] In some specific embodiments, the yard status data also includes the type of automated equipment in each yard area, the height limit of each bay, and the operation conflict markers of adjacent container positions in each row.

[0074] Among them, the equipment type parameter enables the model to adapt to the operating radius and operating characteristics of different machines; the layer height limit data of the Bay dimension enforces the stacking safety specifications to prevent stacking of excessively high layers; the conflict marker of the row dimension identifies the risk of interference between adjacent container positions and ensures the safety of multi-equipment collaborative operation. The integration of these constraints makes the recommendation results conform to the safety procedures of on-site operation.

[0075] In some specific embodiments, the attribute data of the stored containers includes the container's cargo category, size, loading and unloading priority, and storage time;

[0076] The attribute data of containers waiting to enter the terminal includes the cargo category, size, weight, destination port and transportation priority of the containers waiting to enter the terminal.

[0077] Among them, cargo category characteristics drive the isolated storage of special containers such as dangerous goods and refrigerated containers to meet cargo storage requirements; loading and unloading priority parameters guide the model to generate container layouts that conform to the operation sequence, accelerating the turnover efficiency of key containers; and stacking time data assists in predicting container occupancy cycles and optimizing medium- and long-term allocation strategies for storage space. This attribute-aware mechanism can improve the matching degree between recommended solutions and business needs.

[0078] Step S2: Encode the yard status data and the attribute data of the containers to be delivered into a joint state vector.

[0079] Transforming complex and diverse data into a joint state vector format that is easy for models to process, and integrating various information features, helps models to learn the relationships between data in depth, improves data processing efficiency, and enhances the model's ability to understand input information.

[0080] In some specific embodiments, the encoding rules for the joint state vector include:

[0081] The yard status data is encoded into a two-dimensional yard status matrix. The yard status matrix includes yard status sub-matrices with the same number of yard areas. Each yard status sub-matrice corresponds to a yard area. The rows and columns of the yard status sub-matrices correspond to the bays and rows of the yard area, respectively. Each element in the yard status sub-matrices represents the available status, the type of container already stored, and the layer height limit of the corresponding location.

[0082] The attribute data of containers waiting to enter the yard are converted into feature vectors through one-hot encoding or embedding encoding, and then concatenated with the yard state matrix to form a joint state vector.

[0083] Two-dimensional yard state matrix encoding can preserve the spatial topology of the yard. The sub-matrix division of the yard area dimension maintains the independence of the yard's physical layout, and the matrix structure of the bay coordinates intuitively reflects the spatial distribution of container positions. The matrix elements of the vacant state and the layer height limit clearly mark the boundaries of the operable area. One-hot encoding processes discrete attributes to ensure feature parsing, and embedded encoding compresses high-dimensional business tags to improve computational efficiency. By concatenating the feature vector with the yard state matrix into a joint state vector, a structured expression of the complex state of the yard can be realized.

[0084] Step S3: Input the joint state vector into the trained bin recommendation model and calculate the action value of all available bins. The bin recommendation model is a reinforcement learning algorithm model.

[0085] Select the box with the highest action value as the recommended box and output the location information of the recommended box, including the combination of the area, shell, and row where the recommended box is located.

[0086] By using reinforcement learning algorithms to calculate the value of actions, the optimal container location can be selected. Compared with manual or simple rule allocation, this can optimize container allocation, improve yard space utilization, reduce container handling costs, reduce interference between different operations, and improve the overall smoothness of operations.

[0087] In some specific embodiments, the layout and operation of a container terminal are modeled as a Markov decision process (MDP), which includes the following elements:

[0088] State (S): The current state of the yard, including the distribution of stacked containers, container attributes, location of each area, and currently available empty container spaces.

[0089] Action (A): For each newly arrived container, the algorithm can choose the set of available empty container slots in all current yard areas. Possible actions include stacking in different pallets or at different levels.

[0090] Reward function (R): The reward function is calculated based on the impact of each container placement on the overall system efficiency, such as reducing the number of future container repositionings or improving space utilization. A typical reward might be a negative repositioning cost (reducing the number of future operations) and a negative yard space waste.

[0091] State transition (T): After a container is placed, the system transitions from one state to another. The state transition is affected by the current position of the container and the order in which subsequent containers enter and exit.

[0092] In some specific embodiments, the bin location recommendation model is a deep Q-network (DQN) model. The input of the bin location recommendation model is the joint state vector s, and the output is the Q value of each available bin location.

[0093] The bin placement recommendation model includes an online network and a target network, with the target network parameters being synchronized with the online network parameters periodically.

[0094] The deep Q-network with a dual-network architecture can enhance the stability of training and effectively handle the problem of approximating the value function in a high-dimensional state space. The online network performs real-time value calculation, while the target network provides a delayed value estimation benchmark, which can alleviate the Q-value drift problem caused by the bootstrapping method. The periodic parameter synchronization mechanism balances the network update frequency and ensures the policy coherence during the model convergence process.

[0095] In some specific embodiments, the architecture of the bin placement recommendation model includes:

[0096] Input layer: The current state of the storage yard is input, which includes information such as stacking layout and container attributes. This information can be encoded into a vector and input into the neural network.

[0097] Hidden layers: Multi-layer fully connected neural networks or convolutional neural networks (CNNs) are used to extract features of the stacking yard state. Stacking yard layout information can be processed by CNNs to process two-dimensional stacking yard matrix data, or by using fully connected networks to process the feature vectors of stacking information.

[0098] Output layer: Outputs the value of each action. In the bin placement problem, the output dimension is consistent with the size of the action space, that is, the network estimates the value for each possible stacking and hierarchical position.

[0099] The parameters of the neural network are continuously updated through interaction with the environment during the reinforcement learning process;

[0100] In some specific implementations, the training steps of the bin placement recommendation model include:

[0101] S301. Initialize the experience replay pool and build a dual-network structure. The online network calculates the action value in real time, and the target network provides a stable evaluation benchmark.

[0102] S302. Train the container location recommendation model using simulated container entry and exit events. Perform container location selection operations through an exploration strategy. Use the joint state vector (current state), the actual selected container location (action), the feedback signal (immediate reward), and the yard state data after container location selection (next state) as a state-action-reward-new state quadruple. Store in the experience replay pool;

[0103] S303. Randomly sample batches of historical interaction data, and use the online network to calculate the current Q value and the target network to calculate the maximum Q value of the next state respectively;

[0104] S304. Calculate the loss function based on temporal difference error, and update the online network parameters through backpropagation;

[0105] S305. Synchronize online network parameters to the target network at fixed intervals to maintain the stability of policy evaluation;

[0106] S306. Dynamically adjust the exploration probability and monitor training metrics. Complete model iteration when the reward converges and the box-flipping cost stabilizes.

[0107] S307. During the deployment phase, network parameters are locked to retain basic exploration capabilities to cope with unexpected operational scenarios.

[0108] In some specific embodiments, an ε-greedy strategy is used to select recommended bin locations, and the exploration probability ε of the ε-greedy strategy decays exponentially with the number of training rounds.

[0109] The principle of the ε-greedy strategy is: in each state In this scenario, the box placement recommendation model randomly selects an action with a certain probability ε to explore new potentially advantageous actions; otherwise, it selects the current action. Utilize the action with the highest value, that is:

[0110]

[0111] By randomly selecting actions with probability ε, instead of relying on existing Q-value estimates, the box position is randomly selected, which helps the algorithm avoid getting trapped in local optima.

[0112] As training progresses, the value of ϵ gradually decreases (from high to low), causing the algorithm to gradually transition from more exploration in the early stages to more utilization in the later stages, eventually converging near the optimal policy.

[0113] Using an ε-greedy strategy to select recommended bins can achieve a dynamic balance between exploration and utilization, enabling the model to adaptively adjust its decision-making mode at different training stages. The high exploration rate in the initial stage supports global space search and can avoid getting trapped in local optima. Gradually reducing the exploration intensity as training progresses can enhance the stability of mature strategies.

[0114] In some specific embodiments, in the ε-greedy strategy, the initial value of the exploration probability ε is 0.8-0.95, and the decay rate is reduced to 90%-98% of the original value every 50 rounds.

[0115] Setting the decay rate parameter can control the gradient of exploration intensity changes. In this embodiment, the limited wide initial exploration rate ensures full coverage of each area of ​​the yard, and the slow decay rate maintains the continuity of exploration behavior, preventing premature convergence to a suboptimal strategy, which can adapt to the complex and ever-changing operating environment of container yards.

[0116] Step S4: Perform the actual container location selection operation and record the location information of the actual selected container location.

[0117] Clearly defining the actual container location ensures the orderly operation of the yard. The recorded location information can be used to review operations, verify the accuracy of recommendations, and provide data support for improving operational processes and optimizing models.

[0118] Step S5: Generate a feedback signal based on the actual selected container location, and associate the feedback signal with the quality assessment index of the container location selection.

[0119] Feedback signals generated based on actual container location selection and associated quality assessment indicators help to quantitatively evaluate operational effectiveness. They can intuitively reflect whether the container location selection is reasonable, providing direction for adjusting operational strategies and improving models. By analyzing the feedback signals, problems in the operation can be identified, such as insufficient space utilization and unreasonable handling paths, thus enabling targeted optimization.

[0120] In some specific embodiments, the formula for calculating the feedback signal R is:

[0121]

[0122] In the formula, S match The matching coefficient is 1 when the actual selected container location matches the recommended container location, and 0 otherwise.

[0123] N move To estimate the number of moves, calculations are based on the number of operational path conflicts between the target container location and associated containers;

[0124] R waste The space wastage rate is expressed as:

[0125]

[0126] N empty Count the number of currently available positions for the target.

[0127] N max The target is to determine the theoretical maximum number of stacking levels;

[0128] N current The target is to determine the number of containers already stored.

[0129] α, β, and γ are weighting coefficients that satisfy α + β + γ = 1.

[0130] The above formula can be used to calculate feedback signals to achieve multi-dimensional business goal synergistic optimization. The matching coefficient enhances the model's learning of human operational experience, the move number penalty term suppresses inefficient operation paths, and the space waste rate index promotes compact storage. The adjustable weight coefficient supports flexible adaptation to differentiated operation strategies and deeply links technical indicators with business value.

[0131] In some specific embodiments, N move The calculation rules include: when there are containers in the row where the target container slot is located that conflict with the vessel name or bill of lading number of the container waiting to enter the yard, for each conflicting container, N move Increase by 1;

[0132] R waste Satisfy the following condition: If the number of empty positions in the target row is less than 10% of the theoretical maximum heap size, then R is forcibly set. waste =1.

[0133] Step S6: Store the joint state vector, actual selected container location, feedback signal, and yard state data after container location selection into the experience playback pool, and update the parameters of the container location recommendation model by sampling training data.

[0134] By storing various types of data in an experience replay pool, the model can learn from more real-world cases. By sampling training data to update parameters, the model can adapt to different yard conditions and operational needs, continuously optimize recommendation strategies, improve the accuracy and reliability of container location recommendations, and continuously enhance the overall operational efficiency and management level of the yard.

[0135] In some specific embodiments, the update rule for the container placement recommendation model is as follows:

[0136] After each action is executed, the joint state vector (current state), the actual selected container location (action), the feedback signal (immediate reward), and the yard state data after the container location selection (next state) are stored in the experience replay pool. Random sampling is used to train the container location recommendation model, and the Q-value update formula is:

[0137]

[0138] in:

[0139] The current state Next action of value;

[0140] It is to perform an action The instant reward obtained afterward;

[0141] It is a discount factor that measures the importance attached to future rewards;

[0142] The learning rate determines the step size for each update.

[0143] Update the formula for adjustment The value that makes the current state Maximum value and future state As the values ​​gradually become consistent, the bin placement recommendation model can gradually approach the optimal value through updates to the neural network parameters. Value function.

[0144] In one specific embodiment, the container location recommendation method based on reinforcement learning includes:

[0145] Step S1: Obtain yard status data and container attribute data to be delivered;

[0146] The yard status data includes the location and availability of each area, bay, and row in the yard, the location and attribute data of the stored containers, the type of automated equipment in each area, the height limit of each bay, and the operation conflict markers of adjacent container positions in each row.

[0147] The attribute data of the stored containers includes the cargo category, size, loading and unloading priority, and storage time of the containers;

[0148] The attribute data of containers waiting to enter the terminal includes the cargo category, size, weight, destination port and transportation priority of the containers waiting to enter the terminal;

[0149] Step S2: Encode the yard status data and the attribute data of the containers to be delivered into a joint state vector;

[0150] The encoding rules for the joint state vector include:

[0151] The yard status data is encoded into a two-dimensional yard status matrix. The yard status matrix includes yard status sub-matrices with the same number of yard areas. Each yard status sub-matrice corresponds to a yard area. The rows and columns of the yard status sub-matrices correspond to the bays and rows of the yard area, respectively. Each element in the yard status sub-matrices represents the available status, the type of container already stored, and the layer height limit of the corresponding location.

[0152] The attribute data of containers waiting to enter the yard are converted into feature vectors through one-hot encoding or embedded encoding, and then concatenated with the yard state matrix to form a joint state vector.

[0153] Step S3: Input the joint state vector into the trained bin recommendation model and calculate the action value of all available bins.

[0154] Figure 3 This is a schematic diagram of the calculation logic and feedback mechanism of the container placement recommendation model in this embodiment, as shown below. Figure 3 As shown, the bin location recommendation model is a deep Q-network model. The input of the bin location recommendation model is the joint state vector s, and the output is the Q value of each available bin location.

[0155] The bin placement recommendation model includes an online network and a target network, with the target network parameters being periodically synchronized with the online network parameters;

[0156] The architecture of the bin placement recommendation model includes:

[0157] Input layer: The current state of the storage yard is input, which includes information such as stacking layout and container attributes. This information can be encoded into a vector and input into the neural network.

[0158] Hidden layers: Multi-layer fully connected neural networks or convolutional neural networks (CNNs) are used to extract features of the stacking yard state. Stacking yard layout information can be processed by CNNs to process two-dimensional stacking yard matrix data, or by using fully connected networks to process the feature vectors of stacking information.

[0159] Output layer: Outputs the value of each action. In the bin placement problem, the output dimension is consistent with the size of the action space, that is, the network estimates the value for each possible stacking and hierarchical position.

[0160] The parameters of the neural network are continuously updated through interaction with the environment during the reinforcement learning process;

[0161] The training steps for the bin placement recommendation model include:

[0162] S301. Initialize an experience replay pool with a capacity of 1e6 and construct a deep Q-network with a dual network structure: both the online network and the target network use three fully connected layers (512-256-128 neurons), the activation function is ReLU, the output layer is linearly activated, and the network parameters are initialized using He normality.

[0163] S302. Simulate 200 types of cargo (sizes including 20 / 40 / 45 feet, weight following a normal distribution of [5,30] tons) using a container generator. Perform container location selection operation using an ε-greedy strategy. The joint state vector (current state), the actual selected container location (action), the feedback signal (immediate reward), and the yard state data after container location selection (next state) are used as the state-action-reward-new state quadruple. Store in the experience replay pool;

[0164] The formula for calculating the feedback signal R is as follows:

[0165]

[0166] In the formula, S match The matching coefficient is 1 when the actual selected container location matches the recommended container location, and 0 otherwise.

[0167] N move To estimate the number of moves, calculations are based on the number of operational path conflicts between the target container location and associated containers;

[0168] R waste The space wastage rate is expressed as:

[0169]

[0170] N empty Count the number of currently available positions for the target.

[0171] N max The target is to determine the theoretical maximum number of stacking levels;

[0172] N current The target is to determine the number of containers already stored.

[0173] α, β, and γ are weight coefficients, α=0.6, β=0.3, γ=0.1, satisfying α+β+γ=1;

[0174] N moveThe calculation rules include: when there are containers in the row where the target container slot is located that conflict with the vessel name or bill of lading number of the container waiting to enter the yard, for each conflicting container, N move Increase by 1;

[0175] R waste Satisfy the following condition: If the number of empty positions in the target row is less than 10% of the theoretical maximum heap size, then R is forcibly set. waste =1;

[0176] S303. In each training step, 256 sets of data are uniformly sampled from the experience replay pool. The online network is used to calculate the current Q value and the target network is used to calculate the maximum Q value of the next state.

[0177] S304. Calculate the loss function based on the time-series difference error, where the loss function is the mean squared error loss function;

[0178] Use the Adam optimizer to backpropagate and update the online network parameters;

[0179] S305. After every 100 parameter updates, the online network parameters are synchronized to the target network to maintain the stability of the strategy evaluation;

[0180] S306. Dynamically adjust the exploration probability and monitor training metrics. Monitor the moving average reward (threshold > 0.75), standard deviation of box flipping times (threshold < 1.2), and variance of Q-value estimation (threshold < 0.3) in real time with a window size of 50 rounds. Trigger early stop when the three metrics meet the criteria for 10 consecutive rounds, save the optimal parameter checkpoint, and complete the model iteration.

[0181] S307. During the deployment phase, the target network parameters are frozen, the exploration rate is set to 0.05, 5% random exploration is reserved to deal with abnormal operating conditions such as equipment failure, and the smoothness of Q-value estimation is maintained by exponential weighted averaging.

[0182] The ε-greedy strategy is used to select the box with the highest action value as the recommended box, and the location information of the recommended box is output, including the combination of the area, shell, and row where the recommended box is located;

[0183] The exploration probability ε of the ε-greedy strategy decays exponentially with the number of training rounds. The initial value of the exploration probability ε is 0.8-0.95, and the decay rate is 90%-98% of the original value every 50 rounds.

[0184] Step S4: Perform the actual container location selection operation and record the location information of the actual selected container location;

[0185] Step S5: Generate a feedback signal based on the actual selected container location, and associate the feedback signal with the quality assessment index of the container location selection;

[0186] The calculation method for the feedback signal is the same as in step S302;

[0187] Step S6: Store the joint state vector, actual selected container location, feedback signal, and yard state data after container location selection into the experience playback pool. Use random sampling to train the container location recommendation model and update the parameters of the container location recommendation model. The Q-value update formula is:

[0188]

[0189] in:

[0190] The current state Next action of value;

[0191] It is to perform an action The instant reward obtained afterward;

[0192] It is a discount factor that measures the importance attached to future rewards;

[0193] The learning rate determines the step size for each update.

[0194] The following are embodiments of the container space recommendation system based on reinforcement learning provided in this disclosure. This container space recommendation system and the container space recommendation methods based on reinforcement learning in the above embodiments belong to the same inventive concept. For details not described in detail in the embodiments of the container space recommendation system based on reinforcement learning, please refer to the embodiments of the container space recommendation methods based on reinforcement learning described above.

[0195] Mobile terminals implementing various embodiments of the present invention will now be described with reference to the accompanying drawings. In the following description, suffixes such as "module," "part," or "unit" used to denote elements are used only for the convenience of illustrating the embodiments of the present invention and have no specific meaning in themselves. Therefore, "module" and "part" can be used interchangeably.

[0196] like Figure 4 As shown, the container space recommendation system based on reinforcement learning includes:

[0197] The data acquisition module is used to acquire yard status data and attribute data of containers waiting to enter the yard;

[0198] The data encoding module is used to encode yard status data and container attribute data to be delivered into a joint state vector;

[0199] The bin recommendation module is used to input the joint state vector into the trained bin recommendation model, calculate the action value of all available bins, select the bin with the highest action value as the recommended bin, and output the location information of the recommended bin.

[0200] The container location recording module is used to record the location information of the actually selected container location when the actual selected container location is inconsistent with the recommended container location.

[0201] The feedback signal generation module is used to generate a feedback signal based on the actual selected container position.

[0202] The experience replay and model update module is used to store the joint state vector, actual selected container location, feedback signal, and yard state data after container location selection into the experience replay pool, and update the parameters of the container location recommendation model by sampling training data.

[0203] The blackness control system in this embodiment is used to implement a container location recommendation method based on reinforcement learning. The steps include:

[0204] S1. Obtain yard status data and container attribute data to be delivered;

[0205] Among them, the yard status data includes the location and availability of each area, shell, and barn in the yard, as well as the location and attribute data of the stored containers;

[0206] S2. Encode the yard status data and the attribute data of containers waiting to enter the yard into a joint state vector;

[0207] S3. Input the joint state vector into the trained bin recommendation model and calculate the action value of all available bins. The bin recommendation model is a reinforcement learning algorithm model.

[0208] Select the box with the highest action value as the recommended box, and output the location information of the recommended box, including the combination of the area, shell, and row where the recommended box is located;

[0209] S4. Perform the actual container location selection operation and record the location information of the actual selected container location;

[0210] S5. Generate a feedback signal based on the actual selected container location, and associate the feedback signal with the quality assessment indicators of the container location selection;

[0211] S6. Store the joint state vector, actual selected container location, feedback signal, and yard state data after container location selection into the experience playback pool, and update the parameters of the container location recommendation model by sampling training data.

[0212] This application also provides an electronic device for implementing various embodiments of the present invention, the electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor.

[0213] Those skilled in the art will understand that the electronic device structure involved in the embodiments of the present invention does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0214] Figure 5 A schematic diagram of the hardware structure of an electronic device for implementing various embodiments of the present invention.

[0215] Electronic devices include, but are not limited to, components such as processors and memory. Those skilled in the art will understand that the electronic device structures described in the embodiments of this invention do not constitute a limitation on the electronic device. An electronic device may include more or fewer components than illustrated, or may combine certain components, or have different component arrangements.

[0216] In embodiments of the present invention, electronic devices include, but are not limited to, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments described and / or claimed herein.

[0217] In this application embodiment, the processor can be implemented using at least one of an Application-Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a processor, a controller, a microcontroller, a microprocessor, or an electronic unit designed to perform the functions described herein. In some cases, such implementations can be implemented within a controller. For software implementations, implementations such as processes or functions can be implemented with separate software modules that allow the performance of at least one function or operation. The software code can be implemented by a software application (or program) written in any suitable programming language, and the software code can be stored in memory and executed by the controller.

[0218] In addition, the electronic device includes some functional modules not shown, which will not be described in detail here.

[0219] Those skilled in the art will understand that the various aspects of the electronic device provided in this application can be implemented as a system, method, or program product. Therefore, the various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0220] This application also provides a storage medium storing a program product capable of implementing a reinforcement learning-based container placement recommendation method. In some possible implementations, various aspects of this disclosure can also be implemented as a program product comprising program code that, when run on a terminal device, causes the terminal device to perform the steps described in the foregoing "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure.

[0221] The storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example,, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0222] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A container location recommendation method based on reinforcement learning, characterized in that, include: S1. Obtain yard status data and container attribute data to be delivered. Yard status data includes the location and availability of each yard area, bay, and row in the yard, as well as the location and attribute data of the containers already stored. S2. Encode the yard status data and the attribute data of containers waiting to enter the yard into a joint state vector; The encoding rules for the joint state vector include: The yard status data is encoded into a two-dimensional yard status matrix. The yard status matrix includes yard status sub-matrices with the same number of yard areas. Each yard status sub-matrice corresponds to a yard area. The rows and columns of the yard status sub-matrices correspond to the bays and rows of the yard area, respectively. Each element in the yard status sub-matrices represents the available status, the type of container already stored, and the layer height limit of the corresponding location. The attribute data of containers waiting to enter the yard are converted into feature vectors through one-hot encoding or embedded encoding, and then concatenated with the yard state matrix to form a joint state vector. S3. Input the joint state vector into the trained bin recommendation model and calculate the action value of all available bins. The bin recommendation model is a reinforcement learning algorithm model. Select the box with the highest action value as the recommended box and output the location information of the recommended box. Among them, the ε-greedy strategy is used to select recommended bin locations, and the exploration probability ε of the ε-greedy strategy decays exponentially with the number of training rounds; S4. Perform the actual container location selection operation and record the location information of the actual selected container location; S5. Generate a feedback signal based on the actual selected container location, and associate the feedback signal with the quality assessment indicators of the container location selection; The formula for calculating the feedback signal R is: In the formula, S match The matching coefficient is 1 when the actual selected container location matches the recommended container location, and 0 otherwise. N move To estimate the number of moves, calculations are based on the number of operational path conflicts between the target container location and associated containers; R waste The space wastage rate is expressed as: N empty Count the number of currently available positions for the target. N max The target is to determine the theoretical maximum number of stacking levels; N current The target is to determine the number of containers already stored. α, β, and γ are weighting coefficients that satisfy α + β + γ = 1; S6. Store the joint state vector, actual selected container location, feedback signal, and yard state data after container location selection into the experience playback pool, and update the parameters of the container location recommendation model by sampling training data.

2. The container space recommendation method as described in claim 1, characterized in that, In step S1, the attribute data of the stored containers includes the cargo category, size, loading and unloading priority, and storage time of the containers; The attribute data of containers waiting to enter the terminal includes the cargo category, size, weight, destination port and transportation priority of the containers waiting to enter the terminal.

3. The container space recommendation method as described in claim 1, characterized in that, In step S3, the bin location recommendation model is a deep Q-network model. The input of the bin location recommendation model is the joint state vector s, and the output is the Q value of each available bin location. The bin placement recommendation model includes an online network and a target network, with the target network parameters being synchronized with the online network parameters periodically.

4. The container space recommendation method as described in claim 1, characterized in that, N move The calculation rules include: when there are containers in the row where the target container slot is located that conflict with the vessel name or bill of lading number of the container waiting to enter the yard, for each conflicting container, N move Increase by 1; R waste Satisfy the following condition: If the number of empty positions in the target row is less than 10% of the theoretical maximum heap size, then R is forcibly set. waste =1.

5. A container space recommendation system based on reinforcement learning, characterized in that, To implement the container space recommendation method as described in any one of claims 1-4, the method includes: The data acquisition module is used to acquire yard status data and attribute data of containers waiting to enter the yard; The data encoding module is used to encode yard status data and container attribute data to be delivered into a joint state vector; The bin recommendation module is used to input the joint state vector into the trained bin recommendation model, calculate the action value of all available bins, select the bin with the highest action value as the recommended bin, and output the location information of the recommended bin. The container location recording module is used to record the location information of the actually selected container location when the actual selected container location is inconsistent with the recommended container location. The feedback signal generation module is used to generate a feedback signal based on the actual selected container position. The experience replay and model update module is used to store the joint state vector, actual selected container locations, feedback signals, and yard state data after container location selection into the experience replay pool, and update the parameters of the container location recommendation model by sampling training data.

6. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the container space recommendation method as described in any one of claims 1-4.

7. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the steps of the container space recommendation method as described in any one of claims 1-4.

Images