Logistics operation multi-target supervision method based on complex logistics field group

Abstract

The invention discloses a logistics operation multi-target supervision method based on a complex logistics field group, and the method comprises the steps: collecting logistics data: collecting various types of original logistics data in the complex logistics field group; preprocessing the data; carrying out graph structure modeling, constructing graph structure representation of a complex logistics field group by utilizing a graph neural network, and carrying out initialization operation of features; reinforcement learning modeling: modeling logistics operation multi-target supervision under the complex logistics field group into a reinforcement learning problem; hierarchical strategy optimization is carried out, and logistics operation multi-target supervision is divided into two levels by combining a hierarchical reinforcement learning algorithm: a top-level strategy is used for decision distribution of global tasks, and a bottom-level strategy is used for execution optimization of specific sub-tasks; and performing job execution and feedback regulation and control. According to the logistics operation multi-target supervision method based on the complex logistics field group, multiple tasks in logistics operation can be dynamically monitored, and comprehensive supervision of multiple targets such as logistics efficiency, transportation cost and distribution delay is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of logistics management and optimization, and more specifically, to a multi-objective supervision method for logistics operations based on a complex logistics field group. Background Art

[0002] With the rapid development of the global logistics industry and the popularization of e-commerce, the modern logistics system has gradually evolved into a highly complex and widely distributed network system. The logistics network not only covers traditional transportation and warehousing, but also involves multiple links such as inventory management, order processing, sorting and distribution, and reverse logistics. These logistics operations usually occur at multiple logistics nodes in different geographical locations, such as storage centers, distribution centers, transfer stations, etc., forming a complex logistics ecosystem, that is, a logistics cluster. In a complex logistics cluster, there is a dynamic collaborative relationship between the logistics nodes, and the operation of each node will directly or indirectly affect the operating efficiency and reliability of the entire logistics network.

[0003] In complex logistics clusters, traditional logistics operation supervision methods are difficult to cope with: (1) Single-target supervision. Traditional logistics operation management usually focuses on a single target, such as improving transportation speed or reducing inventory costs, and cannot take into account multiple targets such as logistics efficiency, resource utilization, and safety. In actual logistics scenarios, this single-target optimization method often leads to a decline in the performance of other logistics operations, limiting the operational efficiency of the overall logistics system. (2) Inefficient manual supervision. Traditional logistics operation supervision relies on manual monitoring, data collection, and manual scheduling. As the scale of logistics clusters expands, the manual supervision model has become difficult to adapt to the needs of large-scale, multi-node, and multi-link logistics scenarios. (3) Lack of real-time monitoring and dynamic adjustment. In traditional methods, the monitoring and scheduling of logistics operations are usually based on preset rules or empirical judgments, and cannot achieve intelligent analysis and dynamic adjustment of real-time data. When the situation in the logistics cluster suddenly changes (such as sudden order peaks, transportation equipment failures, etc.), the system is difficult to respond in time, resulting in low logistics operation efficiency and waste of resources.

[0004] Complex logistics clusters are usually composed of multiple logistics nodes, each of which has different operational objectives, and these objectives often affect each other. Therefore, in complex logistics clusters, management decision makers need to balance multiple objectives such as logistics efficiency, transportation costs, and delivery delays at the same time, realize the multi-objective supervision needs of logistics operations, and create an efficient, low-cost, safe, and resource-optimized logistics operation environment.

[0005] Therefore, there is an urgent need for a multi-objective supervision method for logistics operations based on complex logistics sites. Summary of the invention

[0006] The purpose of the present invention is to provide a multi-objective supervision method for logistics operations based on complex logistics field groups to solve the problems in the above-mentioned prior art, and to dynamically monitor multiple tasks in logistics operations to achieve comprehensive supervision of multiple objectives such as logistics efficiency, transportation costs and delivery delays.

[0007] The present invention provides a multi-objective supervision method for logistics operations based on a complex logistics field group, which includes:

[0008] Step S1: Logistics data collection, collecting various types of original logistics data in complex logistics field groups;

[0009] Step S2: Data preprocessing, preprocessing the original logistics data;

[0010] Step S3: Graph structure modeling, using graph neural network to construct a graph structure representation of a complex logistics field group and perform feature initialization operations, where nodes represent logistics entities and edges represent transportation connections between logistics nodes. The graph neural network model is used to update and aggregate the features of each node and edge in the logistics network, and a global feature representation of the complex logistics field group network is generated based on the message passing mechanism;

[0011] Step S4: reinforcement learning modeling, modeling the multi-objective supervision of logistics operations under complex logistics field groups into a reinforcement learning problem;

[0012] Step S5: hierarchical strategy optimization, combining the hierarchical reinforcement learning algorithm to divide the multi-objective supervision of logistics operations into two levels: the top-level strategy is used for decision allocation of global tasks, and the bottom-level strategy is used for execution optimization of specific subtasks, and then the top-level strategy and the bottom-level strategy are trained separately;

[0013] Step S6: Operation execution and feedback control. During the execution of logistics operations, the logistics system status is monitored in real time, and the decision is dynamically adjusted according to the feedback mechanism of reinforcement learning.

[0014] In the above-mentioned multi-objective supervision method for logistics operations based on complex logistics field groups, preferably, the step S1: logistics data collection specifically includes:

[0015] Use IoT sensors and GPS devices to collect various raw logistics data in complex logistics clusters, including:

[0016] Deploy a variety of IoT sensing devices at multiple key logistics nodes in the logistics field group, wherein the key logistics nodes include at least one of a storage center, a sorting center, and a transportation vehicle, and the IoT sensing devices include at least one of a camera, an RFID sensor, a GPS positioning device, and a temperature and humidity sensor;

[0017] Collect multi-dimensional data of logistics operations in real time through the IoT sensing device, wherein the multi-dimensional data includes at least one of cargo location, transportation status, storage inventory, and cargo transportation quantity;

[0018] Use GPS positioning devices to obtain real-time path data of transport vehicles, and combine with map API to obtain traffic status and predicted path travel time.

[0019] In the above-mentioned multi-objective supervision method for logistics operations based on complex logistics field groups, preferably, the step S2: data preprocessing specifically includes:

[0020] The raw logistics data is preprocessed by using at least one of missing data filling, outlier processing and normalization processing.

[0021] Wherein, the missing data filling includes: filling the missing items in the collected logistics data using interpolation method or mean filling method;

[0022] The outlier processing includes: using the z-score method to detect outliers in the data, removing outliers or performing data smoothing;

[0023] The normalization process includes: performing min-max normalization on the node data and the path data, and converting data of different dimensions into the same scale range.

[0024] In the above-mentioned multi-objective supervision method for logistics operations based on complex logistics field groups, preferably, the step S3: graph structure modeling specifically includes:

[0025] Abstracting the complex logistics field network into a graph structure , where V represents the set of logistics nodes, and each node With characteristic vector , represents the logistics attributes of the node, and the characteristic matrix of all nodes can be expressed as a matrix , where d represents the feature dimension, the edge set E represents the connection between logistics nodes, and the edge feature is represented as a matrix ,in, Representation Node and nodes The transport relationship between

[0026] In graph neural networks, node features are updated through message transmission and aggregation with the features of adjacent nodes. , its feature update process can be expressed as:

[0027]

[0028] in, represents the feature representation of node i at layer l+1, represents the set of neighbor nodes of node i, and are the weight matrix and bias term of the lth layer respectively, represents the normalization constant between nodes i and j, represents a non-linear activation function.

[0029] In the above-mentioned multi-objective supervision method for logistics operations based on complex logistics field groups, preferably, the step S4: reinforcement learning modeling specifically includes:

[0030] According to the definition of Markov decision process, the logistics operation scheduling and resource optimization process modeling is represented as a five-tuple ,

[0031] Among them, S represents the state space, which is used to represent the current state of the entire logistics field group system. The state space S can be expressed as the following state vector:

[0032]

[0033] in, Represents the state of the i-th logistics node at time t. Due to the initial feature extraction of the graph neural network, the state space is the embedding vector representation of logistics nodes and edges. At time t, the state space It can be expressed as:

[0034]

[0035] in, It is the feature vector embedding representation extracted by the final layer of the graph neural network for logistics node i;

[0036] A represents the action space. Each action in the action space A represents the logistics scheduling decision taken by the system in the current state. The action space A is represented by the following state vector:

[0037]

[0038] in, Represents different scheduling strategies or transportation adjustment plans made by logistics node i at time t;

[0039] P represents the state transition probability, Describes the transition to the next state after taking action a in the current state s probability;

[0040] R represents the reward function, Defines the immediate reward obtained by the logistics system after taking action a in state s, which is used to evaluate the feedback after each action is executed;

[0041] represents the discount factor, , used to measure the importance of current rewards and future rewards, if The larger the value, the more the system will focus on long-term returns. The smaller it is, the more the system will value immediate returns.

[0042] In the above-mentioned multi-objective supervision method for logistics operations based on complex logistics field groups, preferably, the step S5: hierarchical strategy optimization specifically includes:

[0043] During the training process of hierarchical reinforcement learning strategies, strategies at different levels are optimized based on different reinforcement learning methods: the top-level strategy is optimized through a policy-based reinforcement learning method, the bottom-level strategy is optimized through a value-based reinforcement learning method, and the strategy is optimized through repeated state sampling, action selection, and reward calculation during training.

[0044] In the above-mentioned multi-objective supervision method for logistics operations based on complex logistics field groups, preferably, the top-level strategy specifically includes:

[0045] The top-level strategy is used to select macro tasks, thereby guiding the bottom-level strategy to perform specific operations. The optimization goal of the top-level strategy is to maximize the global cumulative return through repeated decision updates. Its optimization goal can be defined as:

[0046]

[0047] in, Indicates that the top-level policy is in state Next select action The probability distribution of represents the parameters of the top-level strategy, represents the discount factor used to weigh immediate rewards against future rewards, represents the immediate reward received by the top-level strategy at time step t,

[0048] The top-level strategy uses the proximal strategy optimization algorithm for reinforcement learning. The optimization of the top-level strategy includes the following steps:

[0049] Sampling phase: At each time step t, the top-level strategy is based on the current state Select Action , and obtain immediate rewards by interacting with the environment and the next state , these state-action pair samples will be collected for subsequent policy updates,

[0050] The strategy update specifically includes: The proximal strategy optimization algorithm uses the clip strategy gradient method to optimize the strategy by maximizing the following loss function:

[0051]

[0052] in, Represents the probability ratio of the new and old strategies, which is used to measure the impact of strategy updates. represents the advantage function, which is used to measure the advantage of the current action relative to the average level, represents a hyperparameter used to control the step size of the policy update.

[0053] Among them, the advantage function The calculation is done in the following way:

[0054]

[0055] in, It represents the cumulative expected return in the future after taking action a in a given state s. Indicates that the agent is in state s and follows the current strategy Action, the expected cumulative rewards in the future;

[0056] Finally, by optimizing the loss function , use the gradient descent algorithm to update the policy parameters :

[0057]

[0058] in, Represents the learning rate, which is used to control the step size of the update.

[0059] In the above-mentioned multi-objective supervision method for logistics operations based on complex logistics field groups, preferably, the underlying strategy specifically includes:

[0060] The underlying strategy is used to execute specific subtasks. Assume that the underlying strategy is , whose goal is to maximize the local reward :

[0061]

[0062] in, represents the discount factor, It represents the instant reward obtained by the underlying strategy at time t. Multiple operation objectives of the logistics task are added to the reward function setting of the underlying strategy. The corresponding mathematical expression is as follows:

[0063]

[0064] in, It represents the logistics efficiency of a single transport task, that is, the order processing volume. Indicates the transportation time of a single transportation task. represents the transportation cost of a single transportation task, represents the weight coefficient;

[0065] The optimization of the underlying strategy is performed through a reinforcement learning method based on the Q-value function. The Q-value function represents the expected cumulative return after selecting action a in state s. The update of the Q-value follows the Bellman equation:

[0066]

[0067] in, Indicates immediate reward, In the new state The best action to choose The corresponding Q value, that is, the future state , the highest reward the system can obtain by choosing the best action,

[0068] The goal of the underlying strategy is to maximize the Q value so that the underlying strategy can select the action that accumulates the maximum reward in the local task. A deep Q network is used to learn the Q value function, and the mean square error is used as the loss function to optimize the deep Q network. The corresponding expression is as follows:

[0069]

[0070] in, represents the parameters of the neural network, represents the target Q value, Represents the Q value predicted by the current network. By minimizing the loss function, the underlying strategy can learn to select the optimal action to execute the logistics strategy.

[0071] As described above, in the multi-objective supervision method for logistics operations based on complex logistics field groups, preferably, in the step S5: hierarchical strategy optimization, the overall training process of the hierarchical reinforcement learning includes:

[0072] Initialization, initializing top-level strategy and bottom-level strategy parameters;

[0073] Sampling optimization, where the top-level strategy selects a target for each local task, and the bottom-level strategy performs the specific logistics operations for that target;

[0074] Store experience, sample and store Experience right;

[0075] Update the underlying strategy, sample from the experience replay pool, and use the deep Q network to update the underlying strategy;

[0076] Update the top-level policy, combine the global reward signal and the end-policy optimization algorithm to update the top-level policy and optimize its subtask selection;

[0077] Repeat the above steps until the strategy converges.

[0078] The present invention provides a multi-objective supervision method for logistics operations based on complex logistics sites. Aiming at the shortcomings of single-objective supervision, inefficient manual supervision, lack of real-time monitoring and dynamic adjustment in traditional logistics operation supervision methods in complex logistics sites, a multi-objective supervision strategy for logistics operations based on complex logistics sites is provided, which can dynamically monitor multiple tasks in logistics operations and realize comprehensive supervision of multiple objectives such as logistics efficiency, transportation cost and delivery delay; can realize dynamic and multi-objective logistics operation management in complex logistics sites, and create an efficient, low-cost, safe and resource-optimized logistics operation environment; can dynamically monitor multiple tasks in logistics operations and realize comprehensive supervision of logistics operations with multiple objectives such as logistics efficiency, transportation cost and delivery delay, effectively improving the supervision efficiency and optimizing scheduling decisions in complex logistics scenarios, which is of great significance for optimizing logistics management in modern logistics systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described below with reference to the accompanying drawings, in which:

[0080] Figure 1 A flowchart of an embodiment of a multi-objective supervision method for logistics operations based on a complex logistics field group provided by the present invention;

[0081] Figure 2 A model architecture diagram provided by the present invention;

[0082] Figure 3 This is a diagram of the internal structure of the deep Q network used in the underlying strategy of the present invention. DETAILED DESCRIPTION

[0083] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. The description of the exemplary embodiments is merely illustrative and is in no way intended to limit the present disclosure and its application or use. The present disclosure can be implemented in many different forms and is not limited to the embodiments described herein. These embodiments are provided to make the present disclosure thorough and complete and to fully express the scope of the present disclosure to those skilled in the art. It should be noted that unless otherwise specifically stated, the relative arrangement of the components and steps, the composition of the materials, the numerical expressions and the numerical values ​​set forth in these embodiments should be interpreted as being merely exemplary and not as limiting.

[0084] The words "first", "second" and similar words used in this disclosure do not indicate any order, quantity or importance, but are only used to distinguish different parts. The words "include" or "comprises" and similar words mean that the elements before the word include the elements listed after the word, and do not exclude the possibility of including other elements. "Up", "down" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0085] In the present disclosure, when a specific component is described as being located between a first component and a second component, there may or may not be an intermediate component between the specific component and the first component or the second component. When a specific component is described as being connected to other components, the specific component may be directly connected to the other components without an intermediate component, or may not be directly connected to the other components but have an intermediate component.

[0086] All terms (including technical terms or scientific terms) used in the present disclosure have the same meanings as those understood by ordinary technicians in the field to which the present disclosure belongs, unless otherwise specifically defined. It should also be understood that terms defined in general dictionaries, such as general dictionaries, should be interpreted as having meanings consistent with their meanings in the context of the relevant technology, and should not be interpreted in an idealized or extremely formal sense, unless explicitly defined herein.

[0087] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0088] Logistics operation management is an important part of the modern logistics system. Reasonable and effective logistics operation management methods can provide the logistics industry with more efficient, accurate and flexible scheduling and management tools, and improve the overall efficiency of logistics operations.

[0089] like Figure 1 and Figure 2 As shown, the multi-objective supervision method for logistics operations based on complex logistics field groups provided in this embodiment includes the following steps in actual implementation:

[0090] Step S1: Logistics data collection: collecting various types of original logistics data in complex logistics sites.

[0091] Specifically, IoT sensors and GPS devices are used to collect various types of raw logistics data in complex logistics clusters. In one embodiment of the multi-objective supervision method for logistics operations based on complex logistics clusters of the present invention, step S1 may specifically include:

[0092] Step S11: deploy multiple IoT sensing devices at multiple key logistics nodes in the logistics cluster.

[0093] Among them, the key logistics nodes include at least one of a warehousing center, a sorting center and a transport vehicle, and the Internet of Things sensing equipment includes at least one of a camera, an RFID sensor, a GPS positioning device and a temperature and humidity sensor.

[0094] Step S12: Collect multi-dimensional data of logistics operations in real time through the IoT sensing device.

[0095] The multi-dimensional data includes at least one of cargo location, transportation status, warehouse inventory and cargo transportation quantity.

[0096] Step S13: Use the GPS positioning device to obtain the real-time path data of the transport vehicle, and combine it with the map API to obtain the traffic status and predicted path travel time.

[0097] Step S2: Data preprocessing, preprocessing the original logistics data.

[0098] Data preprocessing can provide high-quality data input for subsequent complex logistics field network modeling and hierarchical strategy optimization. Specifically, in the present invention, at least one of missing data filling, outlier processing and normalization processing is used to preprocess the original logistics data.

[0099] The missing data filling includes: filling the missing items in the collected logistics data using interpolation or mean filling method, and for the missing values ​​of the node inventory, linear interpolation can be performed using the inventory data of the previous and next time points;

[0100] The outlier processing includes: using the z-score method to detect outliers in the data (such as abnormal transportation time or inventory fluctuations), removing outliers or performing data smoothing;

[0101] The normalization process includes: performing min-max normalization on the node data and the path data, and converting data of different dimensions into the same scale range.

[0102] Among them, in the missing data filling, assuming that the inventory data of a logistics node is missing, linear interpolation can be performed based on the data of adjacent time points:

[0103]

[0104] For non-time series data, historical averages are used to fill missing values.

[0105] In outlier processing, the standardization calculation process for feature x is as follows:

[0106]

[0107] in, represents the mean of feature x, Indicates standard deviation. When the absolute value of z is greater than a certain threshold (for example, 3), the data can be considered as an outlier and needs to be removed or corrected.

[0108] In the normalization process, for example, inventory, transportation time, order quantity, etc. can be normalized to the [0,1] interval to improve the efficiency of model training. The min-max normalization calculation process is as follows:

[0109]

[0110] Step S3: Graph structure modeling, using graph neural network to construct a graph structure representation of the complex logistics field group, and perform feature initialization operations, where nodes represent logistics entities, edges represent transportation connections between logistics nodes, and use the graph neural network model to update and aggregate the features of each node and edge in the logistics network, and generate a global feature representation of the complex logistics field group network based on the message passing mechanism.

[0111] Among them, logistics entities can be, for example, storage centers, distribution centers, transport vehicles, etc. A complex logistics field group is usually a network composed of multiple distribution centers, storage centers, transport vehicles and other logistics nodes. It can be modeled by a graph structure, with each logistics node represented as a vertex in the graph and the relationship between nodes represented as edges, thereby accurately describing the dynamic relationship of the entire logistics network. The present invention uses a graph neural network to perform graph structure modeling and feature extraction on the logistics network. Specifically, the complex logistics field group network is abstracted into a graph structure. , where V represents a collection of logistics nodes, including warehousing facilities, distribution centers, transportation vehicles and other logistics nodes. Each node With characteristic vector , represents the logistics attribute of the node, which includes at least one of inventory, cargo quantity and vehicle status. The feature matrix of all nodes can be expressed as a matrix , where d represents the feature dimension, the edge set E represents the connection between logistics nodes, including at least one of the attributes such as transportation time, transportation distance and path congestion, and the edge feature is represented as a matrix ,in, Representation Node and nodes The transport relationship between

[0112] In graph neural networks, node features are updated through message transmission and aggregation with the features of adjacent nodes. , its feature update process can be expressed as:

[0113]

[0114] in, represents the feature representation of node i at layer l+1, represents the set of neighbor nodes of node i, and are the weight matrix and bias term of the lth layer respectively, represents the normalization constant between nodes i and j, represents a non-linear activation function.

[0115] By iteratively updating multiple layers, the graph neural network can extract the global feature representation of logistics nodes in the entire logistics field group network layer by layer.

[0116] Step S4: Reinforcement learning modeling, modeling the multi-objective supervision of logistics operations under complex logistics field groups into a reinforcement learning problem.

[0117] Through reinforcement learning modeling, the subsequent hierarchical reinforcement learning strategy execution is facilitated. The present invention adopts a hierarchical reinforcement learning algorithm to optimize the scheduling and resource allocation of logistics operations. Specifically, according to the definition of Markov decision process, the logistics operation scheduling and resource optimization process modeling is represented as a five-tuple ,

[0118] Among them, S represents the state space, which is used to represent the current state of the entire logistics field group system, such as the inventory level, traffic conditions, vehicle location, warehouse storage status, etc. of each logistics node. The state space S can be expressed as the following state vector:

[0119]

[0120] in, Represents the state of the i-th logistics node at time t. Due to the initial feature extraction of the graph neural network, the state space is the embedding vector representation of logistics nodes and edges. At time t, the state space It can be expressed as:

[0121]

[0122] in, It is the feature vector embedding representation extracted by the final layer of the graph neural network for logistics node i;

[0123] A represents the action space. Each action in the action space A represents the logistics scheduling decision taken by the system in the current state, such as vehicle path selection, resource reallocation, and order priority adjustment. The action space A is represented by the following state vector:

[0124]

[0125] in, Represents different scheduling strategies or transportation adjustment plans made by logistics node i at time t;

[0126] P represents the state transition probability, Describes the transition to the next state after taking action a in the current state s probability.

[0127] The state transfer function can be estimated through historical data or obtained through simulation. For example, suppose the current system is in state , take an action in this state After that, transfer to the state The probability is:

[0128]

[0129] If the action of "dispatching more vehicles to a certain route" is taken, the next state may be increased vehicle utilization, improved transportation efficiency, reduced order processing time, etc.

[0130] R represents the reward function, Defines the immediate reward obtained by the logistics system after taking action a in state s, which is used to evaluate the feedback after each action is executed;

[0131] represents the discount factor, , used to measure the importance of current rewards and future rewards, if The larger the value, the more the system will focus on long-term returns. The smaller it is, the more the system will value immediate returns.

[0132] Step S5: Hierarchical strategy optimization, combining the hierarchical reinforcement learning algorithm to divide the multi-objective supervision of logistics operations into two levels: the top-level strategy is used for decision allocation of global tasks, and the bottom-level strategy is used for execution optimization of specific subtasks. Then the top-level strategy and the bottom-level strategy are trained separately.

[0133] Specifically, during the training process of hierarchical reinforcement learning strategies, strategies at different levels are optimized based on different reinforcement learning methods: the top-level strategy is optimized by a policy-based reinforcement learning method, and the bottom-level strategy is optimized by a value-based reinforcement learning method. This training method enables each layer of strategy to better optimize its goals, and optimizes strategies through repeated state sampling, action selection, and reward calculation during training.

[0134] The top-level strategy specifically includes:

[0135] The top-level strategy is used to select macro tasks (such as cargo scheduling or resource optimization) to guide the bottom-level strategy to perform specific operations. The optimization goal of the top-level strategy is to maximize the global cumulative return through repeated decision updates. Its optimization goal can be defined as:

[0136]

[0137] in, Indicates that the top-level policy is in state Next select action The probability distribution of represents the parameters of the top-level strategy, represents the discount factor used to weigh immediate rewards against future rewards, represents the immediate reward received by the top-level strategy at time step t,

[0138] The top-level policy uses the proximal policy optimization algorithm for reinforcement learning. The proximal policy optimization algorithm is a policy-based reinforcement learning algorithm that uses a clipped objective function to ensure the stability of policy updates. The optimization of the top-level policy includes the following steps:

[0139] Sampling phase: At each time step t, the top-level strategy is based on the current state Select Action , and obtain immediate rewards by interacting with the environment and the next state , these state-action pair samples will be collected for subsequent policy updates,

[0140] The strategy update specifically includes: The proximal strategy optimization algorithm uses the clip strategy gradient method to optimize the strategy by maximizing the following loss function:

[0141]

[0142] in, Represents the probability ratio of the new and old strategies, which is used to measure the impact of strategy updates. represents the advantage function, which is used to measure the advantage of the current action relative to the average level, Represents a hyperparameter, which is used to control the step size of the policy update to prevent excessive policy deviation.

[0143] Among them, the advantage function The calculation is done in the following way:

[0144]

[0145] in, It represents the cumulative expected return in the future after taking action a in a given state s. Indicates that the agent is in state s and follows the current strategy Action, the expected cumulative rewards in the future;

[0146] Finally, by optimizing the loss function , use the gradient descent algorithm to update the policy parameters :

[0147]

[0148] in, Represents the learning rate, which is used to control the step size of the update.

[0149] Through the proximal policy optimization algorithm, the top-level policy can maximize the cumulative reward of the global task. Through continuous policy updates, the top-level policy can maximize the global reward in global decision-making.

[0150] Furthermore, the underlying strategy specifically includes:

[0151] The underlying strategy is used to perform specific subtasks, such as optimizing vehicle scheduling and resource allocation at a logistics site. Assume that the underlying strategy is , whose goal is to maximize the local reward :

[0152]

[0153] in, represents the discount factor, It represents the instant reward obtained by the underlying strategy at time t. Multiple operation objectives of logistics tasks are added to the reward function setting of the underlying strategy, such as logistics efficiency, transportation time and transportation cost of a single transportation task. The design of this multi-objective reward function will help the underlying strategy to balance different logistics operation objectives during the optimization process to ensure the best overall effect. The corresponding mathematical expression is as follows:

[0154]

[0155] in, It represents the logistics efficiency of a single transport task, that is, the order processing volume. Indicates the transportation time of a single transportation task. represents the transportation cost of a single transportation task, Represents the weight coefficient. By adjusting these weight values, different logistics operation objectives can be flexibly balanced according to actual needs.

[0156] The optimization of the underlying strategy is performed through a reinforcement learning method based on the Q-value function. The Q-value function represents the expected cumulative return after selecting action a in state s. The update of the Q-value follows the Bellman equation:

[0157]

[0158] in, Indicates immediate reward, In the new state The best action to choose The corresponding Q value, that is, the future state , the highest reward the system can obtain by choosing the best action,

[0159] The goal of the underlying strategy is to maximize the Q value so that the underlying strategy can choose the action that accumulates the maximum reward in the local task, such as Figure 3 As shown, the present invention adopts a deep Q network to learn the Q value function, and uses the mean square error as the loss function to optimize the deep Q network, and its corresponding expression is as follows:

[0160]

[0161] in, represents the parameters of the neural network, represents the target Q value, Represents the Q value predicted by the current network. By minimizing the loss function, the underlying strategy can learn to select the optimal action to execute the logistics strategy.

[0162] Furthermore, in the step S5: hierarchical strategy optimization, the overall training process of the hierarchical reinforcement learning includes:

[0163] Initialization, initializing top-level strategy and bottom-level strategy parameters;

[0164] Sampling optimization, where the top-level strategy selects a goal for each local task (e.g., distribution center selection), and the bottom-level strategy performs the specific logistics operations for that goal (e.g., vehicle scheduling);

[0165] Store experience, sample and store Experience right;

[0166] Update the underlying strategy, sample from the experience replay pool, and use the deep Q network to update the underlying strategy;

[0167] Update the top-level policy, combine the global reward signal and the end-policy optimization algorithm to update the top-level policy and optimize its subtask selection;

[0168] Repeat the above steps until the strategy converges.

[0169] Through hierarchical optimization strategies, a globally efficient and low-cost logistics scheduling strategy can be output.

[0170] Step S6: Operation execution and feedback control. During the execution of logistics operations, the logistics system status is monitored in real time, and the decision is dynamically adjusted according to the feedback mechanism of reinforcement learning.

[0171] Through feedback control, the system can automatically optimize scheduling and adjust resource allocation when facing external changes (such as traffic jams, sudden orders, etc.).

[0172] The present invention is applicable to large-scale, multi-level, multi-objective logistics scheduling and resource optimization scenarios.

[0173] The multi-objective supervision method for logistics operations based on complex logistics sites provided in the embodiments of the present invention aims at the shortcomings of single-objective supervision, inefficient manual supervision, lack of real-time monitoring and dynamic adjustment in traditional logistics operation supervision methods in complex logistics sites, and provides a multi-objective supervision strategy for logistics operations based on complex logistics sites. The strategy can dynamically monitor multiple tasks in logistics operations to achieve comprehensive supervision of multiple objectives such as logistics efficiency, transportation costs and delivery delays; it can achieve dynamic and multi-objective logistics operation management in complex logistics sites to create an efficient, low-cost, safe and resource-optimized logistics operation environment; it can dynamically monitor multiple tasks in logistics operations to achieve comprehensive supervision of logistics operations with multiple objectives such as logistics efficiency, transportation costs and delivery delays, and effectively improve the supervision efficiency and optimize scheduling decisions in complex logistics scenarios, which is of great significance for optimizing logistics management in modern logistics systems.

[0174] So far, various embodiments of the present disclosure have been described in detail. In order to avoid obscuring the concept of the present disclosure, some details known in the art are not described. Based on the above description, those skilled in the art can fully understand how to implement the technical solution disclosed here.

[0175] Although some specific embodiments of the present disclosure have been described in detail by way of examples, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present disclosure. It should be understood by those skilled in the art that the above embodiments may be modified or some technical features may be replaced by equivalents without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. A multi-objective supervision method for logistics operations based on complex logistics sites, characterized in that: include: Step S1: Logistics data collection, collecting various types of original logistics data in complex logistics field groups; Step S2: Data preprocessing, preprocessing the original logistics data; Step S3: Graph structure modeling, using graph neural network to construct a graph structure representation of a complex logistics field group and perform feature initialization operations, where nodes represent logistics entities and edges represent transportation connections between logistics nodes. The graph neural network model is used to update and aggregate the features of each node and edge in the logistics network, and a global feature representation of the complex logistics field group network is generated based on the message passing mechanism; Step S4: reinforcement learning modeling, modeling the multi-objective supervision of logistics operations under complex logistics field groups into a reinforcement learning problem; Step S5: hierarchical strategy optimization, combining the hierarchical reinforcement learning algorithm to divide the multi-objective supervision of logistics operations into two levels: the top-level strategy is used for decision allocation of global tasks, and the bottom-level strategy is used for execution optimization of specific subtasks, and then the top-level strategy and the bottom-level strategy are trained separately; Step S6: Operation execution and feedback control. During the execution of logistics operations, the logistics system status is monitored in real time, and the decision is dynamically adjusted according to the feedback mechanism of reinforcement learning.

2. The multi-objective supervision method for logistics operations based on complex logistics field groups according to claim 1 is characterized in that: The step S1: logistics data collection specifically includes: Use IoT sensors and GPS devices to collect various raw logistics data in complex logistics clusters, including: Deploy a variety of IoT sensing devices at multiple key logistics nodes in the logistics field group, wherein the key logistics nodes include at least one of a storage center, a sorting center, and a transportation vehicle, and the IoT sensing devices include at least one of a camera, an RFID sensor, a GPS positioning device, and a temperature and humidity sensor; Collect multi-dimensional data of logistics operations in real time through the IoT sensing device, wherein the multi-dimensional data includes at least one of cargo location, transportation status, storage inventory, and cargo transportation quantity; Use GPS positioning devices to obtain real-time path data of transport vehicles, and combine with map API to obtain traffic status and predicted path travel time.

3. The multi-objective supervision method for logistics operations based on complex logistics field groups according to claim 1 is characterized in that: The step S2: data preprocessing, specifically includes: The raw logistics data is preprocessed by using at least one of missing data filling, outlier processing and normalization processing. Wherein, the missing data filling includes: filling the missing items in the collected logistics data using interpolation method or mean filling method; The outlier processing includes: using the z-score method to detect outliers in the data, removing outliers or performing data smoothing; The normalization process includes: performing min-max normalization on the node data and the path data, and converting data of different dimensions into the same scale range.

4. The multi-objective supervision method for logistics operations based on complex logistics field groups according to claim 1 is characterized in that: The step S3: graph structure modeling, specifically includes: Abstracting the complex logistics field network into a graph structure , where V represents the set of logistics nodes, and each node With characteristic vector , represents the logistics attributes of the node, and the characteristic matrix of all nodes can be expressed as a matrix , where d represents the feature dimension, the edge set E represents the connection between logistics nodes, and the edge feature is represented as a matrix ,in, Representation Node and nodes The transport relationship between In graph neural networks, node features are updated through message transmission and aggregation with the features of adjacent nodes. , its feature update process can be expressed as: ,in, represents the feature representation of node i at layer l+1, represents the set of neighbor nodes of node i, and are the weight matrix and bias term of the lth layer respectively, represents the normalization constant between node i and node j, represents a non-linear activation function.

5. The multi-objective supervision method for logistics operations based on complex logistics field groups according to claim 1 is characterized in that: The step S4: reinforcement learning modeling, specifically includes: According to the definition of Markov decision process, the logistics operation scheduling and resource optimization process modeling is represented as a five-tuple , Among them, S represents the state space, which is used to represent the current state of the entire logistics field group system. The state space S can be expressed as the following state vector: ,in, Represents the state of the i-th logistics node at time t. Due to the initial feature extraction of the graph neural network, the state space is the embedding vector representation of logistics nodes and edges. At time t, the state space It can be expressed as: ,in, It is the feature vector embedding representation extracted by the final layer of the graph neural network for logistics node i; A represents the action space. Each action in the action space A represents the logistics scheduling decision taken by the system in the current state. The action space A is represented by the following state vector: ,in, Represents different scheduling strategies or transportation adjustment plans made by logistics node i at time t; P represents the state transition probability, Describes the transition to the next state after taking action a in the current state s probability; R represents the reward function, Defines the immediate reward obtained by the logistics system after taking action a in state s, which is used to evaluate the feedback after each action is executed; represents the discount factor, , used to measure the importance of current rewards and future rewards, if The larger the value, the more the system will focus on long-term returns. The smaller it is, the more the system will value immediate returns.

6. The multi-objective supervision method for logistics operations based on complex logistics field groups according to claim 1 is characterized in that: The step S5: layered strategy optimization specifically includes: During the training process of hierarchical reinforcement learning strategies, strategies at different levels are optimized based on different reinforcement learning methods: the top-level strategy is optimized through a policy-based reinforcement learning method, the bottom-level strategy is optimized through a value-based reinforcement learning method, and the strategy is optimized through repeated state sampling, action selection, and reward calculation during training.

7. The multi-objective supervision method for logistics operations based on complex logistics field groups according to claim 6 is characterized in that: The top-level strategy specifically includes: The top-level strategy is used to select macro tasks, thereby guiding the bottom-level strategy to perform specific operations. The optimization goal of the top-level strategy is to maximize the global cumulative return through repeated decision updates. Its optimization goal can be defined as: ,in, Indicates that the top-level policy is in state Next select action The probability distribution of represents the parameters of the top-level strategy, represents the discount factor used to weigh immediate rewards against future rewards, represents the immediate reward received by the top-level strategy at time step t, The top-level strategy uses the proximal strategy optimization algorithm for reinforcement learning. The optimization of the top-level strategy includes the following steps: Sampling phase: At each time step t, the top-level strategy is based on the current state Select Action , and obtain immediate rewards by interacting with the environment and the next state , these state-action pair samples will be collected for subsequent policy updates, The strategy update specifically includes: The proximal strategy optimization algorithm uses the clip strategy gradient method to optimize the strategy by maximizing the following loss function: ,in, Represents the probability ratio of the new and old strategies, which is used to measure the impact of strategy updates. represents the advantage function, which is used to measure the advantage of the current action relative to the average level, represents a hyperparameter used to control the step size of the policy update. Among them, the advantage function The calculation is done in the following way: ,in, It represents the cumulative expected return in the future after taking action a in a given state s. Indicates that the agent is in state s and follows the current strategy Action, the expected cumulative rewards in the future; Finally, by optimizing the loss function , use the gradient descent algorithm to update the policy parameters : ,in, Represents the learning rate, which is used to control the step size of the update.

8. The multi-objective supervision method for logistics operations based on complex logistics field groups according to claim 6 is characterized in that: The underlying strategy specifically includes: The underlying strategy is used to execute specific subtasks. Assume that the underlying strategy is , whose goal is to maximize the local reward : ,in, represents the discount factor, It represents the instant reward obtained by the underlying strategy at time t. Multiple operation objectives of the logistics task are added to the reward function setting of the underlying strategy. The corresponding mathematical expression is as follows: ,in, It represents the logistics efficiency of a single transport task, that is, the order processing volume. Indicates the transportation time of a single transportation task. represents the transportation cost of a single transportation task, represents the weight coefficient; The optimization of the underlying strategy is performed through a reinforcement learning method based on the Q-value function. The Q-value function represents the expected cumulative return after selecting action a in state s. The update of the Q-value follows the Bellman equation: ,in, Indicates immediate reward, In the new state The best action to choose The corresponding Q value, that is, the future state , the highest reward the system can obtain by choosing the best action, The goal of the underlying strategy is to maximize the Q value so that the underlying strategy can select the action that accumulates the maximum reward in the local task. A deep Q network is used to learn the Q value function, and the mean square error is used as the loss function to optimize the deep Q network. The corresponding expression is as follows: ,in, represents the parameters of the neural network, represents the target Q value, Represents the Q value predicted by the current network. By minimizing the loss function, the underlying strategy can learn to select the optimal action to execute the logistics strategy.

9. The multi-objective supervision method for logistics operations based on complex logistics field groups according to claim 6 is characterized in that: In the step S5: hierarchical strategy optimization, the overall training process of the hierarchical reinforcement learning includes: Initialization, initializing top-level strategy and bottom-level strategy parameters; Sampling optimization, where the top-level strategy selects a target for each local task, and the bottom-level strategy performs the specific logistics operations for that target; Store experience, sample and store Experience right; Update the underlying strategy, sample from the experience replay pool, and use the deep Q network to update the underlying strategy; Update the top-level policy, combine the global reward signal and the end-policy optimization algorithm to update the top-level policy and optimize its subtask selection; Repeat the above steps until the strategy converges.

Images