A Pallet Transportation Management Method and System Based on Reinforcement Learning

Abstract

This invention provides a pallet transportation management method and system based on reinforcement learning, belonging to the field of pallet transportation management technology. The method includes: collecting and preprocessing multi-dimensional data to obtain transportation characteristics and interaction correlation data; conducting an initial assessment of transportation status to obtain a multi-AGV diagnostic set and initial assessment results; constructing a dynamic transportation topology graph to infer optimization trends; constructing and correcting a transportation bottleneck causal graph to obtain the corrected bottleneck probability; adjusting scheduling priorities and verifying bottleneck correlations to obtain the final scheduling level, verification data, and transportation risk level; constructing and updating a multi-modal transportation embedding model; and formulating a full-cycle optimization plan. This invention uses multi-dimensional rules and intelligent models to initially assess status and screen anomalies; combines dynamic topology and causal models to predict bottlenecks and locate root causes; dynamically adjusts scheduling and formulates a full-cycle plan to achieve adaptive scheduling optimization, improve resource allocation and equipment collaboration efficiency, ensure continuous and stable production, and improve economic benefits.

Description

Technical Field

[0001] This invention belongs to the field of pallet transportation management technology, specifically a pallet transportation management method and system based on reinforcement learning. Background Technology

[0002] In modern industrial production, material transportation is a core link in ensuring continuous and efficient production. Its operational efficiency and stability directly affect the overall production progress and economic benefits. Currently, material transportation in industrial settings largely relies on the collaborative operation of automated equipment. Among these, Automated Guided Vehicles (AGVs), as the core equipment of automated transportation, are widely used in various industrial scenarios, undertaking key tasks such as material transfer and pallet conveying.

[0003] However, in practical applications, many problems still need to be solved. Traditional transportation management methods mostly adopt fixed scheduling strategies, which are difficult to adapt to dynamic changes in the production process. They cannot respond in real time to fluctuations in AGV operating status, changes in environmental parameters, and various anomalies in multi-AGV collaborative interactions, easily leading to frequent transportation bottlenecks and unreasonable resource allocation. At the same time, existing management methods lack comprehensiveness and accuracy in assessing AGV transportation status, making it difficult to predict potential AGV failures and transportation bottleneck risks in advance. Often, problems are only addressed reactively after they occur, resulting in production interruptions and decreased transportation efficiency. In addition, the root cause analysis of AGV transportation bottlenecks is not in-depth enough, making it impossible to accurately locate the source of the problem. This leads to a lack of targeted optimization solutions, making it difficult to fundamentally improve the stability and efficiency of the AGV transportation system and meet the high-quality material transportation requirements of modern industrial large-scale and refined production. Summary of the Invention

[0004] This invention provides a pallet transportation management method and system based on reinforcement learning to address the shortcomings of existing technologies.

[0005] On one hand, the present invention provides a pallet transportation management method based on reinforcement learning, comprising: Collect AGV operation data, workstation status data, and pallet transportation interaction data, and preprocess them to obtain transportation characteristic data and interaction correlation data.

[0006] Based on transportation characteristic data, interactive correlation data, and production scheduling requirements, a preliminary assessment of transportation status is conducted, resulting in a multi-AGV diagnostic set and preliminary transportation assessment results.

[0007] Based on the multi-AGV diagnostic set and the initial transportation evaluation results, a dynamic transportation topology graph is constructed, and the transportation optimization trend is inferred through multi-agent deep reinforcement learning.

[0008] By combining historical bottleneck data and historical transportation status data, a causal graph of transportation bottlenecks is constructed and corrected to obtain the corrected bottleneck probability.

[0009] Based on the revised bottleneck probability and transportation optimization trend, the scheduling priority is adjusted, the bottleneck correlation is verified, and verification data is collected as needed to obtain the final scheduling level, verification data and transportation risk level.

[0010] Based on transportation feature data, interactive correlation data, and simulation parameters, a formatted training dataset is constructed to build a multimodal transportation embedding model, outputting transportation status assessment results. The multimodal transportation embedding model is then updated using the final scheduling level, validation data, and transportation risk level.

[0011] By combining transportation status assessment results with historical scheduling data, short-term, medium-term, and long-term full-cycle transportation optimization plans are formulated. Interpretable algorithms are used to obtain the root causes of bottlenecks and the links affecting transportation, generating scheduling suggestions, transportation optimization plans, and risk assessment reports.

[0012] According to the pallet transportation management method based on reinforcement learning provided by the present invention, the process of conducting an initial assessment of transportation status includes: Based on transportation characteristic data and interactive correlation data, basic transportation assessment rules are constructed.

[0013] The transportation feature data and interactive correlation data are compared and matched with the basic transportation assessment rules feature by feature. Abnormal features that exceed the normal value range are marked, and the degree of deviation and occurrence time of each abnormal feature are recorded.

[0014] The labeled abnormal features are input into a locally deployed deep learning edge model. The model analyzes the correlation between abnormal features, eliminates mislabeling caused by isolated noise points, outputs the transportation status level, and generates a preliminary transportation assessment result document.

[0015] According to the pallet transportation management method based on reinforcement learning provided by the present invention, the process of constructing basic transportation evaluation rules includes: The transportation assessment dimensions are divided. Based on the AGV's factory standard parameters, the production line's rated operating parameters, and historical bottleneck-free data, the normal value range of each dimension's characteristics is determined through statistical analysis, forming basic threshold rules.

[0016] Based on scheduling requirements, feature association constraint rules are set to clarify the linkage, matching, and response matching thresholds of features between the same or different AGVs.

[0017] Set an environment adaptation coefficient, and adjust the basic threshold rules according to the environmental parameters of the production workshop to obtain the threshold rules after environment adaptation.

[0018] Based on the threshold rules and feature association constraint rules after environmental adaptation, initial basic transportation assessment rules are formed.

[0019] Optimize and validate through historical bottleneck cases until the preset performance requirements are met, and complete the rule construction.

[0020] According to the pallet transportation management method based on reinforcement learning provided by the present invention, the process of constructing a dynamic transportation topology map includes: AGVs and workstations are mapped as graph nodes, and transportation intensity, transportation frequency, and bottleneck correlation are mapped as graph edge weights.

[0021] Based on the multi-AGV diagnostic set and the initial transportation evaluation results, the attribute features of the graph nodes and the weights of the graph edges are updated.

[0022] A graph construction algorithm is used to generate a dynamic transportation topology graph.

[0023] According to the pallet transportation management method based on reinforcement learning provided by the present invention, the process of obtaining the corrected bottleneck probability includes: The features of the dynamic transportation topology are input into a multi-agent deep reinforcement learning network, which outputs the initial bottleneck probability.

[0024] By combining historical bottleneck data with historical transportation status data, a cause-and-effect graph of transportation bottlenecks is constructed and corrected.

[0025] By incorporating the causal correlation strength of the transportation bottleneck causal graph into the initial bottleneck probability calculation as a weight, the corrected bottleneck probability is obtained.

[0026] According to the pallet transportation management method based on reinforcement learning provided by the present invention, the process of outputting the initial bottleneck probability includes: The node and edge features of the dynamic transportation topology graph are dimensionally aligned and normalized to obtain standardized graph features.

[0027] The standardized graph features are divided into continuous feature sequences according to the time series, and then input into the temporal convolutional layer of a multi-agent deep reinforcement learning network to extract the temporal dependencies of the feature sequences.

[0028] The output of the temporal convolutional layer is input into the graph convolutional layer to capture the spatial correlation features between nodes, thus obtaining a feature vector that integrates spatiotemporal information.

[0029] The feature vectors that integrate spatiotemporal information are input into the fully connected layer, and the initial bottleneck probability of each transportation node is calculated through the activation function, and the initial bottleneck probability is output.

[0030] According to the pallet transportation management method based on reinforcement learning provided by the present invention, the process of correcting the cause-effect graph of transportation bottlenecks includes: Based on historical bottleneck data and historical transportation status data, a Bayesian network is used to construct an initial transportation bottleneck causal graph, where nodes represent transportation characteristics and bottleneck types, edges represent the causal relationships between the two, and edge weights represent the initial causal relationship strength.

[0031] Based on bottleneck case data and corresponding transportation status evolution data, the confidence level of each causal relationship is calculated, and the initial causal relationship strength is updated.

[0032] If a new relationship exists between transportation characteristics and bottleneck types, it is added to the transportation bottleneck causal graph to obtain an updated transportation bottleneck causal graph.

[0033] The updated transportation bottleneck cause-effect graph is validated, unreasonable causal relationships are eliminated, and the strength of abnormal causal relationships is adjusted to obtain the corrected transportation bottleneck cause-effect graph.

[0034] Calculate the goodness-of-fit index of the corrected cause-effect graph of the transportation bottleneck. If the goodness-of-fit index reaches the preset threshold, the correction is completed.

[0035] If the requirement is not met, repeat the above process until the fit index meets the requirements.

[0036] According to the pallet transportation management method based on reinforcement learning provided by the present invention, the process of adjusting scheduling priorities and verifying bottleneck correlations includes: The initial scheduling level is obtained by setting three scheduling thresholds and matching and correcting the bottleneck probability.

[0037] The collaborative confidence level is calculated by combining the number of abnormal associations in the multi-AGV diagnostic set with the rate of change of the transportation optimization trend.

[0038] Adjust the initial scheduling level based on the collaborative confidence level, and output the adjusted scheduling level and the corresponding abnormal transportation scope that needs to be checked.

[0039] Based on the abnormal transportation range that needs to be checked, the operation-load characteristics of the corresponding abnormal transportation nodes in the multi-AGV diagnostic set are extracted and compared with the preset normal feature library to obtain the matching degree of operation-load characteristics and the corresponding abnormal feature information.

[0040] Based on the abnormal feature information corresponding to the running-load feature matching degree, visual evidence of abnormal transportation nodes, including driving trajectory and stopping time, is collected.

[0041] The analysis examines the matching degree of operational-load features and the feature synchronization and correlation of visual evidence to verify the effectiveness of bottleneck correlation.

[0042] According to the reinforcement learning-based pallet transportation management method provided by the present invention, the process of constructing a multimodal transportation embedding model includes: A multimodal fusion network is constructed, and transportation feature data, interaction correlation data and simulation parameters are input as training samples to build a formatted training dataset.

[0043] Using transportation status levels as labels, the network is trained on a formatted training dataset using the cross-entropy loss function to obtain the network parameters to be optimized.

[0044] The network parameters to be optimized are iteratively optimized to obtain a multimodal transportation embedding model that outputs transportation state assessment results.

[0045] On the other hand, the present invention also provides a pallet transportation management system based on reinforcement learning, comprising: The data preprocessing module is used to collect AGV operation, workstation status and pallet interaction data. Through preprocessing such as noise reduction, missing value filling and normalization, it outputs transportation feature data and interaction correlation data.

[0046] The initial status assessment module is used to construct assessment rules based on transportation characteristics, interactive data, and scheduling requirements, compare abnormal features, eliminate noise through edge models, and output a multi-AGV diagnostic set and initial transportation assessment results.

[0047] The topology causality module is used to construct a dynamic transportation topology graph based on the diagnostic set and initial evaluation results, and infer transportation optimization trends through multi-agent reinforcement learning; it also constructs and corrects the transportation bottleneck causality graph by combining historical data, and outputs the corrected bottleneck probability.

[0048] The scheduling verification module is used to adjust scheduling priorities based on the probability of correcting bottlenecks and transportation optimization trends, extract abnormal node features and cross-verify them with visual evidence, and output the final scheduling level, verification data and transportation risk level.

[0049] The model building module is used to construct a formatted training set based on transportation features, interactive data, and simulation parameters, train a multimodal transportation embedding model, output evaluation results, and update model parameters using feedback such as the final scheduling level.

[0050] The scheme generation module is used to combine transportation status assessment results with historical scheduling data to formulate short-, medium-, and long-term full-cycle optimization schemes; it uses interpretable algorithms to locate the root causes of bottlenecks and influencing links, and generates scheduling suggestions, optimization schemes, and risk assessment reports.

[0051] This invention provides a pallet transportation management method and system based on reinforcement learning. By comprehensively collecting and standardizing transportation-related data, it achieves a comprehensive perception of the transportation system's operational status, breaking through the limitations of scattered and ineffective data utilization in traditional management, and providing reliable support for subsequent evaluation and scheduling. Based on multi-dimensional evaluation rules and intelligent models, it achieves accurate initial evaluation and anomaly screening of transportation status, effectively eliminating noise interference, improving the accuracy of status evaluation, and identifying potential anomalies in advance, avoiding production losses caused by passive handling. By constructing a dynamic topology structure and causal relationship model, combined with intelligent learning algorithms, it achieves accurate prediction and root cause location of transportation bottlenecks, solving the problems of traditional methods being unable to predict bottlenecks and having ambiguous root cause analysis, providing a scientific basis for optimized scheduling. By dynamically adjusting scheduling priorities and formulating full-cycle optimization schemes, it achieves adaptive optimization of scheduling strategies, improves the rationality of resource allocation and equipment coordination efficiency, effectively reduces transportation bottlenecks, ensures continuous and stable production operation, ultimately reduces production costs, and improves overall production economic benefits. Attached Figure Description

[0052] The invention will now be further described with reference to the accompanying drawings.

[0053] Figure 1 This is a flowchart illustrating a pallet transportation management method based on reinforcement learning in this invention. Figure 2 This is a schematic diagram of a pallet transportation management system based on reinforcement learning in this invention. Detailed Implementation

[0054] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0055] like Figures 1 to 2 As shown in the figure, the present invention provides a pallet transportation management method and system based on reinforcement learning. The executing entity can be a pallet transportation management method based on reinforcement learning, and the method includes: like Figure 1 As shown in the figure, an embodiment of the present invention provides a pallet transportation management method based on reinforcement learning, comprising: S1: Collect AGV operation data, workstation status data, and pallet transportation interaction data, and preprocess them to obtain transportation characteristic data and interaction correlation data.

[0056] AGV operation data is fundamental data reflecting the operational status of the AGV's hardware and core components, and is the core basis for evaluating the transportation performance of a single AGV. It is primarily acquired through real-time data collection via speed sensors (drive wheels), load sensors (pallet carrying platform), battery voltage sensors, and current sensors (drive circuits) mounted on the AGV itself, with a sampling frequency of 10Hz. The data is transmitted to the edge computing node via the AGV's local wireless communication module. Workstation status data is automatically recorded by the PLC controllers at each workstation on the production line, including information such as workstation processing status, pallet arrival time, pallet departure time, and workstation congestion duration, and is exported in batches to local storage at the minute level. Pallet transportation interaction data is collected through UWB positioning base stations, high-definition cameras, millimeter-wave radar deployed in the production workshop, and the AGV's built-in communication module. The UWB positioning data sampling frequency is 20Hz, and the camera sampling frequency is 5Hz. The data is aggregated by the workshop edge gateway and then transmitted to the analysis node.

[0057] The preprocessing steps include: Outliers caused by sensor malfunctions, missing values ​​generated during data transmission, and duplicate data are removed. For missing values, nearest neighbor interpolation is used to fill in the gaps; for outliers, the 3σ criterion is used to identify and remove them, i.e., if a data point x satisfies |x-μ|>3σ, where μ is the data mean and σ is the standard deviation, it is determined to be an outlier.

[0058] To map raw data of different dimensions to a unified interval and avoid interference from differences in magnitude to subsequent analysis, a min-max normalization method is used, expressed by the formula:

[0059] in, These are the normalized eigenvalues, with values ​​ranging from [0,1]. This refers to the original feature data; This is the minimum value of the feature; This is the maximum value of this feature.

[0060] Core features (such as average AGV speed, battery voltage fluctuation, and load weight deviation) are extracted from the cleaned and standardized AGV operation data to form transportation feature data; interactive collaboration features are extracted from the pallet transportation interaction data to form interactive correlation data.

[0061] S2: Based on transportation characteristic data, interactive correlation data, and production scheduling requirements, conduct an initial assessment of transportation status to obtain a multi-AGV diagnostic set and initial transportation assessment results.

[0062] The multi-AGV diagnostic set contains records of abnormal characteristics for each AGV. The initial transportation assessment results include transportation status level, description of abnormal characteristics, etc.

[0063] The process of conducting an initial assessment of the transport status includes: S21: Construct basic transportation assessment rules based on transportation characteristic data and interactive correlation data.

[0064] The basic transportation assessment rules are a set of rules built based on the standard requirements for AGV operation, environmental adaptability, and interactive collaboration needs, used to judge whether transportation characteristics are normal and whether the correlation between characteristics is reasonable.

[0065] The process of developing basic transportation assessment rules includes: S211: Divide the dimensions of transportation assessment.

[0066] The dimensional division criteria are based on the AGV operating mechanism and production line collaboration requirements, dividing the system into 3 primary dimensions and 8 secondary dimensions. The primary dimensions include AGV performance, interaction and collaboration, and environmental adaptation. The three primary dimensions are then further divided into secondary dimensions, as detailed below: The AGV performance dimension includes three sub-dimensions: driving status, battery performance, and load capacity.

[0067] The interactive collaboration dimension includes three sub-dimensions: relative positioning accuracy, task collaboration synchronization, and information interaction stability.

[0068] The environmental adaptability dimension includes two sub-dimensions: ground flatness adaptability and obstacle avoidance response.

[0069] Each secondary dimension corresponds to several specific transportation characteristics, such as the driving status corresponding to AGV driving speed, acceleration fluctuation, path deviation, and other characteristics.

[0070] S212: Based on the AGV's factory standard parameters, the production line's rated operating parameters, and historical bottleneck-free data, the normal value range of each dimension's characteristics is determined through statistical analysis, forming basic threshold rules.

[0071] The standard parameters for AGVs are obtained from the technical manuals provided by the AGV manufacturers, including the rated operating parameters of each component. The rated operating parameters of the production line are determined based on the production line design specifications and production task requirements. Historical bottleneck-free data is extracted from the production line's historical operation database, filtering transportation data with no bottleneck records within the past 12 months, with a sample size of no less than 1000 sets. Statistical analysis uses a normal distribution statistical model to determine the normal value ranges for each feature. The normal value ranges for each dimension of features are compiled into a basic threshold rule table, including fields such as feature name, dimension, normal range, and reference standard.

[0072] S213: Based on scheduling requirements, set feature association constraint rules and clarify the linkage, matching and response matching thresholds of features between the same or different AGVs.

[0073] Feature association constraint rules are used to regulate the linkage relationship, matching accuracy, and response time between different features of the same AGV and between similar features of different AGVs, ensuring coordinated operation of individual AGVs and collaborative interaction among groups.

[0074] Linkage constraints for the same AGV feature are set by establishing linkage thresholds for associated features within the AGV performance dimension. Matching constraints for different AGV features are set by establishing matching thresholds for features within the interaction and collaboration dimension. Response matching constraints are set by establishing response delay thresholds for information interaction features. Finally, the above constraint relationships and corresponding thresholds are compiled into a feature association constraint rule table.

[0075] S214: Set the environment adaptation coefficient, and modify the basic threshold rules according to the environmental parameters of the production workshop to obtain the threshold rules after environment adaptation.

[0076] The environmental adaptation coefficient is used to quantify the impact of production workshop environmental parameters (such as ground flatness, light intensity, obstacle density, etc.) on the normal value range of AGV transportation characteristics, enabling adaptive adjustment of basic threshold rules. It is primarily achieved by acquiring environmental parameters in real time through environmental sensors deployed in the production workshop, with a sampling frequency of 1Hz, and transmitting the data to edge computing nodes for processing. The calculation method of the environmental adaptation coefficient is expressed as follows:

[0077] in, For environmental adaptability coefficient, For the ground flatness adaptor coefficient, For illumination aptamer coefficients, The obstacle density adapter coefficient, , , The weights of each factor satisfy the following conditions: The values ​​were determined using the analytic hierarchy process and were 0.5, 0.2, and 0.3.

[0078] Ground flatness adaptor coefficient: ,and

[0079] in To monitor ground flatness in real time, This represents the lower limit of optimal flatness. The upper limit of optimal flatness, The flatness influence coefficient is obtained through correlation analysis between historical bottleneck data and ground flatness, and its value ranges from [0.1, 0.3].

[0080] Illumination aptamer coefficient: ,and

[0081] in To detect light intensity in real time, the unit is lux; The lower limit for optimal illumination is set at 500 lux. The optimal upper limit for illumination is set at 2000 lux. The light influence coefficient is 0.2.

[0082] Obstacle density aptamer coefficient: ,and

[0083] in To detect obstacle density in real time, the unit is obstacles per square meter; The upper limit for the density of safety obstacles is set at 0.1 obstacles per square meter. The obstacle density influence coefficient is set to 0.5.

[0084] The threshold after environment adaptation is expressed as:

[0085] in, This represents the normal value range after environmental adaptation. , These are the upper and lower limits of the normal range in the basic threshold rule.

[0086] S215: Based on the threshold rules and feature association constraint rules after environmental adaptation, integrate the threshold rules and feature association constraint rules after environmental adaptation to form the initial basic transportation assessment rules.

[0087] The rules are built by verifying and optimizing historical bottleneck cases until the preset performance requirements are met. Specifically, 500 typical bottleneck cases are extracted from the historical bottleneck database of the production line. The transportation characteristic data before the bottleneck occurred are input into the initial rules to determine whether the rules can accurately identify abnormal features. The accuracy (P), recall (R), and F1 score of the rules are then calculated.

[0088] S22: Compare and match each feature of the transportation feature data and interactive correlation data with the basic transportation assessment rules, and mark abnormal features that exceed the normal value range.

[0089] If a feature does not meet the rule requirements, it is marked as an anomalous feature, and the anomalous type is recorded. The degree of deviation of the anomalous feature is quantified using relative deviation, and the occurrence time is recorded. The calculation method for relative deviation is as follows:

[0090] in, The degree of relative deviation; The actual value of the abnormal feature; This is the mean value of the normal range for this characteristic.

[0091] Feature comparison and matching involves comparing the preprocessed transportation feature data and interactive correlation data feature by feature with the basic transportation assessment rules, including comparing single features with thresholds after environmental adaptation; and comparing related features with feature correlation constraint rules.

[0092] S23: Input the labeled abnormal features into the locally deployed deep learning edge model, analyze the correlation between abnormal features through the model, eliminate mislabeling caused by isolated noise points, output the transportation status level, and generate a preliminary transportation assessment result document.

[0093] The specific steps include: It adopts a lightweight structure of "convolutional layer + LSTM layer + fully connected layer". The input is an abnormal feature sequence (dimension is 1×m, where m is the number of abnormal features) and the output is a feature correlation score, ranging from 0 to 1. A score ≥0.6 is judged as an associated anomaly, and <0.6 is judged as an isolated noise point.

[0094] The model was trained using the cross-entropy loss function, with Adam as the optimizer, a learning rate of 0.001, and 50 training iterations. After training, the model was deployed to an edge computing node.

[0095] The labeled anomalous features are input into the model, and the model outputs the correlation score of each anomalous feature, removing isolated noise points with a score <0.6.

[0096] Based on the number of remaining abnormal features, the degree of deviation, and the scope of correlation, the transportation status is divided into 5 levels. The specific classification of the levels is shown in Table 1.

[0097] Table 1:

[0098] Organize information such as transportation status level, list of valid abnormal features, and abnormality type to generate a preliminary transportation assessment result document in JSON format.

[0099] S3: Based on the multi-AGV diagnostic set and the initial transportation evaluation results, a dynamic transportation topology graph is constructed, and the transportation optimization trend is inferred through multi-agent deep reinforcement learning; the transportation bottleneck causal graph is constructed and corrected by combining historical bottleneck data and historical transportation status data, and the corrected bottleneck probability is obtained.

[0100] The dynamic transportation topology graph is a time-series dynamic graph structure with AGVs and workstations as nodes and the transportation relationship between AGVs and workstations as edges. The edge weights quantify the transportation intensity, frequency, and bottleneck correlation, and can reflect the bottleneck correlation status in multi-AGV collaborative transportation in real time.

[0101] The process of constructing a dynamic transportation topology graph includes: S31: Map AGVs and workstations as graph nodes, and map transportation intensity, transportation frequency, and bottleneck correlation as graph edge weights.

[0102] Each AGV and workstation is considered a node. Node attributes include the node's transportation status level, core anomaly characteristics, and basic parameters (such as AGV model and workstation number), represented as follows:

[0103] in, For node attributes, Let i be the transport status level. Let be the deviation of node i from the j-th anomalous feature. Let be the k-th basic parameter of node i.

[0104] The edge weight is calculated as follows:

[0105] in, For edge weights, Transportation intensity, Transportation frequency, This represents the bottleneck correlation degree.

[0106] Transportation intensity Based on the completion rate of transportation tasks at AGVi and workstation j The calculation is expressed as:

[0107] The value range is [0,1]. The larger the size, the greater the transport intensity. To achieve the minimum task completion rate, To achieve the maximum task completion rate.

[0108] Transportation frequency is the number of times AGVi and workstation j are transported per unit time. Normalization yields the following, expressed as:

[0109] in, The maximum transport frequency is set to 30 times / hour, with a value range of [0,1].

[0110] Bottleneck correlation is calculated based on the similarity of the transportation status levels between node i and node j, and is expressed as:

[0111] in, , The level represents the transportation status, ranging from 1 to 5, with a value range of [0,1]. The closer the levels are, the higher the bottleneck correlation.

[0112] S32: Based on the multi-AGV diagnostic set and the initial transportation evaluation results, update the node attributes and edge weights.

[0113] Specifically, node attributes and edge weights are updated every 5 minutes to ensure that the graph structure reflects the latest transportation and interaction status.

[0114] S33: Use graph construction algorithms to generate dynamic transportation topology graphs.

[0115] Specifically, a dynamic graph construction algorithm based on the adjacency matrix is ​​adopted, including constructing an initial adjacency matrix; updating the adjacency matrix according to the updated edge weights; and generating a dynamic transportation topology graph based on the updated adjacency matrix and node attributes.

[0116] The initial adjacency matrix is ​​constructed by initializing it as an N×N square matrix, where N is the total number of AGVs and workstations in the production workshop, and the diagonal elements are... Off-diagonal elements (i≠j) is assigned the edge weight between node i and node j calculated earlier. To form the initial adjacency matrix .

[0117] The adjacency matrix is ​​updated by recalculating the edge weights between nodes based on the latest collected transportation data, according to the 5-minute update cycle set in S32. The new value is used to update only the off-diagonal elements of the adjacency matrix, resulting in an updated adjacency matrix that matches the current time. .

[0118] The updated adjacency matrix Node attribute matrix updated at the same time (Including the transportation status level, deviation of abnormal features, and basic parameters of each node at time t) as input, a dynamic transportation topology graph is constructed through a graph structure encapsulation algorithm. Where t is the timestamp of the current update cycle, used to mark the time dimension information corresponding to the graph structure, and finally generated It contains four core elements: “node set, edge set, edge weight, and timestamp”, which are used as feature inputs for multi-agent deep reinforcement learning networks.

[0119] The process of obtaining the corrected bottleneck probability includes: S34: Input the features of the dynamic transportation topology into the multi-agent deep reinforcement learning network and output the initial bottleneck probability.

[0120] Multi-agent deep reinforcement learning networks are neural network models that integrate temporal convolutional networks and graph convolutional networks. They can simultaneously capture temporal dynamic features and spatial correlation features, and are used to infer the optimization trend and initial bottleneck probability of transportation systems.

[0121] The process of outputting the initial bottleneck probability includes: S341: Perform dimension alignment and normalization on the node and edge features of the dynamic transportation topology graph to obtain standardized graph features.

[0122] Dimension alignment employs zero-padding or feature filtering to ensure consistent feature dimensions across all nodes and edges. Node features are normalized using min-max normalization, while edge features are normalized using Z-score normalization.

[0123] S342: Divide the standardized graph features into continuous feature sequences according to the time series, as follows:

[0124] in, It is a continuous characteristic sequence. For standardized graph features, The time window length is set to 12, which is 60 minutes of historical data, and t is the current time.

[0125] The temporal convolutional layers of a multi-agent deep reinforcement learning network are input to extract the temporal dependencies of the feature sequences. The temporal convolutional network uses 1D convolutional kernels and ReLU activation function, extracting dependencies at different time scales through multi-scale dilated convolutions, and outputting a temporal feature sequence. , represented as ,in, For time-series feature dimensions, For the set of real numbers, The time window length is N, and the total number of AGVs and workstations in the production workshop is N.

[0126] S343: Input the output of the temporal convolutional layer into the graph convolutional layer to capture the spatial relationship features between nodes. The formula is as follows:

[0127] in, , , for The degree matrix, The spatial correlation features at time t are represented by the output of the graph convolutional layer, used to characterize the spatial relationships between nodes at that time. , For spatial feature dimensions. Represented as the normalized adjacency matrix, The graph convolution weight matrix; Let be the bias term of the graph convolutional layer, D be the degree matrix, and be a diagonal matrix where the diagonal elements are the sum of the edge weights of the corresponding nodes. Let A be the original adjacency matrix, where each element represents the combined edge weight between nodes. I is the identity matrix, used to preserve the characteristics of the nodes themselves. This is the ReLU activation function.

[0128] By concatenating temporal and spatial features, a feature vector integrating spatiotemporal information is obtained. .

[0129] S344: The feature vectors fused with spatiotemporal information are input into the fully connected layer. The initial bottleneck probability of each transportation node is calculated using the sigmoid activation function, and the initial bottleneck probability is output. The formula is as follows:

[0130] in, This represents the initial bottleneck probability for a single node. This is the weight matrix of the fully connected layer; For bias terms, sigmoid activation function 'a' represents the linear output of the fully connected layer. The final output initial bottleneck probability is represented by a set. .

[0131] S35: Construct and correct the cause-and-effect graph of transportation bottlenecks by combining historical bottleneck data and historical transportation status data.

[0132] The transportation bottleneck causal graph is a directed graph built on a Bayesian network. Nodes represent transportation characteristics and bottleneck types, edges represent the causal relationships between them, and edge weights represent the strength of the causal relationship. It is used to quantify the impact of abnormal transportation characteristics on bottleneck occurrence. Historical bottleneck data comes from a production line bottleneck record database, containing 800 bottleneck cases from the past two years, including bottleneck type, transportation characteristic data before the bottleneck occurred, bottleneck occurrence time, environmental parameters, etc.; historical transportation status data consists of transportation status records from the past two years, with a sample size of 5000 sets.

[0133] The process of revising the cause-and-effect graph of transportation bottlenecks includes: S351: Based on historical bottleneck data and historical transportation status data, an initial transportation bottleneck causal graph is constructed using a Bayesian network, where nodes represent transportation characteristics and bottleneck types, edges represent the causal relationships between the two, and edge weights represent the initial causal relationship strength.

[0134] The steps include: The node set is determined, which includes 20 transportation characteristic nodes and 10 bottleneck type nodes; Based on domain knowledge and historical data, determine the initial causal relationships between nodes; The initial causal relationship strength (edge ​​weight) is calculated using the conditional probability P(B|A), where A is the cause node and B is the result node, i.e., the probability that B occurs when A occurs. This is obtained through historical data statistics, and the formula is as follows:

[0135] in, This represents the number of cases where A and B occur simultaneously. Let A be the number of cases that occurred.

[0136] S352: Based on bottleneck case data and corresponding transportation status evolution data, calculate the confidence level of each causal relationship, and update the initial causal relationship strength. That is, select the most recent 500 cases from the above 800 historical bottleneck cases as the latest case data, and calculate the confidence level of each causal relationship. The formula is expressed as:

[0137] in, This refers to the number of times A and B occur simultaneously in the latest case; The number of times A occurred in the latest case; The weighting factor is set to 0.6 in this embodiment to balance the influence of new data and historical data. The larger the value, the more significant the impact of the most recent case on the confidence level.

[0138] If a new relationship exists between a transportation feature and a bottleneck type, the corresponding node and edge are added, their confidence level is calculated as the edge weight, and they are added to the transportation bottleneck causal graph to obtain an updated transportation bottleneck causal graph.

[0139] S353: Verify the updated causal diagram of transportation bottlenecks and eliminate unreasonable causal relationships. Specifically, this is done by combining expert review and data consistency testing. The expert review is conducted by three experts in the field of production logistics, who score the reasonableness of the causal relationships (0-10 points, ≥6 points is reasonable). The data consistency test calculates the support of the relationship (a frequency of occurrence ≥5 times is considered valid) and eliminates unreasonable causal relationships with an expert score <6 points or a support <5 times.

[0140] Adjust the strength of abnormal causal relationships. For retained causal relationships, if their confidence level exceeds the normal range (0.1~0.9), adjust them to a reasonable range by combining domain knowledge to obtain the corrected causal graph of transportation bottlenecks.

[0141] S354: Calculate the goodness-of-fit index of the corrected cause-effect graph of the transportation bottleneck. If the goodness-of-fit index reaches the preset threshold, the correction is complete. If it does not reach the threshold, repeat steps S352-S353 until the goodness-of-fit index meets the requirements.

[0142] The Akaike Information Criterion is used as the goodness-of-fit index to evaluate the fitting effect of the modified causal graph on historical bottleneck data, as shown in the following figure:

[0143] Here, AIC is the goodness-of-fit index, k is the number of parameters in the causal graph (i.e., the number of edges), and L is the likelihood function value of the causal graph, calculated based on historical bottleneck data. The smaller the AIC value, the better the fit. The preset AIC threshold is 100. If the AIC value of the corrected causal graph is ≤100, the correction is complete; otherwise, repeat steps S352-S353 until the AIC value meets the requirement.

[0144] S36: The causal correlation strength of the transportation bottleneck cause-effect graph is incorporated as a weight into the initial bottleneck probability calculation to obtain the corrected bottleneck probability, expressed by the formula:

[0145] in, To correct for the bottleneck probability, M is the number of causal relationships associated with this transportation node; Let be the strength weight of the s-th causal relationship, satisfying ; This represents the initial bottleneck probability for a single node. The bottleneck probability is derived from the causal graph of the transportation bottleneck. Specifically, it is obtained by tracing the corresponding "bottleneck node" in the corrected causal graph of the transportation bottleneck through the "abnormal transportation feature node" of the transportation node, and then deriving the probability of bottleneck occurrence, which reflects the direct causal impact of abnormal transportation features on the bottleneck.

[0146] S4: Adjust scheduling priorities based on the corrected bottleneck probability and transportation optimization trend, verify bottleneck correlation, collect verification data as needed, and obtain the final scheduling level, verification data, and transportation risk level.

[0147] The process of adjusting scheduling priorities and verifying bottleneck correlations includes: S41: Preset three-level scheduling thresholds, match and correct bottleneck probabilities to obtain the initial scheduling level, and the specific correspondence is shown in Table 2.

[0148] Table 2:

[0149] S42: Calculate the collaborative confidence level by combining the number of abnormal associations in the multi-AGV diagnostic set and the rate of change of the transportation optimization trend.

[0150] Collaborative confidence score is an indicator that quantifies the reliability and severity of anomalies by combining the number of associated anomalies with the rate of change of transportation optimization trends. The specific calculation method is as follows:

[0151] in, The collaborative confidence level has a value range of [0,1]. To represent the maximum possible number of abnormal associations, this embodiment sets it to 10; This represents the number of abnormal associations. The normalized rate of change of the transportation optimization trend is dimensionless and calculated as follows:

[0152] in, The rate of change of the original transportation optimization trend is defined as the decrease in the transportation status level per unit time. The maximum possible rate of change is set to 4 levels per hour (corresponding to the extreme rate at which the transport status decreases from level 5 (extremely severe anomaly) to level 1 (smooth) after normalization. The value range is [0,1].

[0153] S43: Adjust the initial scheduling level based on the collaborative confidence level, and output the adjusted scheduling level and the corresponding abnormal transportation range that needs to be checked.

[0154] The scheduling level adjustment strategy is as follows: If The dispatch level is upgraded by 1 level; if The scheduling level remains unchanged; if The scheduling level is reduced by 1 level. After adjustment, the final scheduling level and the scope of abnormal transportation that need to be checked are output, namely transportation nodes with a collaboration confidence level ≥ 0.5.

[0155] S44: Based on the abnormal transportation range that needs to be checked, extract the operation-load characteristics of the corresponding abnormal transportation nodes in the multi-AGV diagnostic set, compare them with the preset normal feature library, and obtain the operation-load characteristic matching degree and the corresponding abnormal feature information.

[0156] The operation-load characteristic is a combination of operation signals and load signals generated by components such as the drive motor and the pallet-bearing platform during AGV operation. It is an important basis for determining whether there is a bottleneck in transportation. Operation data is collected by the speed sensor mounted on the AGV body, with a sampling frequency of 10Hz. Load data is collected by the weight sensor mounted on the pallet-bearing platform, with a sampling frequency of 1Hz. A preset normal feature library contains 100 sets of operation-load characteristic samples for smooth transportation, which are constructed and stored in the edge node. The matching method of operation-load characteristic matching degree is expressed as follows:

[0157] in, To obtain the run-load feature vector, Fast Fourier Transform is used to extract run features (12 dimensions), and statistical analysis is used to extract load features (16 dimensions). The two vectors are then concatenated to obtain the run-load feature vector. . These are the sample feature vectors from the normal feature library. It is an L2 norm. If If so, it is determined to be an abnormal operation-load characteristic, and the corresponding abnormal characteristic information is recorded.

[0158] S45: Based on the abnormal feature information corresponding to the running-load feature matching degree, collect visual evidence such as the driving trajectory and stopping time of abnormal transportation nodes.

[0159] Visual data is collected by high-definition cameras (30fps) deployed in the production workshop to collect data on the AGV's driving trajectory and stopping time.

[0160] S46: Analyze the matching degree of operation-load features and the feature synchronization and correlation of visual evidence to verify the effectiveness of bottleneck correlation.

[0161] Feature synchronicity refers to the degree of synchronization between abnormal operation-load characteristics and visual anomaly characteristics in the time dimension. Synchronicity analysis is performed by calculating the time difference between the occurrence of abnormal operation-load characteristics and the occurrence of visual anomaly characteristics. ,like If so, it is determined to be time synchronization. Correlation analysis involves calculating the correlation coefficient r between the two anomalous features. If... If both time synchronization and feature association are satisfied, the bottleneck association is verified as valid; otherwise, it is determined as invalid association, and the scheduling level is readjusted.

[0162] S5: Based on transportation feature data, interactive correlation data, and simulation parameters, construct a formatted training dataset, build a multimodal transportation embedding model, output transportation status assessment results, and update the multimodal transportation embedding model using the final scheduling level, validation data, and transportation risk level.

[0163] The process of constructing a multimodal transportation embedding model includes: S51: Construct a multimodal fusion network, inputting transportation feature data, interaction correlation data, and simulation parameters as training samples to build a formatted training dataset.

[0164] To address the heterogeneity of data across different modalities, preprocessing and modal alignment are performed to ensure data consistency. The specific steps are as follows: Outliers were removed from the simulation parameter data, and missing values ​​were filled in. The transportation features and interactive correlation features data were verified a second time to ensure that the data format was consistent with the data format after S1 preprocessing.

[0165] Based on the sampling timestamp of transportation feature data, timestamp matching is performed on interactive correlation data and simulation parameters, and a resampling method is used to unify the sampling frequency to 10Hz. Through feature filtering and zero padding, the dimensions of transportation features are fixed to 15, the dimensions of interactive correlation features are fixed to 12, and the dimensions of simulation parameters are fixed to 8.

[0166] The transportation features and interaction correlation features follow the min-max normalization of S1; the simulation parameters are normalized using Z-score to ensure that the range of values ​​for each modality is uniformly [0,1] or a standard normal distribution, so as to avoid the impact of magnitude differences on model training.

[0167] S52: Using transportation status levels as labels, train the network on the formatted training dataset using the cross-entropy loss function to obtain the network parameters to be optimized.

[0168] The transportation feature extraction branch adopts a "convolutional layer + pooling layer" structure to accommodate the local temporal correlation of transportation features. The specific structure consists of an input layer, a 1D convolutional layer 1, a max-pooling layer 1, a 1D convolutional layer 2, a global average pooling layer, and an output transportation feature vector. .

[0169] The interaction-related feature extraction branch employs a graph convolutional layer structure to capture the spatial correlation characteristics of multi-AGV interactions. The specific structure consists of an input layer, graph convolutional layer 1, graph convolutional layer 2, node feature aggregation, and an output interaction feature vector. .

[0170] The simulation parameter extraction branch adopts a fully connected layer structure to adapt to the high-dimensional numerical characteristics of simulation parameters. The specific structure consists of an input layer, fully connected layer 1, a Dropout layer, fully connected layer 2, and an output simulation feature vector. .

[0171] S53: Iteratively optimize the network parameters to obtain a multimodal transportation embedding model that outputs a unified dimension transportation feature embedding vector.

[0172] The fused feature vector is represented as:

[0173] in, , , The attention weights are respectively for transportation features, interaction-related features, and simulation parameters, satisfying... The values ​​were obtained through adaptive learning by training the model, with the initial values ​​all set to 1 / 3. , , These are transportation features, interaction-related features, and simulation parameters, each with a 32-dimensional dimension. Therefore, the feature vector is fused. The dimension is 32.

[0174] The transportation feature embedding vector is represented as:

[0175] in, Embedded vectors for 32-dimensional transportation features; This is the embedding layer weight matrix, with a dimension of 32×32; This is the embedding layer bias term, with a dimension of 1×32; The sigmoid activation function maps the values ​​of the embedding vector to the interval [0,1].

[0176] S6: Combining transportation status assessment results with historical scheduling data, formulate short-term, medium-term, and long-term full-cycle transportation optimization plans, and use interpretable algorithms to obtain the root causes of bottlenecks and the links affecting transportation, generating scheduling suggestions, transportation optimization plans, and risk assessment reports to realize the practical application of bottleneck prediction results.

[0177] For interpretable algorithms, the "Bayesian interpretability analysis + SHAP value fusion algorithm" is selected. Bayesian interpretability analysis is adapted to the probabilistic reasoning characteristics of the causal graph of transportation bottlenecks, and SHAP value can quantify the contribution of each node to the bottleneck result. The combination of the two achieves dual analysis of "causal logic + contribution quantification".

[0178] The process of using interpretable algorithms to obtain the root causes of bottlenecks and the links affecting transportation includes: S61: Use an interpretable algorithm to analyze the cause-effect graph of transportation bottlenecks and filter out key nodes with high contribution to the bottleneck and node association information.

[0179] Key bottleneck nodes with high contribution refer to transportation characteristic nodes or intermediate bottleneck nodes whose contribution to the probability of bottleneck occurrence meets a preset threshold. They are core candidate nodes for the root cause of the bottleneck. Node association information includes quantitative information such as causal direction, association strength, and propagation delay between nodes.

[0180] The steps for identifying key bottleneck nodes with high contribution include: Initialize the Bayesian interpretability analysis module and the SHAP value calculation module, and set the core parameters. Contribution screening threshold. Through verification using historical bottleneck cases, this threshold can balance screening precision and recall; the SHAP value is calculated using a sample size of 1000 groups, taken from historical bottleneck data that match the current bottleneck type; the confidence level of Bayesian inference is 0.95.

[0181] The revised causal graph of the transportation bottleneck is input into the Bayesian interpretability analysis module. Using the bottleneck type nodes determined in S4 as the target output nodes, the posterior probabilities of each predecessor node (causal node) are inferred, and the causal effect values ​​between nodes are calculated. These causal effect values ​​reflect the degree of influence of the causal nodes on the bottleneck nodes.

[0182] The node characteristics and corresponding bottleneck probabilities of the transportation bottleneck causal graph are input into the SHAP value calculation module to obtain the SHAP value of each node, where a positive value indicates that the node increases the bottleneck probability and a negative value indicates that the bottleneck probability is suppressed. The absolute value of the SHAP value is taken as the contribution metric. The comprehensive contribution is calculated by integrating the Bayesian causal effect value and the SHAP value, and the calculation method is expressed as follows:

[0183] in, The absolute value of the SHAP value, normalized to [0,1]; is the Bayesian causal effect value, normalized to [0,1]; 0.6 and 0.4 are... and The weighting coefficients are determined through expert review, with priority given to the accuracy of the quantitative assessment of contribution.

[0184] The key node identification method is as follows: nodes with a comprehensive contribution greater than or equal to a preset screening threshold are selected as high-contribution key nodes for the bottleneck. Their core function is to filter out nodes with weak impact on the bottleneck, focusing on key influencing factors and providing a precise set of nodes for subsequent causal path tracing and root cause link construction. The preset screening threshold is set to 0.2, and historical bottleneck cases are used for verification to balance screening precision and recall. The association information for each key node includes the associated node ID and causal direction, association strength, propagation delay, and matched verification data features. Association strength corresponds to the confidence level in the transportation bottleneck causal graph. Propagation delay is the average time from the anomaly of the cause node to the anomaly of the result node, obtained from historical data statistics. Matched verification data features include, for example, "AGV speed anomaly" corresponding to "0-5Hz operating amplitude anomaly" in the operation-load features.

[0185] S62: Combine the final scheduling level, verification data, key nodes and node association information to trace the causal relationship path between each key node.

[0186] Using the final bottleneck node as the endpoint, bidirectional tracing is conducted based on the node association information of the transportation bottleneck causal graph. Root cause localization starts from the bottleneck node and traces the preceding key nodes along the result-to-cause direction, recording the association strength and transmission delay of each node to form multiple potential root cause paths. Impact range prediction starts from the initial cause node obtained from reverse tracing and traces the subsequent impact nodes along the cause-to-effect direction, recording the transmission relationship between nodes, laying the foundation for the construction of subsequent transportation impact links.

[0187] The completeness of the potential root cause paths traced is verified. If there are logical gaps, such as the "battery voltage fluctuation → AGV stagnation" missing the intermediate node "AGV speed abnormality", the missing intermediate key nodes are supplemented by combining domain knowledge and historical bottleneck cases to ensure the causal logic of the path is coherent.

[0188] S63: The causal relationship paths are sorted according to their degree of influence to form the bottleneck root cause link, and the scope and transmission path of the transportation impact are deduced to obtain the transportation impact link.

[0189] Bottleneck root cause path ranking prioritizes the completed potential root cause paths using three ranking indicators: path contribution, path confidence, and path transmission efficiency. The weights of these indicators are determined using the analytic hierarchy process (AHP) combined with historical data validation. In this embodiment, the weights are set as follows: path contribution 0.5, path confidence 0.3, and path transmission efficiency 0.2. The comprehensive ranking score for each path is calculated using the following formula:

[0190] in, The path contribution is dimensionless and ranges from [0,1]. The path confidence score is dimensionless and ranges from [0,1]. The normalized path propagation efficiency is dimensionless and is calculated as follows:

[0191] in, The total path propagation delay is expressed in seconds. The minimum total propagation delay for all historical bottleneck paths is obtained through statistical analysis of historical data and then normalized. The value range is (0,1], and the smaller the conduction delay, the better. The larger the value, the higher the path transmission efficiency.

[0192] The top three paths with the highest comprehensive ranking scores are selected as the core bottleneck root cause links; if there are fewer than three, all are retained. Paths are then sorted from highest to lowest score to identify the primary and secondary root cause links. Based on the root cause localization results, combined with the multi-AGV dynamic transportation topology map, the transportation impact range is defined, including the individual impact range and the group impact range. The individual impact range refers to components within the bottleneck AGV affected by the root cause node, such as an AGV motor failure affecting the drive and control systems. The group impact range refers to other AGVs affected by the interaction with the bottleneck AGV; for example, in collaborative transportation, the stagnation of the bottleneck AGV affects the transportation efficiency of adjacent AGVs. AGVs with a collaborative confidence level ≥ 0.5 are identified as the core impact objects.

[0193] The transportation impact link construction integrates the scope of transportation impact with the forward tracing transmission path. Following the logic of "initial root cause node → intermediate impact node → terminal impact node", the transportation impact link is constructed, and the impact degree and transmission path type of each node are marked.

[0194] Based on the priority and key node characteristics of bottleneck root cause links, hierarchical scheduling recommendations are generated, including emergency scheduling recommendations, routine scheduling recommendations, and preventative scheduling recommendations. Emergency scheduling recommendations primarily target the initial root cause node of the primary root cause link, clarifying emergency handling measures, operational procedures, required resources, and safety precautions. Routine scheduling recommendations primarily target key nodes of secondary root cause links, developing a 24-hour scheduling plan, clarifying scheduling steps, route adjustment lists, and acceptance criteria. Preventative scheduling recommendations primarily target intermediate nodes in the transportation-affected link, proposing regular inspection and maintenance measures to prevent bottlenecks from escalating.

[0195] By combining the transportation status assessment results of the multimodal transportation embedding model with historical scheduling data, short-term, medium-term, and long-term full-cycle transportation optimization schemes are formulated. Short-term optimization focuses on key nodes corresponding to current anomalies, develops high-frequency inspection plans, sets inspection indicators and anomaly thresholds, and clarifies data recording and reporting processes. Medium-term optimization expands the optimization scope to all nodes covered by the transportation impact link, conducts in-depth detection, and formulates transportation route optimization plans. Long-term optimization, based on AGV service life, hardware aging patterns, and trend prediction results from the multimodal transportation embedding model, develops AGV update plans and system upgrade plans.

[0196] In summary, this embodiment provides a pallet transportation management method based on reinforcement learning. By comprehensively collecting and standardizing transportation-related data, it achieves a comprehensive understanding of the transportation system's operational status, breaking through the limitations of scattered and ineffective data utilization in traditional management, and providing reliable support for subsequent evaluation and scheduling. Based on multi-dimensional evaluation rules and intelligent models, it achieves accurate initial evaluation and anomaly screening of transportation status, effectively eliminating noise interference, improving the accuracy of status evaluation, and identifying potential anomalies in advance, avoiding production losses caused by passive handling. By constructing a dynamic topology structure and causal relationship model, combined with intelligent learning algorithms, it achieves accurate prediction and root cause localization of transportation bottlenecks, solving the problems of traditional methods' inability to predict bottlenecks and fuzzy root cause analysis, providing a scientific basis for optimized scheduling. By dynamically adjusting scheduling priorities and formulating full-cycle optimization schemes, it achieves adaptive optimization of scheduling strategies, improving the rationality of resource allocation and equipment coordination efficiency, effectively reducing transportation bottlenecks, ensuring continuous and stable production operation, ultimately reducing production costs and improving overall production economic benefits.

[0197] Based on the same general inventive concept, this invention also protects a pallet transportation management system based on reinforcement learning. The following describes a pallet transportation management system based on reinforcement learning provided by this invention. The pallet transportation management system based on reinforcement learning described below can be referred to in correspondence with the pallet transportation management method and system based on reinforcement learning described above.

[0198] A pallet transportation management system based on reinforcement learning includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor. The processor includes a data preprocessing module, a state initial assessment module, a topology causality module, a scheduling verification module, a model building module, and a scheme generation module.

[0199] The data preprocessing module is used to collect AGV operation, workstation status and pallet interaction data. Through preprocessing such as noise reduction, missing value filling and normalization, it outputs transportation feature data and interaction correlation data.

[0200] The initial status assessment module is used to construct assessment rules based on transportation characteristics, interactive data, and scheduling requirements, compare abnormal features, eliminate noise through edge models, and output a multi-AGV diagnostic set and initial transportation assessment results.

[0201] The topology causality module is used to construct a dynamic transportation topology graph based on the diagnostic set and initial evaluation results, and infer transportation optimization trends through multi-agent reinforcement learning; it also constructs and corrects the transportation bottleneck causality graph by combining historical data, and outputs the corrected bottleneck probability.

[0202] The scheduling verification module is used to adjust scheduling priorities based on the probability of correcting bottlenecks and transportation optimization trends, extract abnormal node features and cross-verify them with visual evidence, and output the final scheduling level, verification data and transportation risk level.

[0203] The model building module is used to construct a formatted training set based on transportation features, interactive data, and simulation parameters, train a multimodal transportation embedding model, output evaluation results, and update model parameters using feedback such as the final scheduling level.

[0204] The solution generation module is used to combine transportation status assessment results with historical scheduling data to formulate short-, medium-, and long-term full-cycle optimization solutions; it uses interpretable algorithms to locate the root causes of bottlenecks and influencing links, and generates scheduling suggestions, optimization solutions, and risk assessment reports.

[0205] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0206] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A pallet transportation management method based on reinforcement learning, characterized in that, include: Collect AGV operation data, workstation status data, and pallet transportation interaction data, and preprocess them to obtain transportation characteristic data and interaction correlation data; Based on the transportation feature data, the interactive correlation data, and production scheduling requirements, a preliminary assessment of the transportation status is conducted to obtain a multi-AGV diagnostic set and the preliminary transportation assessment results. Based on the multi-AGV diagnostic set and the initial transportation evaluation results, a dynamic transportation topology map is constructed, and the transportation optimization trend is inferred through multi-agent deep reinforcement learning. By combining historical bottleneck data and historical transportation status data, a cause-and-effect graph of transportation bottlenecks is constructed and corrected to obtain the corrected bottleneck probability. Adjust scheduling priorities based on the corrected bottleneck probability and the transportation optimization trend, verify bottleneck correlation, collect verification data as needed, and obtain the final scheduling level, verification data, and transportation risk level. Based on the transportation feature data, interaction correlation data, and simulation parameters, a formatted training dataset is constructed, a multimodal transportation embedding model is built, transportation status assessment results are output, and the multimodal transportation embedding model is updated using the final scheduling level, validation data, and transportation risk level. Based on the transportation status assessment results and historical scheduling data, short-term, medium-term, and long-term full-cycle transportation optimization schemes are formulated. Interpretable algorithms are used to obtain the root causes of bottlenecks and the links affecting transportation, generating scheduling suggestions, transportation optimization schemes, and risk assessment reports.

2. The pallet transportation management method based on reinforcement learning according to claim 1, characterized in that, The process of conducting an initial assessment of the transport status includes: Based on the transportation characteristic data and the interactive correlation data, basic transportation assessment rules are constructed; The transportation feature data and the interactive correlation data are compared and matched with the basic transportation evaluation rules feature by feature, and abnormal features that exceed the normal value range are marked, and the degree of deviation and occurrence time of each abnormal feature are recorded. The labeled abnormal features are input into a locally deployed deep learning edge model. The model analyzes the correlation between abnormal features, eliminates mislabeling caused by isolated noise points, outputs the transportation status level, and generates a preliminary transportation assessment result document.

3. The pallet transportation management method based on reinforcement learning according to claim 2, characterized in that, The process of developing basic transportation assessment rules includes: The transportation assessment dimensions are divided. Based on the AGV's factory standard parameters, the production line's rated operating parameters, and historical bottleneck-free data, the normal value range of each dimension's characteristics is determined through statistical analysis, forming basic threshold rules. Based on the aforementioned scheduling requirements, feature association constraint rules are set to clarify the linkage, matching, and response matching thresholds of features between the same or different AGVs. Set an environment adaptation coefficient, and modify the basic threshold rules according to the environmental parameters of the production workshop to obtain the environment-adapted threshold rules; Based on the threshold rules adapted to the environment and the feature association constraint rules, an initial basic transportation assessment rule is formed. Optimize and validate through historical bottleneck cases until the preset performance requirements are met, and complete the rule construction.

4. The pallet transportation management method based on reinforcement learning according to claim 1, characterized in that, The process of constructing a dynamic transportation topology includes: Map AGVs and workstations as graph nodes, and map transportation intensity, transportation frequency, and bottleneck correlation as graph edge weights; Based on the multi-AGV diagnostic set and the initial transportation assessment results, update the attribute features of the graph nodes and the weights of the graph edges; The dynamic transportation topology graph is generated using a graph construction algorithm.

5. The pallet transportation management method based on reinforcement learning according to claim 1, characterized in that, The process of obtaining the corrected bottleneck probability includes: The features of the dynamic transportation topology are input into the multi-agent deep reinforcement learning network, and the initial bottleneck probability is output. The transportation bottleneck cause-effect graph is constructed and corrected by combining historical bottleneck data and historical transportation status data. The causal correlation strength of the transportation bottleneck causal graph is used as a weight to incorporate into the initial bottleneck probability calculation to obtain the corrected bottleneck probability.

6. The pallet transportation management method based on reinforcement learning according to claim 5, characterized in that, The process of outputting the initial bottleneck probability includes: The node and edge features of the dynamic transportation topology graph are dimensionally aligned and normalized to obtain standardized graph features. The standardized graph features are divided into continuous feature sequences according to the time series, and input into the temporal convolutional layer of the multi-agent deep reinforcement learning network to extract the temporal dependencies of the feature sequences. The output of the temporal convolutional layer is input into the graph convolutional layer to capture the spatial correlation features between nodes and obtain a feature vector that integrates spatiotemporal information. The feature vectors of the fused spatiotemporal information are input into the fully connected layer, and the initial bottleneck probability of each transportation node is calculated through the activation function, and the initial bottleneck probability is output.

7. The pallet transportation management method based on reinforcement learning according to claim 5, characterized in that, The process of correcting the cause-effect graph of the transportation bottleneck includes: Based on historical bottleneck data and historical transportation status data, a Bayesian network is used to construct an initial transportation bottleneck causal graph, where nodes represent transportation characteristics and bottleneck types, edges represent the causal relationships between the two, and edge weights represent the initial causal relationship strength. Based on bottleneck case data and corresponding transportation status evolution data, the confidence level of each causal relationship is calculated, and the initial causal relationship strength is updated. If a new relationship exists between transportation features and bottleneck types, it is added to the transportation bottleneck causal graph to obtain an updated transportation bottleneck causal graph. The updated transportation bottleneck cause-effect graph is verified, unreasonable causal relationships are removed, and the strength of abnormal causal relationships is adjusted to obtain the corrected transportation bottleneck cause-effect graph. Calculate the goodness-of-fit index of the corrected transportation bottleneck cause-effect graph. If the goodness-of-fit index reaches a preset threshold, the correction is completed. If the requirement is not met, repeat the above process until the fit index meets the requirements.

8. The pallet transportation management method based on reinforcement learning according to claim 1, characterized in that, The process of adjusting scheduling priorities and verifying bottleneck correlations includes: A three-level scheduling threshold is preset, and the initial scheduling level is obtained by matching the corrected bottleneck probability. The collaborative confidence level is calculated by combining the number of abnormal associations in the multi-AGV diagnostic set and the rate of change of the transportation optimization trend. Adjust the initial scheduling level based on the collaborative confidence level, and output the adjusted scheduling level and the corresponding abnormal transportation range that needs to be checked in detail. Based on the abnormal transportation range that needs to be checked, the operation-load characteristics of the corresponding abnormal transportation nodes in the multi-AGV diagnostic set are extracted and compared with the preset normal feature library to obtain the operation-load characteristic matching degree and the corresponding abnormal feature information. Based on the abnormal feature information corresponding to the operation-load feature matching degree, visual evidence of abnormal transportation nodes, including driving trajectory and stopping time, is collected. The effectiveness of bottleneck association is verified by analyzing the feature synchronicity and correlation between the operation-load feature matching degree and the visual evidence.

9. The pallet transportation management method based on reinforcement learning according to claim 2, characterized in that, The process of constructing a multimodal transportation embedding model includes: Construct a multimodal fusion network, inputting the transportation feature data, the interaction correlation data, and simulation parameters as training samples to build a formatted training dataset; Using the transportation status level as a label, the network is trained on the formatted training dataset using the cross-entropy loss function to obtain the network parameters to be optimized; The network parameters to be optimized are iteratively optimized to obtain the multimodal transportation embedding model that outputs transportation status assessment results.

10. A pallet transport management system based on reinforcement learning, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements a pallet transportation management method based on reinforcement learning as described in any one of claims 1 to 9.

Images