An artificial intelligence-based production line scheduling optimization method and medium

Abstract

The application discloses a production line scheduling optimization method and medium based on artificial intelligence: according to a scheduling behavior sequence fragment set, local Bellman residuals and global Bellman residuals of each sequence fragment are calculated respectively; according to the local Bellman residuals and the global Bellman residuals, neighborhood coupling disturbance residuals of each sequence fragment are calculated; according to the local Bellman residuals and the neighborhood coupling disturbance residuals, priority weights of each sequence fragment are calculated; according to each sequence fragment and the corresponding priority weight, a scheduling behavior-effect mapping sample set is constructed; according to the scheduling behavior-effect mapping sample set, each sequence fragment is weighted sampled according to the priority weight; according to a weighted training sample batch, a current strategy model is updated to obtain an updated decision strategy model; and according to the updated decision strategy model, process tasks are cooperatively scheduled. The application overcomes the training signal deficiency and strategy update direction deviation caused by only relying on local residuals.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a production line scheduling optimization method and medium based on artificial intelligence. Background Technology

[0002] With the deepening advancement of smart industry, automated production lines have become a core component of modern manufacturing. The level of intelligence in their process scheduling directly affects production efficiency, resource utilization, and product quality. Traditional automated production line scheduling relies heavily on preset fixed logic or centralized control systems, such as sequential control based on PLCs (Programmable Logic Controllers) and static scheduling algorithms based on mathematical programming (e.g., genetic algorithms, simulated annealing algorithms). Centralized control systems typically handle the allocation of all process tasks and resource coordination through a single decision center. While structurally easy to achieve unified management, they often suffer from problems such as delayed decision response, concentrated computational load, and insufficient system robustness when facing complex and ever-changing production environments (e.g., equipment failures, order changes, material shortages, and other dynamic disturbances). In recent years, distributed artificial intelligence technology (especially multi-agent systems) has received widespread attention in the field of industrial scheduling. Multi-agent systems effectively reduce the pressure of centralized decision-making and improve the system's flexibility and scalability by decomposing complex tasks into multiple sub-tasks, which are then collaboratively completed by intelligent agent nodes with autonomous decision-making capabilities.

[0003] In typical multi-process collaborative scheduling scenarios such as non-standard customized intelligent spraying production lines, the production line includes multiple intelligent agent nodes such as loading, spraying, and unloading nodes. Each node is executed by a robot and vision unit to perform scheduling actions such as workpiece gripping, spraying, and pallet transfer, and the material flow between processes is realized through an AGV logistics system. In the above scenario, the process state transition of each intelligent agent node is not only determined by the scheduling actions of the node itself, but is also coupled with the intervention of the concurrent scheduling behavior of neighboring nodes in real time: when the loading node delays the delivery of workpieces due to the extended time for recognizing irregularly shaped workpieces, the spraying node will be in a waiting-for-materials-idle state; when the unloading node experiences blockage feedback due to a full pallet or AGV scheduling delay, the output buffer of the spraying node will continue to accumulate, forcing it to slow down or suspend operation. However, existing multi-agent scheduling technologies, when constructing training samples for decision-making strategies, only calculate the Bellman residual based on the state-action sequence of the current node and use this to determine the training priority of each sequence segment. This approach can only reflect the learning error of the current node itself and cannot measure the coupling disturbance introduced by the concurrent scheduling behavior of the neighboring region, such as upstream waiting for materials and downstream blockage, on the value estimation of the current node. As a result, the sequence segments generated during scheduling periods with strong upstream and downstream coupling intervention are underestimated in training priority and fail to obtain training weights that match their actual learning value. In such high-coupling disturbance process scenarios, the decision-making strategy model is in a state of insufficient training signal for a long time, and the strategy update direction continues to deviate from the optimal collaborative scheduling goal, ultimately affecting core production line indicators such as single-coat spraying time, product qualification rate, and fault-free operation time. Summary of the Invention

[0004] One aspect of this application provides an artificial intelligence-based production line scheduling optimization method, comprising:

[0005] Obtain the set of scheduling behavior sequence fragments of each agent node, as well as the current decision-making strategy model of each agent node;

[0006] Based on the set of scheduling behavior sequence segments, calculate the local Bellman residual and the global Bellman residual for each sequence segment. The local Bellman residual is calculated based on the state-action sequence information of the local node, and the global Bellman residual is calculated based on the joint state-action sequence information of the local node and its neighboring nodes.

[0007] Based on the local Bellman residual and the global Bellman residual, calculate the neighborhood coupling perturbation residual for each sequence segment;

[0008] The priority weights of each sequence segment are calculated based on the local Bellman residual and the neighborhood coupling perturbation residual.

[0009] Based on each sequence segment and its corresponding priority weight, a scheduling behavior-effect mapping sample set is constructed;

[0010] Based on the scheduling behavior-effect mapping sample set, each sequence segment is weighted by priority weight to obtain a weighted training sample batch; the current policy model is updated based on the weighted training sample batch to obtain the updated decision policy model;

[0011] Based on the updated decision-making strategy model, the process tasks are coordinated and scheduled to output the production line process scheduling scheme.

[0012] Another aspect of this application provides a computer-readable storage medium that stores computer instructions that, when executed by a processor, implement the method of one aspect of this application.

[0013] Compared to existing technologies, the advantages of this application are:

[0014] In multi-agent automated production line scheduling systems, neighboring concurrent scheduling behaviors such as upstream node preemptive task allocation and downstream node blocking feedback can interfere with the state transition of a node's process in real time. Existing technologies only calculate the Bellman residual based on the state-action sequence of the node and prioritize the training of sequence segments. This results in scheduling segments with strong neighboring coupling interference not obtaining training weights that match their actual learning value. This application overcomes the problems of insufficient training signals and biased policy update direction in decision-making strategy models for scenarios with high coupling disturbances such as upstream preemption and downstream blocking. Attached Figure Description

[0015] This application will be further described by way of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not limiting; in these embodiments, the same reference numerals denote the same structures, wherein:

[0016] Figure 1 This is an exemplary flowchart of an artificial intelligence-based production line scheduling optimization method according to some embodiments of this application;

[0017] Figure 2 This is an exemplary flowchart illustrating the construction of a set of scheduling behavior sequence fragments according to some embodiments of this application;

[0018] Figure 3 This is an exemplary flowchart illustrating the calculation of local Bellman residuals according to some embodiments of this application;

[0019] Figure 4 This is an exemplary flowchart illustrating the calculation of the initial coupling condition state-action value estimate according to some embodiments of this application;

[0020] Figure 5 This is an exemplary flowchart illustrating the calculation of global Bellman residuals according to some embodiments of this application;

[0021] Figure 6 This is an exemplary flowchart illustrating how a production line performs process tasks according to some embodiments of this application. Detailed Implementation

[0022] The methods and systems provided in the embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0023] like Figure 1 As shown, the AI-based production line scheduling optimization method of this application aims to address the problems in multi-agent automated production line scheduling systems. Existing technologies only calculate Bellman residuals and prioritize training based on the state-action sequence of the current node, failing to measure the coupling perturbation introduced by the concurrent scheduling behavior of neighboring nodes on the value estimation of the current node. This results in underestimated training weights for highly coupled perturbation scheduling segments, insufficient training signals for the decision-making strategy model, and deviations in the policy update direction. This application constructs neighboring coupled perturbation residuals by simultaneously calculating local and global Bellman residuals and jointly measures the training priority of each sequence segment. This allows the decision-making strategy model to preferentially obtain sufficient training signals from scheduling segments in highly coupled perturbation scenarios, continuously improving the accuracy of multi-agent collaborative scheduling.

[0024] Taking a non-standard customized intelligent spraying production line as an example, this production line includes multiple intelligent agent nodes such as the material loading process node, the spraying process node, and the unloading process node. Each node is executed by a robot and vision unit to perform scheduling actions such as workpiece gripping, spraying operations, and material tray transfer, and the material flow between processes is realized through an AGV logistics system. The process state transition of the spraying process node is simultaneously affected by the preemptive task allocation of the upstream material loading node and the blocking feedback of the downstream unloading node.

[0025] Step S100: Obtain the set of scheduling behavior sequence fragments for each agent node, and the current decision-making strategy model for each agent node, such as... Figure 2 As shown.

[0026] The set of scheduling behavior sequence fragments for each agent node refers to the entire set of sequence fragments containing continuous multi-step state-action-transition relationships extracted in a time-series structure from the historical scheduling interaction data continuously collected from each agent node. The current decision-making strategy model for each agent node refers to the neural network model currently used by each node to evaluate the joint value of the process task state and scheduling actions, and output scheduling decisions accordingly. This model includes functional subnetworks such as a value estimation network and a conditional reward function estimation network.

[0027] In this embodiment, obtaining the set of scheduling behavior sequence fragments for each agent node specifically includes the following sub-steps:

[0028] Step S110: Obtain the scheduling behavior data of each intelligent agent node and the collaborative response data of neighboring nodes.

[0029] The scheduling behavior data of each agent node refers to the scheduling interaction data continuously recorded by each node during the historical scheduling process. This includes the process task status of the node at each moment, the agent scheduling actions executed by the node, and the production state transition information caused by the scheduling actions on the production line (including changes in equipment occupancy status, updates to the process task queue, material flow results, etc.) and the immediate production scheduling rewards (i.e., the single-step performance feedback obtained by the node after the execution of the scheduling action, such as quantitative indicators like process completion rate, equipment utilization rate, and waiting time penalties). The scheduling behavior data of each agent node is recorded in real time by the node's data acquisition module and cached in the local experience buffer.

[0030] The collaborative response data of neighboring nodes refers to the concurrent scheduling actions performed by neighboring intelligent agent nodes that have upstream and downstream process connections with this node at corresponding historical moments. Taking the spraying process node as an example, its collaborative response data includes the workpiece conveying scheduling actions performed by the upstream loading node at each moment (such as "starting irregular workpiece recognition and grabbing" and "suspending conveying and waiting for materials") and the material tray transfer scheduling actions performed by the downstream unloading node at each moment (such as "triggering AGV transfer" and "material tray full load waiting"). The collaborative response data of neighboring nodes is broadcast to this node in real time through the inter-node communication interface and stored in accordance with the scheduling behavior data of this node according to the timestamp.

[0031] Step S120: Calculate the estimated clock deviation between each intelligent agent node, and align and correct the timestamps of the scheduling behavior data of each node according to the estimated clock deviation to obtain cross-node clock synchronization data.

[0032] In a multi-agent production line scheduling system, each agent node is deployed on an independent control unit. The hardware clocks of each node have inherent drift differences, resulting in systematic deviations between the timestamps of the scheduling behavior data recorded by different nodes. If this is not corrected, directly using the original timestamps of each node for cross-node data alignment will cause data from different nodes at the "same moment" to actually correspond to different production state moments, thus introducing false temporal correlations in subsequent sequence fragment extraction.

[0033] The clock skew estimate refers to the systematic offset of the node's clock relative to the reference clock, measured using a clock synchronization protocol (in this embodiment, the PTP Precision Time Protocol is used, with a typical synchronization accuracy better than 1 millisecond, meeting the clock alignment requirements under the condition of a single-step sampling period of 500 milliseconds in this scenario), expressed in milliseconds. Using the clock of the production line's central control server as a reference, each intelligent agent node periodically sends synchronization requests to the reference clock according to a preset synchronization period (in this embodiment, the synchronization period is set to 10 seconds, and this value should not exceed the ratio of the node's hardware clock drift rate to the upper limit of the acceptable timestamp error; in this embodiment, the node's hardware clock drift rate is approximately ±5 milliseconds / minute, and the upper limit of the acceptable error is set to ±5 milliseconds, thus obtaining an upper limit of approximately 60 seconds for the synchronization period; 10 seconds is chosen to retain sufficient safety margin). The clock skew estimate of the node is calculated using round-trip delay estimation.

[0034] Based on the estimated clock skew, the timestamps of the scheduling behavior data of each node are linearly corrected, and the local timestamps of each node are uniformly converted to the reference clock coordinate system to obtain cross-node clock synchronization data. Cross-node clock synchronization data refers to the data set that, after clock skew correction, the scheduling behavior data of each node and the collaborative response data of neighboring nodes are aligned under a unified time base, thus eliminating the time base drift problem caused by the asynchrony of the hardware clocks of each node from the source.

[0035] Step S130: Extract timing segments from cross-node clock synchronization data using a sliding window approach, and align the timing segments extracted from each node across node boundaries using a unified segment boundary benchmark to obtain scheduling behavior sequence segments; the scheduling behavior sequence segments include the process task status of multiple consecutive steps, agent scheduling actions, and production state transition information caused by scheduling actions.

[0036] The sliding window method refers to a data extraction method that sequentially slides and extracts continuous time-series segments on the time axis of cross-node clock synchronization data, using a preset window length (in this embodiment, the window length is set to n steps, where n is a positive integer and can be set according to the production cycle, for example, n=8 steps) and a preset sliding step size (in this embodiment, the sliding step size is set to 1 step). Each extracted segment is a candidate scheduling behavior sequence segment, which includes a continuous n-step process task state sequence, an agent scheduling action sequence, a production state transition sequence, and a production scheduling instant reward sequence.

[0037] A unified segment boundary benchmark refers to using fixed time grid points in a reference clock coordinate system as the alignment benchmark for the start and end boundaries of segments at each node. This ensures that the start and end times of time segments extracted by different nodes under the same segment number are strictly consistent in the reference clock coordinate system. The introduction of a cross-node boundary alignment mechanism is to address residual segment boundary deviations that may remain even after clock correction due to differences in sampling rates or inconsistent event trigger times among nodes. This ensures that the scheduling behavior sequence segments of different nodes correspond precisely within the time window, laying the foundation for the subsequent gradual alignment and extraction of coupling condition contexts.

[0038] Step S140: Perform time sequence continuity verification on the scheduling behavior sequence segments to obtain a set of scheduling behavior sequence segments.

[0039] Temporal continuity verification refers to checking the validity of each candidate scheduling behavior sequence segment, determining whether the timestamp interval between adjacent time steps within the segment exceeds a preset maximum allowable interval threshold (set to twice the single-step sampling period in this embodiment), and whether the jump amplitude of the task state within the segment exceeds a preset state continuity threshold. If there is data loss or temporal breakage within the segment due to communication interruption, node restart, or device malfunction, the segment is determined to be a temporally broken segment and is removed; only scheduling behavior sequence segments that pass the temporal continuity verification are retained, and these are aggregated to form the scheduling behavior sequence segment set for each agent node.

[0040] The set of scheduling behavior sequence fragments refers to the complete set of valid scheduling behavior sequence fragments that retain the Markov transfer chain structure after clock synchronization, cross-node boundary alignment, and timing continuity verification. It serves as the data basis for subsequent residual calculation and priority ranking.

[0041] Step S200: Based on the set of scheduling behavior sequence segments, calculate the local Bellman residual and the global Bellman residual for each sequence segment. The local Bellman residual is calculated based on the state-action sequence information of the local node, and the global Bellman residual is calculated based on the joint state-action sequence information of the local node and its neighboring nodes.

[0042] Local Bellman residuals refer to the Bellman value estimation error calculated solely based on the state-action sequence information of the current node, reflecting the inherent learning bias of the current node's decision-making strategy model towards its scheduling behavior. Global Bellman residuals refer to the Bellman value estimation error calculated when concurrent scheduling actions of neighboring nodes are introduced as a coupling condition, reflecting the value estimation bias of the current node's decision-making strategy model in a real upstream-downstream coupled environment.

[0043] In this embodiment, the local Bellman residuals of each sequence segment are calculated, such as... Figure 3 As shown, the specific steps include the following:

[0044] Step S210: Extract the local node component from the aligned scheduling behavior sequence fragment to obtain the local node state-action subsequence.

[0045] The current node state-action subsequence refers to a subsequence extracted from the scheduling behavior sequence segment that contains only the task status of each step of the current node and the agent's scheduling actions. Specifically, from the cross-node aligned data of the scheduling behavior sequence segment, the task status vector and agent scheduling action vector of the current node at each step are extracted according to the current node identifier index and arranged in time step order to form the current node state-action subsequence. Taking the spraying process node as an example, the task status vector includes features such as the current occupancy status of the spraying machine, the length of the workpiece queue to be sprayed, the upstream material buffer inventory, and the downstream output buffer inventory; the agent scheduling action vector is the encoding of the scheduling instruction executed by the current spraying node, such as the one-hot encoding representation of discrete actions such as "continue the current spraying task", "pause and wait for material", and "switch to the standby machine".

[0046] Step S220: Based on the state-action subsequence of this node, evaluate the process task state and agent scheduling action of the first step of the subsequence using the current decision strategy model to obtain the state-action value estimate of the first step; sum the cumulative discounts of the real-time production scheduling rewards of each step and add the discounted state value of the last step to obtain the Bellman target value of the corresponding first step.

[0047] The first-step state-action value estimation refers to the value estimation network of the current decision-making strategy model. It takes the process task state vector and the agent scheduling action vector of the first step of the state-action subsequence of this node as input, and outputs the scalar estimate of the long-term cumulative reward of the state-action through forward inference.

[0048] The Bellman objective value refers to the value learning objective calculated based on the actual historical data of the current node's state-action subsequence through n-step reward expansion. Specifically, let the subsequence length be n, and the immediate reward for each step of production scheduling be denoted as... (k=0, 1, ..., n-1), the discount factor is denoted as γ (0.9 to 0.99, preferably 0.95 in this embodiment), and the discount value of the task status of the final process is denoted as... Then Bellman's objective value is The final step discount state value is obtained by forward reasoning of the final step task state vector by the state value network of the current decision strategy model.

[0049] Step S230: The absolute value of the difference between the first-step state-action value estimate and the corresponding first-step Bellman target value is used as the local Bellman residual of each scheduling action sequence segment.

[0050] The formula for calculating the local Bellman residual is: ,in, For the initial state-action value estimation, The length of the subsequence; Discount factor; The instant reward for step k; This represents the final state value. The larger the local Bellman residual value, the greater the deviation in the current decision-making strategy model's value estimation of the corresponding process scenario, and the higher the inherent learning value of this segment for the policy update of this node.

[0051] In this embodiment, the calculation of the global Bellman residual for each sequence segment specifically includes the following sub-steps:

[0052] like Figure 4 As shown, in step S240, the current node's state-action subsequence and the concurrent scheduling actions of each neighboring node are extracted from the aligned scheduling behavior sequence fragments, and the concurrent scheduling actions of each neighboring node constitute the coupling condition context of the corresponding step.

[0053] In a multi-agent production line scheduling system, each agent node outputs a scheduling action at each decision step, such as "assign process task A to machine 2", "postpone receiving upstream materials", or "prioritize handling urgent order insertions". While this node is executing its own scheduling action at a certain decision step, its neighboring nodes are also executing their respective scheduling actions at the same time. The scheduling actions executed by these neighboring nodes at that moment are the concurrent scheduling actions for that step.

[0054] The coupling condition context refers to a set of step-level coupling intervention information, which organizes the concurrent scheduling actions of each neighboring node corresponding to each step of a scheduling behavior sequence segment in time step order. Specifically, from the scheduling behavior sequence segment aligned across nodes, the scheduling action vector of each neighboring node at each step is extracted according to the neighboring node identifier. The set of concurrent scheduling action vectors of all neighboring nodes at each step constitutes the coupling condition entry for that step. Arranging the n-step coupling condition entries in time step order constitutes a complete coupling condition context. Taking the spraying process node as an example, each step coupling condition entry in the coupling condition context includes the workpiece conveying action vector executed by the upstream loading node in that step (reflecting the upstream resource supply status) and the material tray transfer action vector executed by the downstream unloading node in that step (reflecting the downstream output blockage status).

[0055] Step S250: Based on the current node's state-action subsequence and the corresponding coupling condition context, the current decision strategy model is used to evaluate the process task state and agent scheduling action of the first step of the subsequence, and obtain the first step coupling condition state-action value estimate.

[0056] The first-step coupling condition state-action value estimation refers to the scalar estimate of the joint utility of the first-step process task state and agent scheduling action by the current decision strategy model when the first-step neighborhood concurrent scheduling action is introduced as a coupling condition. It reflects the true value level of the first-step decision of this node under the coupling intervention of upstream and downstream.

[0057] Specifically, the coupling condition entry corresponding to the first step of the state-action subsequence of this node is indexed from the coupling condition context, and the concurrent scheduling actions of each neighboring node at the first step are extracted from this coupling condition entry. Based on the pre-built upstream and downstream topology table of the production line nodes, the roles of each neighboring node are classified by looking up the table: neighboring nodes that send process tasks to this node are classified as upstream nodes, and their corresponding first-step concurrent scheduling actions are marked as upstream resource constraint actions; neighboring nodes that receive process outputs from this node are classified as downstream nodes, and their corresponding first-step concurrent scheduling actions are marked as downstream blocking feedback actions. The upstream and downstream topology table of the production line nodes refers to a static mapping table pre-configured according to the production line process flow, recording the upstream and downstream connection relationships between each agent node, which is built and stored in the local configuration file of each agent node during the system initialization phase.

[0058] The upstream resource constraint actions are sequentially concatenated according to the topology numbering of the upstream nodes to form an upstream resource constraint sub-vector. Similarly, the downstream blocking feedback actions are sequentially concatenated according to the topology numbering of the downstream nodes to form a downstream blocking feedback sub-vector. The upstream resource constraint sub-vector and the downstream blocking feedback sub-vector are then sequentially concatenated to obtain the first-step directional coupling condition vector. This first-step directional coupling condition vector, by categorizing and arranging upstream constraints and downstream blocking according to topological direction, enables the subsequent value estimation network to perceive the directional differences in coupling interventions and distinguish the different impact mechanisms of upstream resource compression and downstream blocking backlog on the value estimation of this node.

[0059] Extract the first-step process task state vector and agent scheduling action vector from the current node's state-action subsequence. Concatenate these vectors with the first-step directional coupling condition vector at the feature level to obtain the first-step coupling conditional evaluation input vector. Feed this first-step coupling conditional evaluation input vector into the value estimation network of the current decision strategy model. The value estimation network performs forward inference on the first-step coupling conditional evaluation input vector, outputting a scalar value representing the joint utility of the first-step process task state and agent scheduling action under upstream and downstream coupling intervention as represented by the first-step directional coupling condition vector. This scalar value is then determined as the first-step coupling conditional state-action value estimate.

[0060] Step S260: Apply the coupling conditional context of each step to the production scheduling instant reward of the corresponding step to obtain the coupling conditional instant reward of each step; perform cumulative discount summation on the coupling conditional instant rewards of each step, and add the discounted state value under the coupling conditional context of the last step, and determine the sum of the two as the global Bellman target value of the corresponding first step.

[0061] In a multi-agent production line scheduling system, the immediate reward observed by a node at a certain step and the next state after the jump are not solely determined by the node's scheduling action, but are the result of real-time intervention from the concurrent actions of neighboring nodes at that step. Preemptive task allocation by upstream nodes directly compresses the available process resources of this node, and blocking feedback from downstream nodes directly alters the actual production efficiency of this node's actions. This intervention occurs in real-time at every decision step. This application, by progressively extracting the coupling condition context, ensures that the reward and final state value of the global Bellman objective value are calculated under the conditions of concurrent actions in the corresponding neighboring nodes. The calculation results directly correspond to the value deviation actually experienced by the node in the real coupled environment.

[0062] Specifically, each time step in the state-action subsequence of this node is traversed, and the coupling condition entries corresponding to each time step are indexed from the coupling condition context. Based on the upstream and downstream topology relationship table of the production line node, the concurrent scheduling actions of each neighboring node in the coupling condition entries of each step are divided into upstream resource constraint actions and downstream blocking feedback actions. They are then concatenated in the order of topology number to form upstream resource constraint sub-vectors and downstream blocking feedback sub-vectors for each step. Finally, the two are concatenated in order to obtain the upstream and downstream topology neighboring condition vectors for each step.

[0063] The upstream and downstream topological neighborhood condition vectors of each step are concatenated sequentially along the feature dimension with the corresponding step's process task state vector, agent scheduling action vector, and the original single-step scheduling performance reward scalar of this node, respectively, to form the input feature vector of the conditional reward function estimation network for each step. The input feature vectors of the conditional reward function estimation network for each step are then fed into the conditional reward function estimation network in the current decision strategy model. This network performs a nonlinear mapping using a fully connected forward propagation method, and outputs a single scalar after linear activation at the output layer. This scalar is used as the estimated value of the single-step scheduling utility modified by the upstream and downstream neighborhood conditions for each step.

[0064] Let the subsequence length be n, and the offset within a step be k (k=0, 1, ..., n-1). Multiply the estimated single-step scheduling utility value of the upstream and downstream neighborhood conditions of step k by the k-th power of the discount factor γ to obtain the discounted weighted scheduling utility value of step k. Accumulate the n discounted weighted scheduling utility values ​​corresponding to k from 0 to n-1 to obtain the cumulative discounted return of n steps.

[0065] Index the coupling condition entries corresponding to the final step from the coupling condition context, and construct the upstream and downstream topological neighborhood condition vectors for the final step using the method described above. Concatenate the upstream and downstream topological neighborhood condition vectors for the final step with the task state vector of the final step along the feature dimension, and feed them into the state value estimation network in the current decision strategy model. After forward inference, output the estimated state value of the upstream and downstream neighborhood conditions for the final step. Calculate the product of the nth power of γ and the estimated state value of the upstream and downstream neighborhood conditions for the final step to obtain the bootstrap discounted state value for the final step. Add the cumulative reward of n steps to the bootstrap discounted state value for the final step, and determine the sum as the global Bellman objective value corresponding to the first step.

[0066] like Figure 5 As shown, in step S270, the absolute value of the difference between the first-step coupled conditional state-action value estimate and the global Bellman objective value is used as the global Bellman residual of each scheduling behavior sequence segment.

[0067] The formula for calculating the global Bellman residual is: ,in, For the first step of coupled conditional state-action value estimation, This represents the global Bellman objective value. A larger global Bellman residual indicates a greater bias in the value estimation of the current decision-making strategy model after introducing neighborhood coupling conditions, and a higher global learning value for the upstream and downstream coupled intervention scenario corresponding to this segment in updating the strategy model. The starting step of the time window in the scheduling behavior sequence segment time; For the agent in the initial state The discrete dispatching decision action executed below is a single-step instruction that selects a process from the current set of schedulable candidate processes and assigns it to the corresponding processing equipment. , For a finite set of discrete integers, Determined by the process-equipment feasible assignment matrix of the current node. For the starting step of the segment At any given time, the collaborative response data splicing vector broadcast by neighboring intelligent agent nodes that have direct upstream and downstream process dependencies with this node reflects the production load status of the neighboring node, the available amount of the upstream output buffer, the remaining capacity of the downstream input buffer, and the expected completion time deviation of the current action of the neighboring node. Describe the internal state of this node. Only the external coupling constraints are described; the two are then combined to form the complete input of the coupled condition Q-network.

[0068] Step S300: Calculate the neighborhood coupling perturbation residuals of each sequence segment based on the local Bellman residuals and the global Bellman residuals.

[0069] Neighborhood coupling perturbation residual refers to a metric that quantifies the perturbation magnitude introduced by concurrent scheduling actions of neighboring nodes on the Bellman value estimation of a local node by comparing the local Bellman residual with the global Bellman residual. Specifically, the difference between the global Bellman residual and the local Bellman residual is calculated, and the absolute value of the difference is taken as the neighborhood coupling perturbation residual of the corresponding sequence segment. : , For the local Bellman residual, This represents the global Bellman residual.

[0070] The neighborhood coupling perturbation residual characterizes the magnitude of the perturbation introduced by the concurrent scheduling actions of neighboring nodes on the Bellman value estimation of the local node. In the painting process scenario, when the upstream loading node experiences preemptive task allocation or the downstream unloading node experiences continuous blockage, the global Bellman residual will be significantly larger than the local Bellman residual, and the value of the neighborhood coupling perturbation residual will increase significantly accordingly, accurately capturing the strength information of neighborhood coupling intervention in the process scenario in which this segment is located.

[0071] Step S400: Calculate the priority weight of each sequence segment based on the local Bellman residual and the neighborhood coupling perturbation residual.

[0072] Priority weight refers to the sampling importance score that comprehensively reflects the inherent learning value of each scheduling behavior sequence segment of the node and the perception value of neighborhood coupling disturbance. It is used to determine the probability of each segment being extracted in subsequent weighted sampling.

[0073] Specifically, the local Bellman residual and the neighborhood coupled perturbation residual of each scheduling behavior sequence segment are normalized to obtain the normalized local Bellman residual and the normalized neighborhood coupled perturbation residual. The normalization process adopts the min-max normalization method to map the residual values ​​to the interval [0, 1], thereby eliminating the problem of inconsistent numerical scales caused by the difference in dimensions between the two types of residuals.

[0074] Using preset local residual weight coefficient α and coupling perturbation weight coefficient β, the normalized local Bellman residual and the normalized neighborhood coupling perturbation residual are weighted and summed to obtain the comprehensive residual score of each scheduling behavior sequence segment: Where α+β=1. The local residual weight coefficient α (ranging from 0.3 to 0.7) reflects the contribution ratio of the inherent learning error of this node to the priority, and the coupling perturbation weight coefficient β reflects the contribution ratio of the neighborhood coupling perturbation strength to the priority. Both can be adjusted according to the prior knowledge of the coupling strength between upstream and downstream of the production line. In the production line scenario with significant coupling intervention, the value of β can be appropriately increased.

[0075] The priority weights of each scheduling action sequence segment are obtained by performing a power transformation on the comprehensive residual score. : ,in, This is the overall residual score for the segment; the exponential parameter η (ranging from 0.5 to 1.5) controls the concentration of the priority weight distribution. When η is small (e.g., 0.5), the priority weight distribution is relatively flat, and the probability difference between the sampling of each segment is small. When η is large (e.g., 1.5), the priority weight distribution is more concentrated.

[0076] Step S500: Construct a scheduling behavior-effect mapping sample set based on each sequence segment and its corresponding priority weight.

[0077] The scheduling behavior-effect mapping sample set refers to a structured sample set that organizes and stores each scheduling behavior sequence segment together with its corresponding local Bellman residual, global Bellman residual, neighborhood coupling perturbation residual, and priority weight as associated attributes. This sample set uses sequence segments as the basic data unit, preserving the complete temporal structure and residual scoring information of each segment, providing a data foundation for subsequent weighted sampling and policy model updates.

[0078] Step S600: Based on the scheduling behavior-effect mapping sample set, each sequence segment is weighted and sampled according to priority weights to obtain a weighted training sample batch; the current policy model is updated according to the weighted training sample batch to obtain the updated decision policy model.

[0079] Specifically, based on the priority weights corresponding to each scheduling action sequence segment, a predetermined number of scheduling action sequence segments are extracted from the scheduling action-effect mapping sample set using a priority proportional sampling method to form a weighted training sample batch. Priority proportional sampling means that the probability of each segment being extracted is proportional to its priority weight. This allows the scheduling segments of high local Bellman residuals and high neighborhood coupled perturbation residuals to have a higher frequency of occurrence in the training batch, thereby providing more sufficient high-value training signals for the decision-making strategy model.

[0080] Based on the local Bellman residuals corresponding to each scheduling behavior sequence segment in the weighted training sample batch, a policy model update loss function is constructed. The policy model update loss function uses the square of the local Bellman residual of each sequence segment as the loss term for each segment, and the average of the loss terms of all segments in the batch is taken to obtain the batch-level mean square Bellman error loss.

[0081] The network parameters of the current decision policy model are updated by gradient using the loss function of the policy model update. The gradient of the loss function with respect to each network parameter is calculated through backpropagation. The network parameters are updated in the gradient direction using a preset optimizer (such as the Adam optimizer) to obtain the updated decision policy model.

[0082] Step S700: Based on the updated decision strategy model, collaboratively schedule the process tasks and output the production line process scheduling scheme, such as...Figure 6 As shown.

[0083] Specifically, the process task state of each agent node and the concurrent scheduling actions of neighboring nodes are obtained at the current moment, forming the coupling condition context for the current moment. Each agent node, based on its current process task state and coupling condition context, performs forward reasoning using the updated decision strategy model to obtain a state-action value estimate for each available scheduling action at the current moment. The scheduling action with the highest value is selected from these state-action value estimates and used as the execution scheduling action for that node at the current moment. Each agent node drives the production line to perform process task allocation and resource scheduling based on these execution scheduling actions. The execution scheduling actions of each agent node and the corresponding process task allocation results are summarized to output the production line process scheduling scheme.

[0084] Taking the spraying process node as an example, at a certain scheduling moment, the spraying node senses that the upstream feeding node is performing a preemptive workpiece conveying action and the downstream unloading node is in a state of full tray blockage. The updated decision strategy model, based on the current process task status and the upstream and downstream coupling conditions, outputs the execution action of "temporarily suspend receiving upstream materials and start buffer clearing priority scheduling", which effectively responds to the upstream and downstream coupling intervention and ensures that the production line achieves the stable operation goal of less than 8 seconds for single spraying and a product qualification rate of 98%.

[0085] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct a computing device to perform the method of this application.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. A production line scheduling optimization method based on artificial intelligence, characterized in that, include: Obtain the set of scheduling behavior sequence fragments of each agent node, as well as the current decision-making strategy model of each agent node; Based on the set of scheduling behavior sequence segments, calculate the local Bellman residual and the global Bellman residual for each sequence segment. The local Bellman residual is calculated based on the state-action sequence information of the local node, and the global Bellman residual is calculated based on the joint state-action sequence information of the local node and its neighboring nodes. Based on the local Bellman residual and the global Bellman residual, calculate the neighborhood coupling perturbation residual for each sequence segment; The priority weights of each sequence segment are calculated based on the local Bellman residual and the neighborhood coupling perturbation residual. Based on each sequence segment and its corresponding priority weight, a scheduling behavior-effect mapping sample set is constructed; Based on the scheduling behavior-effect mapping sample set, each sequence segment is weighted by priority weight to obtain a weighted training sample batch; the current policy model is updated based on the weighted training sample batch to obtain the updated decision policy model; Based on the updated decision-making strategy model, the process tasks are coordinated and scheduled to output a production line process scheduling plan.

2. The production line scheduling optimization method based on artificial intelligence according to claim 1, characterized in that: Obtain a set of scheduling behavior sequence fragments for each agent node, including: Acquire scheduling behavior data of each intelligent agent node and collaborative response data of neighboring nodes; Calculate the estimated clock skew between each agent node, and align and correct the timestamps of the scheduling behavior data of each node based on the estimated clock skew to obtain cross-node clock synchronization data; Timing segments are extracted from cross-node clock synchronization data using a sliding window approach, and cross-node boundary alignment is performed on the timing segments extracted from each node using a unified segment boundary benchmark to obtain scheduling behavior sequence segments. The scheduling behavior sequence segments contain the process task states of multiple consecutive steps, agent scheduling actions, and production state transition information caused by the scheduling actions. Perform time sequence continuity verification on the scheduling behavior sequence segments to obtain a set of scheduling behavior sequence segments.

3. The production line scheduling optimization method based on artificial intelligence according to claim 2, characterized in that: Calculate the local Bellman residuals for each sequence segment, including: Extract the local node component from the aligned scheduling behavior sequence fragment to obtain the local node state-action subsequence; Based on the state-action subsequence of this node, the process task state and agent scheduling action of the first step of the subsequence are evaluated using the current decision strategy model to obtain the state-action value estimate of the first step; the cumulative discount of the real-time production scheduling reward of each step is summed and the discounted state value of the last step is added to obtain the Bellman target value of the corresponding first step. The absolute value of the difference between the initial state-action value estimate and the corresponding Bellman objective value for the initial step is used as the local Bellman residual for each scheduling action sequence segment.

4. The production line scheduling optimization method based on artificial intelligence according to claim 2, characterized in that: Calculating the global Bellman residual includes: From the aligned scheduling behavior sequence fragments, extract the state-action subsequence of this node and the concurrent scheduling actions of each neighboring node at each step, and use the concurrent scheduling actions of the neighboring nodes at each step to form the coupling condition context of the corresponding step. Based on the state-action subsequence of this node and the corresponding coupling condition context, the current decision strategy model is used to evaluate the process task state and agent scheduling action of the first step of the subsequence, and obtain the value estimate of the first step coupling condition state-action. Apply the coupling conditional context of each step to the production scheduling instant reward of the corresponding step to obtain the coupling conditional instant reward of each step; sum the cumulative discount of the coupling conditional instant rewards of each step, and add the discounted state value under the coupling conditional context of the last step, and determine the global Bellman target value of the corresponding first step by summing the two. The absolute value of the difference between the initial coupled conditional state-action value estimate and the global Bellman objective value is used as the global Bellman residual for each scheduling action sequence segment.

5. The production line scheduling optimization method based on artificial intelligence according to claim 4, characterized in that: The initial coupled conditional state-action value estimate is obtained, including: The coupling conditions corresponding to the first step of the subsequence are indexed from the coupling condition context. Based on the pre-built upstream and downstream topology relationship table of the production line node, the first step concurrent scheduling action of each neighboring node in the coupling condition is divided into upstream resource constraint action and downstream blocking feedback action. According to the upstream and downstream topology numbering order, the upstream resource constraint action and the downstream blocking feedback action are vector concatenated to obtain the first step directional coupling condition vector; the first step directional coupling condition vector is concatenated with the first step process task state vector and agent scheduling action vector in the state-action subsequence of this node at the feature level to obtain the first step coupling condition vector. The first-step coupling condition vector is input into the value estimation network of the current decision strategy model for forward inference to obtain the first-step coupling condition state-action value estimate.

6. The production line scheduling optimization method based on artificial intelligence according to any one of claims 2 to 5, characterized in that: Calculate the neighborhood coupling perturbation residuals for each sequence segment, including: Calculate the difference between the global Bellman residual and the local Bellman residual, and use the absolute value of the difference as the neighborhood coupling perturbation residual of the corresponding sequence segment; The neighborhood coupling perturbation residual characterizes the perturbation magnitude introduced by the concurrent scheduling actions of neighboring nodes on the Bellman value estimation of the local node.

7. The production line scheduling optimization method based on artificial intelligence according to claim 6, characterized in that: Based on the local Bellman residual and the neighborhood coupling perturbation residual, the priority weights of each sequence segment are calculated, including: Normalize each local Bellman residual and the neighborhood coupled perturbation residual respectively; The local Bellman residual and the neighborhood coupling perturbation residual after normalization are weighted and summed using preset weighting coefficients to obtain the comprehensive residual. The priority weights of each sequence segment are obtained by performing a power transformation on the comprehensive residual.

8. The production line scheduling optimization method based on artificial intelligence according to claim 7, characterized in that: The updated decision strategy model is obtained, including: Based on the priority weights of each scheduling behavior sequence segment, a preset number of scheduling behavior sequence segments are extracted from the scheduling behavior-effect mapping sample set using a priority ratio sampling method to form a weighted training sample batch. Based on the local Bellman residuals corresponding to each scheduling behavior sequence segment in the weighted training sample batch, a policy model update loss function is constructed. The network parameters of the current decision policy model are updated using gradients with the policy model update loss function to obtain the updated decision policy model.

9. The production line scheduling optimization method based on artificial intelligence according to claim 8, characterized in that: Based on the updated decision-making strategy model, the process tasks are collaboratively scheduled, and a production line process scheduling scheme is output, including: Obtain the current task status of each agent node and the concurrent scheduling actions of neighboring nodes, and use the concurrent scheduling actions of each neighboring node to form the coupling condition context at the current moment. Each intelligent agent node uses the updated decision strategy model to perform forward reasoning based on the current process task state and coupling condition context to obtain the state-action value estimate corresponding to each optional scheduling action at the current moment. Select the scheduling action with the highest value from the state-action value estimates corresponding to each available scheduling action, and use it as the execution scheduling action for this node at the current moment; Each intelligent agent node drives the production line to allocate tasks and schedule resources according to its execution scheduling actions.

10. A computer-readable storage medium storing computer instructions that, when executed by a processor, implement the method as described in any one of claims 1 to 9.

Images