Industrial production line multi-equipment dynamic collaborative scheduling method and system based on reinforcement learning

Abstract

The invention relates to the technical field of industrial production lines, and discloses an industrial production line multi-device dynamic collaborative scheduling method based on reinforcement learning, comprising the following steps: S1, modeling a three-dimensional state space; s2, hierarchical reinforcement learning architecture; and S3, edge-cloud cooperative execution. According to the industrial production line multi-device dynamic collaborative scheduling method and system based on reinforcement learning, device states, task constraints and resource occupation are integrated into a structured matrix through three-dimensional state space modeling, and a global decision-making layer captures production time sequence dependence by using a bidirectional long-short-term memory network; modeling equipment space association and process constraints through a graph attention network, and generating a global strategy including task allocation, capacity adjustment and resource pre-allocation; and after the edge layer detects the dynamic event, the cloud platform generates a candidate scheme through Monte Carlo tree search, and realizes dynamic event response and multi-target collaborative optimization by combining multiple targets such as global value network evaluation task completion time and equipment load balancing.

Description

Technical Field

[0001] The present invention relates to the technical field of industrial production lines, and specifically to a method and system for dynamic collaborative scheduling of multiple devices in an industrial production line based on reinforcement learning. Background Art

[0002] Industrial production line equipment scheduling is the core control link of the industrial production process. Its essence is to achieve efficient and stable collaborative operation of multiple devices through reasonable planning of task allocation, resource coordination and equipment control. Specifically, it is necessary to dynamically adjust equipment processing parameters, material transmission paths and resource allocation strategies under conditions such as task priority, process sequence, and equipment process constraints to achieve the goals of shortening production cycles, improving equipment utilization, and reducing energy costs.

[0003] With the development of Industry 4.0 and intelligent manufacturing, production lines are characterized by diversified equipment types, complex task requirements, and dynamic production environments, which put higher demands on the scheduling system's real-time response capabilities, multi-objective optimization capabilities, and large-scale equipment coordination capabilities.

[0004] However, existing industrial production line scheduling technologies have two key flaws: first, traditional static scheduling methods are difficult to cope with dynamic events such as equipment failures and material shortages, and only optimize a single objective, lacking global coordination such as equipment load and energy consumption, resulting in poor overall efficiency; second, the centralized architecture relies on cloud processing, which has problems such as high communication latency and poor robustness, and insufficient modeling of equipment spatial correlation and production time dependence, which limits the rationality of the scheduling scheme. Therefore, a dynamic collaborative scheduling method and system for multiple devices in industrial production lines based on reinforcement learning is proposed. Summary of the Invention

[0005] (1) Technical problems solved

[0006] In response to the shortcomings of the existing technology, the present invention provides a method and system for dynamic collaborative scheduling of multiple devices in an industrial production line based on reinforcement learning. The method and system have the advantages of coping with dynamic events such as equipment failures and material shortages, multi-objective collaborative optimization, and reasonable modeling of equipment spatial correlation and time dependency. It solves the two key defects of existing industrial production line scheduling technology: first, traditional static scheduling methods are difficult to cope with dynamic events such as equipment failures and material shortages, and only optimize a single target, lacking global coordination such as equipment load and energy consumption, resulting in low overall efficiency; second, the centralized architecture relies on cloud processing, has problems of high communication latency and poor robustness, and insufficient modeling of equipment spatial correlation and production time dependency, which limits the rationality of the scheduling scheme.

[0007] (2) Technical solution

[0008] To achieve the above-mentioned objectives of coping with dynamic events such as equipment failures and material shortages, as well as multi-objective collaborative optimization and reasonable modeling of equipment spatial associations and temporal dependencies, the present invention provides the following technical solutions: a method for dynamic collaborative scheduling of multiple devices in an industrial production line based on reinforcement learning, comprising the following steps:

[0009] S1. 3D State Space Modeling: Build a unified model that includes equipment status, task constraints, and resource usage. This model uses a matrix format to describe the real-time status of the production line. Equipment status includes operating mode, processing progress, and health parameters. Task constraints include priority, process path, and time window. Resource usage includes material inventory, transportation status, and energy consumption.

[0010] S2. Hierarchical reinforcement learning architecture: A two-layer network structure consisting of a global decision-making layer and a local execution layer is designed. The global layer generates global strategies for task allocation, capacity adjustment, and resource provisioning through spatiotemporal joint modeling. The local layer generates device-level control instructions for machining parameter adjustment, tooling switching, and fault response based on device-specific strategies.

[0011] S3. Edge-cloud collaborative execution: The edge layer collects device data in real time and performs local control. The cloud platform trains global strategies through historical data and reconstructs scheduling plans based on dynamic events to achieve collaborative optimization of hierarchical decision-making and real-time response.

[0012] Preferably, the global decision layer adopts a hybrid neural network architecture:

[0013] Use bidirectional long short-term memory networks to capture the time series dependencies of the production process and identify the dynamic characteristics of equipment failure cycles and task delay trends;

[0014] The spatial correlation and process constraints between devices are modeled through graph attention networks, and the device collaboration weights are calculated to optimize task allocation.

[0015] The output layer generates global scheduling instructions including the task-equipment allocation matrix, equipment capacity adjustment coefficient, and resource pre-allocation list, taking into account both rigid scheduling constraints and flexible optimization.

[0016] Preferably, the device-specific policy network design of the local execution layer is:

[0017] The input integrates the real-time status of the device, global policy slices, and historical instructions of adjacent devices to form an input space for device context awareness.

[0018] A deep Q-network is used to build a strategic model, which outputs a discrete action space including multi-level machining parameter adjustment, fixture switching, and fault self-diagnosis triggering.

[0019] The state transition data is stored through the experience replay mechanism, and the network parameters are optimized in combination with the target network soft update algorithm to reduce strategy oscillation during training.

[0020] Preferably, the dynamic event response mechanism in the edge-cloud collaborative execution includes:

[0021] The edge layer detects abnormal conditions, including equipment failures, material shortages, and emergency orders, through multi-sensor data fusion, and generates event description information including event type, impact range, and process constraints.

[0022] The cloud platform freezes the current global policy parameters based on the event description information and reconstructs the global state space by combining the device interaction graph and the task constraint matrix;

[0023] Multiple candidate scheduling plans are generated through Monte Carlo tree search. Combined with the task completion time, equipment load balancing, and resource utilization of the global value network evaluation plan, the optimal plan that meets the process rules is selected and sent to the edge layer for execution.

[0024] Preferably, the collaborative training process of the hierarchical reinforcement learning architecture includes:

[0025] Offline pre-training phase: Utilize historical production data to build a virtual production line environment, train the global decision-making network to learn scheduling strategies for typical production scenarios, and form an initial strategy knowledge base.

[0026] Online fine-tuning: The edge layer collects device execution data in real time to generate experience tuples, which are then aggregated and normalized before being uploaded to the cloud. The cloud then blends historical and real-time data to optimize the global strategy.

[0027] Parameter synchronization mechanism: A soft update strategy is used to achieve interaction between global and local network parameters, and the core parameters of the global strategy are proportionally synchronized to the local network to avoid drastic fluctuations in device control strategies.

[0028] A reinforcement learning-based dynamic collaborative scheduling system for multiple devices in industrial production lines, including:

[0029] Edge layer: Deploy multimodal sensor arrays and edge computing nodes. The sensor array collects equipment vibration signals, workpiece visual features, and material identification information. The edge nodes preprocess the data through three-level filtering and run a containerized scheduling engine to achieve microsecond-level equipment status collection and millisecond-level control instruction generation.

[0030] Cloud platform layer: This includes a global policy server for distributed training, a time series database, and a collaborative optimization module. The server processes training data in parallel using a parameter server-compute node architecture. The database stores historical states using a three-dimensional index of device, time, and parameters. The optimization module generates dynamic event response solutions based on Monte Carlo tree search.

[0031] Communication network layer: supports time-triggered and event-triggered dual-mode communication to ensure low-latency and highly reliable data interaction.

[0032] Preferably, the software architecture of the edge computing node includes:

[0033] Real-time control layer: runs hard real-time tasks to parse device control instructions, generates standard control codes that can be recognized by industrial controllers, and drives device actions through high-precision digital interfaces;

[0034] Policy reasoning layer: Deploys an optimized deep reinforcement learning inference engine to read device status in real time and generate control instructions, supporting independent deployment and hot updates of policy models for different device types.

[0035] Communication management layer: Efficiently compresses and encapsulates sensor data, dynamically selects transmission mode based on data type, and ensures microsecond response of critical control data and batch transmission of non-critical data.

[0036] Preferably, the distributed training mechanism of the global policy server includes:

[0037] The parameter server cluster adopts a master-slave replication mode. The master server synchronizes the latest network parameters in real time, and the slave server supports concurrent pull of edge nodes to ensure parameter consistency for tens of thousands of devices.

[0038] Compute nodes process training data in parallel through task sharding, use an asynchronous gradient descent algorithm to update network parameters, and support group training by device type to improve training efficiency in large-scale scheduling.

[0039] The strategy model management module supports version control and remote deployment, and can batch issue customized strategies according to production line configuration, ensuring that the optimal control strategy for different equipment types is independently updated.

[0040] Preferably, the deployment scheme of the multimodal sensor array is:

[0041] The vibration sensor is installed close to the key components of the equipment to collect high-frequency vibration signals for equipment fault feature extraction;

[0042] The visual recognition device is deployed above the processing station and detects the workpiece position deviation and processing quality through image recognition technology;

[0043] Radio frequency identification equipment is integrated into the material channel to read material batch information and processing progress in real time, forming a multi-dimensional production data collection system.

[0044] Preferably, the generation and evaluation method of the candidate scheduling scheme is:

[0045] Expand feasible scheduling actions based on the device interaction graph, build a search tree with the current state as the root node, and prioritize expanding task reallocation nodes for high-load devices;

[0046] The long-term reward of the solution is predicted through the global value network, and infeasible solutions are filtered out by combining manually defined process rules to ensure that the generated solution meets the actual constraints of process sequence and equipment compatibility.

[0047] A non-dominated sorting genetic algorithm is used to perform multi-objective optimization on candidate solutions and generate a Pareto optimal solution set for automatic selection by the system or manual intervention.

[0048] (3) Beneficial effects

[0049] Compared with the existing technology, the present invention provides a method and system for dynamic collaborative scheduling of multiple devices in an industrial production line based on reinforcement learning, which has the following beneficial effects:

[0050] 1. This reinforcement learning-based dynamic collaborative scheduling method and system for multiple devices in industrial production lines integrates device status, task constraints, and resource occupancy into a structured matrix through three-dimensional state space modeling. The global decision-making layer uses a bidirectional long-short-term memory network to capture production time series dependencies, and uses a graph attention network to model device spatial associations and process constraints to generate a global strategy that includes task allocation, capacity adjustment, and resource provisioning. After the edge layer detects dynamic events, the cloud platform generates candidate solutions through Monte Carlo tree search, and combines the global value network to evaluate multiple objectives such as task completion time and device load balancing to achieve dynamic event response and multi-objective collaborative optimization, solving the problem that traditional static scheduling is difficult to cope with dynamic events and the low efficiency caused by single-objective optimization.

[0051] 2. This reinforcement learning-based dynamic collaborative scheduling method and system for multiple devices in an industrial production line adopts an edge-cloud collaborative execution architecture. The edge layer collects device data in real time and performs local control through multimodal sensors. The cloud platform uses historical data to train global strategies and freezes parameters and reconstructs the global state space during dynamic events. The communication network layer supports time-triggered and event-triggered dual-mode communication to ensure low-latency and high-reliability interaction. In the layered reinforcement learning architecture, the global decision layer jointly models the spatial correlation and temporal dependence of devices in time and space. The local execution layer integrates global strategy slices, device status and adjacent instructions to generate control instructions. The parameter synchronization mechanism avoids drastic policy fluctuations through soft updates, reduces dependence on the cloud and communication delay, improves the rationality of device association modeling and system robustness, and solves the defects of high communication delay, poor robustness and insufficient association modeling in the centralized architecture. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a flow chart of the method for dynamic collaborative scheduling of multiple devices in an industrial production line of the present invention;

[0053] Figure 2 This is the architecture diagram of the industrial production line multi-device dynamic collaborative scheduling system of the present invention. DETAILED DESCRIPTION

[0054] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the embodiments of the present invention and the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0055] See also Figure 1-2 , a dynamic collaborative scheduling method for multiple devices in an industrial production line based on reinforcement learning, comprising the following steps:

[0056] S1. 3D State Space Modeling: Build a unified model that includes equipment status, task constraints, and resource usage. This model uses a matrix format to describe the real-time status of the production line. Equipment status includes operating mode, processing progress, and health parameters. Task constraints include priority, process path, and time window. Resource usage includes material inventory, transportation status, and energy consumption.

[0057] S2. Hierarchical reinforcement learning architecture: A two-layer network structure consisting of a global decision-making layer and a local execution layer is designed. The global layer generates global strategies for task allocation, capacity adjustment, and resource provisioning through spatiotemporal joint modeling. The local layer generates device-level control instructions for machining parameter adjustment, tooling switching, and fault response based on device-specific strategies.

[0058] S3. Edge-cloud collaborative execution: The edge layer collects device data in real time and performs local control. The cloud platform trains global strategies through historical data and reconstructs scheduling plans based on dynamic events to achieve collaborative optimization of hierarchical decision-making and real-time response.

[0059] A reinforcement learning-based dynamic collaborative scheduling system for multiple devices in industrial production lines, including:

[0060] Edge layer: Deploy multimodal sensor arrays and edge computing nodes. The sensor array collects equipment vibration signals, workpiece visual features, and material identification information. The edge nodes preprocess the data through three-level filtering and run a containerized scheduling engine to achieve microsecond-level equipment status collection and millisecond-level control instruction generation.

[0061] Cloud platform layer: This includes a global policy server for distributed training, a time series database, and a collaborative optimization module. The server processes training data in parallel using a parameter server-compute node architecture. The database stores historical states using a three-dimensional index of device, time, and parameters. The optimization module generates dynamic event response solutions based on Monte Carlo tree search.

[0062] Communication network layer: supports time-triggered and event-triggered dual-mode communication to ensure low-latency and highly reliable data interaction.

[0063] Example 1:

[0064] This embodiment designs a technical implementation of three-dimensional state space modeling for a dynamic collaborative scheduling method of multiple devices in an industrial production line based on reinforcement learning.

[0065] In the equipment status dimension, the operating mode uses a one-hot encoding method, and standby, processing, and fault correspond to unit vectors of different dimensions, such as [1,0,0] for standby, [0,1,0] for processing, and [0,0,1] for fault. The processing progress is quantified in the range of 0 to 1 as the ratio of the actual completion amount of the process to the total workload. The health parameter integrates indicators such as the root mean square value of the vibration signal, the gradient change rate of the temperature sensor, and the harmonic distortion rate of the current sensor to form a vector consisting of 10 to 15 features.

[0066] In the task constraint dimension, the priority is assigned an integer code from 0 to 10 based on the urgency of the order, with larger values ​​indicating higher priority. The process path is modeled using a directed acyclic graph and converted into an N×N adjacency matrix, where N is the number of processes and a matrix element of 1 indicates the existence of a direct predecessor relationship. The time window is based on the planned start and end times and is represented by a Gaussian distribution vector with a fluctuation range of ±10%.

[0067] In the resource occupancy dimension, material inventory is represented by the ratio of real-time inventory to safety inventory. The ratio range is set to 0 to 2, and an early warning is triggered when it is lower than 0.8; the transportation status is constructed by constructing a 5- to 8-dimensional vector through the coordinates, operating speed, and load type of the carrier. The coordinates use three-dimensional space coordinates, and the speed is a normalized value from 0 to the maximum operating speed; energy consumption is calculated by integrating the equipment power curve, and the result is normalized to the range of 0 to 1.

[0068] After Z-score normalization, the three-dimensional state is combined into a state matrix of N×(10-15+3+5-8) dimensions in the order of device state, task constraint, and resource occupancy, where N is the number of devices, providing a structured input space for the reinforcement learning model.

[0069] Example 2:

[0070] This embodiment aims at the global decision-making layer hybrid neural network architecture and constructs a specific network structure for spatiotemporal joint modeling.

[0071] The bidirectional long short-term memory network adopts a two-hidden layer design, each layer contains 200 to 300 memory units. The input is a device state sequence with a time window length of 20 to 50, which is compressed into a 200-dimensional vector. The time series dependency is captured through forward and backward propagation. The output layer uses the Sigmoid activation function to generate the device failure probability vector, with a value range of 0 to 1, and the task delay risk index, with a value range of 0 to 100.

[0072] The graph attention network constructs a device interaction graph. The node features include 8 to 12-dimensional attributes such as device type, processing accuracy, and production capacity limit. The type is encoded using one-hot encoding, and the accuracy and capacity are numerical parameters. The edge feature is a process constraint matrix, where an element of 1 indicates that the devices can collaborate, and 0 indicates that they cannot collaborate. The collaboration weight is calculated through a 3 to 6-head attention mechanism, and the dynamic association matrix between devices is output, with element values ​​ranging from 0 to 1.

[0073] After the spatiotemporal features are fused through the cascade layer, they are processed by two layers of fully connected networks. Each layer contains 500 to 800 neurons, and the activation function uses LeakyReLU. The output layer uses the Softmax function, Tanh function and Argmax function to generate the task-equipment allocation matrix, probability distribution, capacity adjustment coefficient with a value range of 0.5 to 1.5, resource pre-allocation list, discrete allocation results, and imposes penalties on outputs that violate the process sequence or equipment load limit through the constraint projection layer to ensure that the scheduling instructions meet the actual production constraints.

[0074] Example 3:

[0075] This embodiment designs a complete technical link of the deep Q network for the local execution layer device-specific policy network.

[0076] The input layer receives the real-time status of the equipment, which consists of 10 to 15-dimensional health parameters, 3-dimensional operating mode unique hot encoding, 1-dimensional progress ratio, global strategy slice, including 1-bit task assignment identifier and 1 capacity coefficient floating-point value, and unique hot encoding of the previous 2 to 4 steps of actions of adjacent equipment. The encoding dimension of each step of action is the size of the action space. After normalization, it is input into the three-layer fully connected network.

[0077] The first layer contains 200 to 300 neurons, and the activation function is ELU; the second layer contains 100 to 200 neurons, and the activation function is ReLU; the third layer contains 50 to 100 neurons, and the activation function is a linear function.

[0078] The number of neurons in the output layer is consistent with the size of the discrete action space, which is set to 15 to 20, corresponding to 3 to 5 levels of processing parameter adjustment, such as the speed is set in stages from 1000 to 4000 rpm, 3 to 4 types of tooling fixture switching, different fixtures corresponding to different action codes, fault self-diagnosis triggering, emergency stop response, idling waiting and other actions.

[0079] The capacity of the experience replay pool is set to 80,000 to 120,000 entries. The priority experience replay mechanism is adopted, and the sampling probability is calculated according to the time difference error. The target network is soft-updated every 800 to 1200 steps, with an update ratio of 0.0005 to 0.0015.

[0080] The Huber loss function is used in the training process, and the optimizer adopts an adaptive learning rate algorithm. The initial value of the learning rate is 0.0005 to 0.001, and it decays linearly to 10% of the initial value with each training round.

[0081] Example 4:

[0082] This embodiment designs a multi-stage exception handling process for a dynamic event response mechanism.

[0083] Edge-layer multi-sensor fusion uses a cascade processing architecture: the vibration signal is first filtered with a 50 to 5000 Hz bandpass filter, and then a short-time Fourier transform is used to generate a time-frequency diagram, which is then input into a convolutional neural network, such as a ResNet18 variant model, to detect equipment failures. The model accuracy must reach more than 90%. Visual images are detected through target detection models, such as the simplified version of YOLOv5, to detect workpiece position deviations, with the threshold set to ±0.5mm to ±1mm. RFID data is compared with the production plan of the manufacturing execution system, and a material shortage event is triggered when the deviation exceeds 5% to 10%.

[0084] The event description information is encapsulated as structured data containing the event type code, where 0 to 2 correspond to failure, material shortage, inserted order, affected equipment set, equipment ID list, and process constraint relaxation, respectively. The interval value from 0 to 0.5 represents the process constraint ratio that is allowed to be adjusted.

[0085] After receiving the event information, the cloud platform immediately freezes the global policy parameters and generates candidate scheduling plans based on Monte Carlo tree search: taking the current state as the root node, the upper confidence bound strategy is adopted to expand the node, the number of single-node simulations is 50 to 200 times, the search depth is 10 to 20 layers, and the task reallocation nodes of high-load devices are expanded first.

[0086] The global value network is based on the LSTM-Attention architecture, takes as input the state matrix and action encoding, and outputs the predicted value of task completion time with an error range of ±5% to ±10%, equipment load standard deviation, and resource utilization. It uses a rule base containing more than 200 process rules to filter out invalid solutions, and finally generates a solution set consisting of 5 to 10 Pareto optimal solutions through a non-dominated sorting genetic algorithm. The algorithm sets the population size to 80 to 120, the iterations to 40 to 60 generations, the crossover probability to 0.8 to 0.95, and the mutation probability to 0.005 to 0.02.

[0087] Embodiment 5:

[0088] This embodiment designs a full-cycle training framework for the hierarchical reinforcement learning collaborative training process.

[0089] Offline pre-training is based on a virtual simulation environment, such as Unity ML-Agents or the Gazebo customized platform, which simulates 8 to 12 typical production scenarios, including normal production, equipment failures, and emergency orders.

[0090] The global decision layer adopts a proximal policy optimization algorithm. The policy network consists of a three-layer convolutional neural network and a two-layer fully connected network. The convolution layer uses a 3×3 convolution kernel with a step size of 2. The number of channels is 32, 64, and 128 respectively. Each layer of the fully connected layer contains 300 to 500 neurons. The value network structure is the same as that of the policy network.

[0091] The loss function consists of a policy loss with a weight of 1.0, a value loss with a weight of 0.5, and an entropy regularization term with a weight of 0.01. The training cycle is 1,000 to 2,000 rounds, and 80,000 to 120,000 steps of experience data are collected in each round. Training is stopped when the average reward reaches a preset threshold, such as 700 to 900, to form an initial policy knowledge base.

[0092] During the online fine-tuning phase, the experience tuples collected in real time by the edge layer, including state, action, reward, and next state, are processed through feature engineering. The state features are compressed to 20 to 40 dimensions through principal component analysis. The reward function is designed to be a task completion reward of +100 for each completed task, a device overload penalty of -50 for each overload, an energy consumption cost of -0.1 to -0.5 per kilowatt-hour, and a delayed order penalty of -20 to -50 per order. These are uploaded to the cloud through a distributed message queue, such as Kafka, with a cycle of 300 to 700ms.

[0093] A hybrid strategy is used for parameter synchronization: key parameters, such as the attention network weights and Bi-LSTM memory unit parameters shown in the figure, are fully synchronized daily. Non-key parameters, such as the fully connected layer bias items, are updated hourly using an exponential moving average algorithm. The update coefficient τ is set to 0.99 to 0.999. The local network retains 5% to 15% of independent exploration weights to meet the personalized control needs of the device.

[0094] Example 6:

[0095] This embodiment designs an end-to-end technical implementation solution for the scheduling system architecture.

[0096] The edge layer deploys embedded computing nodes equipped with real-time operating systems such as QNX or VxWorks. The real-time control layer executes hard real-time tasks in a cycle of 1 to 5ms, and converts control instructions into codes recognizable by programmable logic controllers through industrial communication protocols such as OPC UA or EtherCAT to drive device actions.

[0097] The policy inference layer integrates lightweight inference engines such as TensorFlowLite or ONNXRuntime. The model is deployed after 8-bit quantization, and the inference delay is controlled within 5 to 10ms. Hot updates are supported, and loading is completed within 100 to 200ms after the model file is replaced.

[0098] Multimodal sensor deployment plan: The vibration sensor uses a piezoelectric accelerometer with a sampling frequency set to 5kHz to 20kHz. It uses a second-order Butterworth high-pass filter with a cutoff frequency of 10Hz to remove low-frequency noise. The visual recognition device uses an industrial camera with a resolution of 1920×1080 to 2592×1944 pixels, a frame rate of 20 to 30fps, and a fixed-focus lens. The RFID reader operates at a frequency of 900MHz to 2.4GHz, supports batch readings of 30 to 70 times per second, and is integrated on both sides of the material channel.

[0099] The cloud platform adopts a distributed cluster architecture. The global policy server consists of 1 master parameter server and 8 to 12 slave parameter servers. It synchronizes parameters through the remote procedure call protocol and supports concurrent pull requests from tens of thousands of devices. The computing nodes are grouped by equipment type, such as processing equipment, logistics equipment, and testing equipment. Each group contains 50 to 100 instances. The asynchronous stochastic gradient descent algorithm is used to update the network parameters. The learning rate attenuation factor is 0.9 to 0.99 / every 100,000 steps, supporting the processing of 80,000 to 120,000 training data per second.

[0100] The communication network layer adopts time-sensitive network technology, and the time-triggered frame period is set to 1 to 5ms for transmitting control instructions and critical status data. The IEEE802.1Qbv gating mechanism is used to ensure latency determinism. The event-triggered frame adopts IEEE802.1Qci priority scheduling, with the abnormal data priority set to the highest level 7, the control instruction priority to level 5, and the non-critical status data priority to level 3. The end-to-end latency 99% percentile is ≤500μs, ensuring low latency and high reliability of data interaction.

[0101] In summary, the reinforcement learning-based dynamic collaborative scheduling method and system for multiple devices in industrial production lines integrates device status, task constraints, and resource occupancy into a structured matrix through three-dimensional state space modeling. The global decision-making layer uses a bidirectional long-short-term memory network to capture production time series dependencies, and models device spatial associations and process constraints through a graph attention network to generate a global strategy that includes task allocation, capacity adjustment, and resource provisioning. After the edge layer detects dynamic events, the cloud platform generates candidate solutions through Monte Carlo tree search, and combines the global value network to evaluate multiple objectives such as task completion time and equipment load balancing, thereby realizing dynamic event response and multi-objective collaborative optimization, solving the problem that traditional static scheduling is difficult to cope with dynamic events and the inefficiency caused by single-objective optimization.

[0102] In addition, the method and system for dynamic collaborative scheduling of multiple devices in industrial production lines based on reinforcement learning adopts an edge-cloud collaborative execution architecture. The edge layer collects device data in real time through multimodal sensors and performs local control. The cloud platform uses historical data to train global strategies and freezes parameters and reconstructs the global state space during dynamic events. The communication network layer supports time-triggered and event-triggered dual-mode communication to ensure low-latency and high-reliability interaction. In the layered reinforcement learning architecture, the global decision layer jointly models the spatial correlation and time dependency of devices in time and space. The local execution layer integrates global strategy slices, device status and adjacent instructions to generate control instructions. The parameter synchronization mechanism avoids The strategy fluctuates violently, reducing dependence on the cloud and communication delay, improving the rationality of equipment association modeling and system robustness, solving the defects of high communication delay, poor robustness and insufficient association modeling of centralized architecture, and solving the two key defects of existing industrial production line scheduling technology: First, traditional static scheduling methods are difficult to deal with dynamic events such as equipment failure and material shortage, and only optimize a single target, lacking global coordination such as equipment load and energy consumption, resulting in low overall efficiency; second, centralized architecture relies on cloud processing, with high communication delay and poor robustness, and insufficient modeling of equipment spatial association and production time dependence, which limits the rationality of scheduling solutions.

[0103] The relevant modules involved in this system are all hardware system modules or functional modules that combine computer software programs or protocols with hardware in the existing technology. The computer software programs or protocols involved in the functional modules are themselves technologies that are well known to those skilled in the art and are not improvements to this system. The improvements to this system are the interaction or connection relationships between the modules, that is, improvements to the overall structure of the system to solve the corresponding technical problems to be solved by this system.

[0104] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A method for dynamic collaborative scheduling of multiple devices in industrial production lines based on reinforcement learning, characterized in that: The following steps are involved: S1. 3D State Space Modeling: Build a unified model that includes equipment status, task constraints, and resource usage. This model uses a matrix format to describe the real-time status of the production line. Equipment status includes operating mode, processing progress, and health parameters. Task constraints include priority, process path, and time window. Resource usage includes material inventory, transportation status, and energy consumption. S2. Hierarchical reinforcement learning architecture: A two-layer network structure consisting of a global decision-making layer and a local execution layer is designed. The global layer generates global strategies for task allocation, capacity adjustment, and resource provisioning through spatiotemporal joint modeling. The local layer generates device-level control instructions for machining parameter adjustment, tooling switching, and fault response based on device-specific strategies. S3. Edge-cloud collaborative execution: The edge layer collects device data in real time and performs local control. The cloud platform trains global strategies through historical data and reconstructs scheduling plans based on dynamic events to achieve collaborative optimization of hierarchical decision-making and real-time response.

2. The method for dynamic collaborative scheduling of multiple devices in an industrial production line based on reinforcement learning according to claim 1 is characterized in that: The global decision layer adopts a hybrid neural network architecture: Use bidirectional long short-term memory networks to capture the time series dependencies of the production process and identify the dynamic characteristics of equipment failure cycles and task delay trends; The spatial correlation and process constraints between devices are modeled through graph attention networks, and the device collaboration weights are calculated to optimize task allocation. The output layer generates global scheduling instructions including the task-equipment allocation matrix, equipment capacity adjustment coefficient, and resource pre-allocation list, taking into account both rigid scheduling constraints and flexible optimization.

3. The method for dynamic collaborative scheduling of multiple devices in an industrial production line based on reinforcement learning according to claim 1 is characterized in that: The device-specific policy network design of the local execution layer is as follows: The input integrates the real-time status of the device, global policy slices, and historical instructions of adjacent devices to form an input space for device context awareness. A deep Q-network is used to build a strategic model, which outputs a discrete action space including multi-level machining parameter adjustment, fixture switching, and fault self-diagnosis triggering. The state transition data is stored through the experience replay mechanism, and the network parameters are optimized in combination with the target network soft update algorithm to reduce strategy oscillation during training.

4. The method for dynamic collaborative scheduling of multiple devices in an industrial production line based on reinforcement learning according to claim 1 is characterized in that: The dynamic event response mechanism in the edge-cloud collaborative execution includes: The edge layer detects abnormal conditions, including equipment failures, material shortages, and emergency orders, through multi-sensor data fusion, and generates event description information including event type, impact range, and process constraints. The cloud platform freezes the current global policy parameters based on the event description information and reconstructs the global state space by combining the device interaction graph and the task constraint matrix; Multiple candidate scheduling plans are generated through Monte Carlo tree search. Combined with the task completion time, equipment load balancing, and resource utilization of the global value network evaluation plan, the optimal plan that meets the process rules is selected and sent to the edge layer for execution.

5. The method for dynamic collaborative scheduling of multiple devices in an industrial production line based on reinforcement learning according to claim 1 is characterized in that: The collaborative training process of the hierarchical reinforcement learning architecture includes: Offline pre-training phase: Utilize historical production data to build a virtual production line environment, train the global decision-making network to learn scheduling strategies for typical production scenarios, and form an initial strategy knowledge base. Online fine-tuning: The edge layer collects device execution data in real time to generate experience tuples, which are then aggregated and normalized before being uploaded to the cloud. The cloud then blends historical and real-time data to optimize the global strategy. Parameter synchronization mechanism: A soft update strategy is used to achieve interaction between global and local network parameters, and the core parameters of the global strategy are proportionally synchronized to the local network to avoid drastic fluctuations in device control strategies.

6. A dynamic collaborative scheduling system for multiple devices in industrial production lines based on reinforcement learning, characterized in that: include: Edge layer: Deploy multimodal sensor arrays and edge computing nodes. The sensor array collects equipment vibration signals, workpiece visual features, and material identification information. The edge nodes preprocess the data through three-level filtering and run a containerized scheduling engine to achieve microsecond-level equipment status collection and millisecond-level control instruction generation. Cloud platform layer: This includes a global policy server for distributed training, a time series database, and a collaborative optimization module. The server processes training data in parallel using a parameter server-compute node architecture. The database stores historical states using a three-dimensional index of device, time, and parameters. The optimization module generates dynamic event response solutions based on Monte Carlo tree search. Communication network layer: supports time-triggered and event-triggered dual-mode communication to ensure low-latency and highly reliable data interaction.

7. The multi-device dynamic collaborative scheduling system for industrial production lines based on reinforcement learning according to claim 6 is characterized in that: The software architecture of the edge computing node includes: Real-time control layer: runs hard real-time tasks to parse device control instructions, generates standard control codes that can be recognized by industrial controllers, and drives device actions through high-precision digital interfaces; Policy reasoning layer: Deploys an optimized deep reinforcement learning inference engine to read device status in real time and generate control instructions, supporting independent deployment and hot updates of policy models for different device types. Communication management layer: Efficiently compresses and encapsulates sensor data, dynamically selects transmission mode based on data type, and ensures microsecond response of critical control data and batch transmission of non-critical data.

8. The multi-device dynamic collaborative scheduling system for industrial production lines based on reinforcement learning according to claim 6 is characterized in that: The distributed training mechanism of the global policy server includes: The parameter server cluster adopts a master-slave replication mode. The master server synchronizes the latest network parameters in real time, and the slave server supports concurrent pull of edge nodes to ensure parameter consistency for tens of thousands of devices. Compute nodes process training data in parallel through task sharding, use an asynchronous gradient descent algorithm to update network parameters, and support group training by device type to improve training efficiency in large-scale scheduling. The strategy model management module supports version control and remote deployment, and can batch issue customized strategies according to production line configuration, ensuring that the optimal control strategy for different equipment types is independently updated.

9. The multi-device dynamic collaborative scheduling system for industrial production lines based on reinforcement learning according to claim 6 is characterized in that: The deployment scheme of the multimodal sensor array is: The vibration sensor is installed close to the key components of the equipment to collect high-frequency vibration signals for equipment fault feature extraction; The visual recognition device is deployed above the processing station and detects the workpiece position deviation and processing quality through image recognition technology; Radio frequency identification equipment is integrated into the material channel to read material batch information and processing progress in real time, forming a multi-dimensional production data collection system.

10. The multi-device dynamic collaborative scheduling system for industrial production lines based on reinforcement learning according to claim 6, characterized in that: The generation and evaluation method of the candidate scheduling scheme is as follows: Expand feasible scheduling actions based on the device interaction graph, build a search tree with the current state as the root node, and prioritize expanding task reallocation nodes for high-load devices; The long-term reward of the solution is predicted through the global value network, and infeasible solutions are filtered out by combining manually defined process rules to ensure that the generated solution meets the actual constraints of process sequence and equipment compatibility. A non-dominated sorting genetic algorithm is used to perform multi-objective optimization on candidate solutions and generate a Pareto optimal solution set for automatic selection by the system or manual intervention.

Images