A production scheduling method, system, equipment, and medium based on reinforcement learning
By using a production scheduling method based on reinforcement learning to predict and avoid resource conflicts, the problem of insufficient dynamic adaptability and difficulty in resource coordination in the mixed production of multiple satellite models is solved, achieving globally optimal production cycle control and improved resource utilization.
Patent Information
- Application Number
- CN202610487388.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2026-04-14
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2046-04-14
AI Technical Summary
Existing technologies suffer from insufficient dynamic adaptability, low resource utilization, and difficulty in coordinating flexible operational resources with rigid process resources in mixed-line production of multiple satellite models, resulting in low production continuity and efficiency.
A production scheduling method based on reinforcement learning is adopted. By training the production scheduling model to predict resource conflicts and select target scheduling actions, the global optimal cycle control is achieved. This includes state prediction and reinforcement learning decision-making modules, which are combined with real-time data for online iterative optimization.
It achieves globally optimal production cycle control, improves resource utilization and production efficiency, eliminates dynamic bottlenecks, ensures stable production cycle for multiple satellite models, and shortens the manufacturing cycle.
Smart Images

Figure CN122284545B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of industrial automation technology, and in particular to a production scheduling method, system, equipment and medium based on reinforcement learning. Background Technology
[0002] Currently, in fields such as spacecraft, aviation, and high-precision electronic products, manufacturing models are rapidly iterating from traditional single-piece customized production to mixed-line production models with multiple varieties and variable batches. For example, in mixed-line production scenarios involving multiple satellite models, the assembly and testing processes for multiple different satellite models proceed in parallel. This production process has strong dynamic and uncertain characteristics. Among these, production scheduling, as the core link in mixed-line production management, is the key to ensuring the continuous and stable operation of mixed-line production and improving the overall production cycle time.
[0003] In existing technologies, satellite mixed-line production and similar discrete manufacturing mixed-line scenarios often employ rule-based traditional scheduling methods, such as first-in-first-out, shortest-time-first, and equipment load balancing. However, in the complex production scenarios of multi-model satellite mixed-line production, traditional scheduling schemes have several technical shortcomings, including: First, traditional scheduling schemes lack dynamic adaptability and cannot adapt to the differences in satellite mixed-line production; second, traditional scheduling schemes can only achieve local optima based on the current state, easily leading to unnecessary long-term idleness of high-value flexible operational resources; third, under traditional scheduling schemes, coordination between flexible operational resources (such as multi-axis robots) and rigid process resources (such as warehousing and material distribution) is difficult, making it impossible to balance their timing requirements. Summary of the Invention
[0004] In view of this, the purpose of this application is to provide a production scheduling method, system, equipment and medium based on reinforcement learning. The production scheduling model obtained by reinforcement learning training can predict and avoid future resource conflicts on the production line, achieve globally optimal cycle control, and improve resource utilization and production efficiency.
[0005] This application provides a reinforcement learning-based production scheduling method applied to a satellite mixed-line production scenario; the satellite mixed-line production scenario includes multi-axis robots, material delivery equipment, and automated warehouses; the method includes: Collect real-time status data of the satellite mixed-line production scenario; The real-time status data is input into a pre-trained production scheduling model to predict resource demand conflicts in the satellite mixed-line production scenario, and target scheduling actions are selected based on the predicted conflict situation; wherein, the production scheduling model is trained based on reinforcement learning. The target scheduling action is converted into industrial control instructions and sent to the underlying execution system of the satellite mixed production scenario to schedule the production of the corresponding equipment in the satellite mixed production scenario.
[0006] Furthermore, the method also includes: Monitor the production line operation data after the target scheduling action is executed, and perform online iterative optimization of the production scheduling model based on the production line operation data.
[0007] Furthermore, the production scheduling model includes a state prediction submodule and a reinforcement learning decision-making submodule; the real-time state data is input into the pre-trained production scheduling model to predict the conflict of production line resource demands in the satellite mixed-line production scenario, and the target scheduling action is selected based on the predicted conflict situation, including: The real-time status data is input into the status prediction submodule, which then predicts the demand data of different types of satellites under construction for target production line resources in the satellite mixed production scenario within a future predetermined time window. Determine whether there are any conflicts between the demand data of the different types of satellites under construction and / or between the demand data of the different types of satellites under construction and the production line data of the mixed-line production scenario of satellites; If a conflict exists, an enhanced feature representation is generated based on the conflict situation and input into the reinforcement learning decision submodule. The reinforcement learning decision submodule then selects the scheduling action with the highest probability as the target scheduling action based on the probability distribution of each scheduling action in the action space output by the policy network.
[0008] Furthermore, the method also includes: when training the production scheduling model, importing the process specifications of the satellite to be produced, the constraints of each piece of equipment on the production line, and the initial storage location status of the automated warehouse to construct the basic operating environment of the production scheduling model; The state space is defined to include at least one of the following: the multi-axis robot, the material delivery equipment, the automated warehouse, and the production status data of the satellite under construction; The defined action space includes at least one of the following: the task adjustment action of the multi-axis robot, the material inbound and outbound control action of the automated warehouse, the delivery control action of the material delivery equipment, the position adjustment action of the in-process satellite, and the standby filling action of the multi-axis robot. The production line cycle time is defined as a reward function based on the aforementioned satellite mixed-line production scenario.
[0009] Furthermore, based on the satellite mixed-line production scenario, the production line cycle time is defined as a reward function, including: The stability of the production line cycle time is taken as the first positive reward item; The average utilization rate and / or on-time delivery rate of the multi-axis robot will be used as the second positive reward. The maximum queue length at the entrance and exit of the automated warehouse, the idle waiting time of the multi-axis robot, and the number of times the material delivery equipment times out are used as negative penalty items. The reward function is defined based on the first positive reward term, the second positive reward term, and the negative penalty term.
[0010] Furthermore, the production scheduling model is iteratively optimized online based on the production line operation data, including: The actual reward value for the current step is determined based on the production line operation data. The advantage function value is determined based on the actual reward value of the current step; When the advantage function value triggers the positive reward condition, the policy gradient is calculated based on the advantage function value, and the policy network parameters of the production scheduling model are updated using the gradient ascent method to increase the probability that the production scheduling model outputs the target scheduling action under the same conditions. When the advantage function value triggers a negative reward condition, the policy gradient is calculated based on the advantage function value, and the policy network parameters of the production scheduling model are corrected using the gradient descent method to reduce the probability that the production scheduling model outputs the target scheduling action under the same conditions.
[0011] Furthermore, based on the predicted conflict situation, target scheduling actions are selected, including: If it is predicted that at least two types of high-priority target satellites under construction will use the same position on the production line within a future predetermined time window, then the scheduling strategy is determined to increase the processing speed of the pre-processing steps of the target satellites under construction, and the target scheduling action is selected according to the scheduling strategy to increase the priority of the multi-axis robot for the pre-processing steps of the target satellites under construction and / or move other satellites under construction at that position on the pre-processing steps to the buffer area. If it is predicted that the target multi-axis robot will wait for the target material after a preset time, the scheduling strategy is determined to advance the arrival time of the target material, and the target scheduling action is selected according to the scheduling strategy to increase the outbound priority of the target material in the automated warehouse and / or increase the delivery priority of the material delivery equipment.
[0012] This application also provides a reinforcement learning-based production scheduling system applied to a satellite mixed-line production scenario; the satellite mixed-line production scenario includes multi-axis robots, material delivery equipment, and automated warehouses; the production scheduling system includes: The environmental perception layer is used to collect real-time status data of the satellite mixed-line production scenario; The action decision layer is used to input the real-time status data into a pre-trained production scheduling model, predict resource demand conflicts in the satellite mixed-line production scenario, and select target scheduling actions based on the predicted conflict situation; wherein, the production scheduling model is trained based on reinforcement learning. The execution layer is used to convert the target scheduling action into industrial control instructions and send them to the underlying execution system of the satellite mixed production scenario to perform production scheduling on the corresponding equipment in the satellite mixed production scenario.
[0013] This application embodiment also provides an electronic device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the steps of the production scheduling method based on reinforcement learning described above are performed.
[0014] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the reinforcement learning-based production scheduling method described above.
[0015] This application provides a production scheduling method, system, device, and medium based on reinforcement learning. The production scheduling model obtained through reinforcement learning training predicts resource demand conflicts in satellite mixed production scenarios. Based on the predicted conflict situation, the target scheduling action is selected and executed to avoid future resource conflicts on the production line, thereby achieving globally optimal cycle control and improving resource utilization and production efficiency.
[0016] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description
[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0018] Figure 1 A flowchart of a production scheduling method based on reinforcement learning provided in an embodiment of this application is shown; Figure 2 This paper illustrates a framework diagram of a production scheduling system based on reinforcement learning, provided in an embodiment of this application. Figure 3 This illustration shows a schematic diagram of the structure of a production scheduling system based on reinforcement learning provided in an embodiment of this application; Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation
[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. Based on the embodiments of this application, every other embodiment obtained by those skilled in the art without inventive effort falls within the scope of protection of this application.
[0020] Research has revealed that existing technologies for mixed-line production of satellites and similar discrete manufacturing often employ rule-based traditional scheduling methods, such as first-in-first-out (FIFO), shortest-time-first (SLT) scheduling, and equipment load balancing. However, in the complex production scenarios of mixed-line production of multiple satellite models, traditional scheduling schemes suffer from several technical shortcomings, including: First, the scheduling scheme lacks dynamic adaptability and cannot adapt to the dynamic changes in satellite mixed-line production. In the process of satellite mixed-line production, there are huge differences in robot task time, material delivery time, and process paths of different satellite models. The scheduling method based on fixed rules can only adapt to the preset steady-state production scenario and cannot respond to the dynamic parameter fluctuations and sudden working conditions in the production process. This can easily lead to resource competition conflicts and local logistics problems, which seriously disrupt the continuity of production.
[0021] Secondly, scheduling schemes can only achieve local optima and cannot achieve global takt time optimization for the entire production line. Existing rule-based scheduling methods make scheduling decisions based solely on the current state data of the production line, lacking the ability to predict takt time and resource conflicts throughout the entire production cycle. Under this model, scheduling decisions can only meet the local operational needs of the current moment and cannot comprehensively consider the rational allocation of all resources throughout the entire production cycle. Especially for high-value multi-axis robot resources, short-sighted task allocation can easily lead to problems such as unnecessary long periods of robot idleness.
[0022] Third, it is impossible to achieve efficient collaboration between flexible operational resources and rigid process resources. In satellite mixed-line production lines, flexible operational units such as multi-axis robots possess high flexibility, with switchable processes and dynamically adjustable work paths. However, rigid process units such as automated storage and retrieval systems (AS / RS) and AGV material delivery systems have strong fixed constraints due to the limitations of their own logic regarding operation timing and inbound / outbound cycles. The operational characteristics and constraints of the two are fundamentally different. Existing rule-based scheduling methods cannot simultaneously consider the dynamic adjustment space of flexible resources and the fixed timing constraints of rigid resources, making it difficult to achieve precise matching of their operation timings and ensuring the smooth and orderly operation of mixed-line production.
[0023] Based on this, this application provides a production scheduling method based on reinforcement learning. The production scheduling model obtained through reinforcement learning training can predict and avoid future resource conflicts on the production line, achieve globally optimal production cycle control, and improve resource utilization and production efficiency.
[0024] Please see Figure 1 , Figure 1 This is a flowchart illustrating a reinforcement learning-based production scheduling method provided in an embodiment of this application. The production scheduling method provided in this embodiment is applied to a satellite mixed-line production scenario, in which different models of satellites under manufacturing exhibit significant differences in process path length, process logic, single-step time, material requirements, specifications, and cycles. In specific implementation, the satellite mixed-line production scenario includes multi-axis robots (such as seven-axis robots), material delivery equipment (such as automated guided vehicles, AGVs), and automated warehouses (such as Automated Storage and Retrieval Systems).
[0025] like Figure 1 As shown in the embodiments of this application, the production scheduling method includes: S101. Collect real-time status data of the satellite mixed-line production scenario.
[0026] In this step, real-time status data can be collected by sensing devices installed in the satellite mixed-line production scenario and obtained in real time through the industrial internet interface; for example, the real-time status data includes at least one of the following: production status data of each multi-axis robot, each material delivery equipment, automated warehouse and satellite in production.
[0027] S102. Input the real-time status data into the pre-trained production scheduling model to predict the conflict of resource demand in the satellite mixed production scenario, and select the target scheduling action according to the predicted conflict.
[0028] In this embodiment, the production scheduling model is trained using reinforcement learning and includes a state prediction submodule and a reinforcement learning decision submodule. In one possible implementation, step S102 may include: S1021. Input the real-time status data into the status prediction submodule, and the status prediction submodule predicts the demand data of different types of satellites under construction for target production line resources in the satellite mixed production scenario within a future predetermined time window.
[0029] In this step, the state prediction submodule can be a time-series prediction model. By running this submodule, the demand data for target production line resources of different types of satellites under construction within a predetermined time window (e.g., T time units) can be predicted based on real-time state data. The target production line resources include key resources affecting the overall production efficiency and unit product manufacturing cost, such as seven-axis robots and automated warehouse entrances / exits. Demand data can include peak demand and demand periods.
[0030] In practical implementation, the status prediction submodule can extract the temporal characteristics of real-time status data, thereby predicting the peak resource demand and occupancy sequence within a predetermined time window in the future, and obtaining the demand data for target production line resources.
[0031] S1022. Determine whether there is a conflict between the demand data of the different types of satellites under construction and / or between the demand data of the different types of satellites under construction and the production line data of the mixed-line production scenario of satellites.
[0032] In this step, the conflict situations include conflicts between demand data of different types of satellites under construction, such as multiple different types of satellites under construction competing for the same resource; the conflict situations also include conflicts between demand data of different types of satellites under construction and production line data in satellite mixed production scenarios, such as the production line resources required for the multi-axis robot corresponding to a certain type of satellite under construction not being ready, resulting in high-value multi-axis robots waiting idly.
[0033] S1023. If a conflict exists, an enhanced feature representation is generated based on the conflict situation and the enhanced feature representation is input into the reinforcement learning decision submodule. The reinforcement learning decision submodule selects the scheduling action with the highest probability as the target scheduling action based on the probability distribution of each scheduling action in the action space output by the policy network.
[0034] In this step, the production scheduling model learns action selection strategies during reinforcement learning training, so in the inference phase, it can select target scheduling actions from the constructed action space based on the reinforced state representation that includes conflict situations.
[0035] In practical implementation, the state prediction submodule, as a time-series prediction model, can be trained using conventional time-series prediction training methods, such as supervised learning. The reinforcement learning decision submodule can be trained using reinforcement learning methods. Therefore, the steps for training the production scheduling model include: Step 1: Import the process specifications of the satellite to be produced, the constraints of each piece of equipment on the production line, and the initial storage location status of the automated warehouse to construct the basic operating environment of the production scheduling model. For example, the satellite process specifications include BOM / SOP, and the equipment constraints include kinematic limitations of the seven-axis robot.
[0036] Step 2: Define the state space to include at least one of the following: the multi-axis robot, the material delivery equipment, the automated warehouse, and the production status data of the satellite in production.
[0037] For example, production status data includes the current task ID, remaining working hours, and standby status of all seven-axis robots; the length of the entrance and exit queues of the automated warehouse; the estimated arrival time (ETA) of the material delivery equipment AGV; and the model, priority, and completed processes of all currently in-process (WIP) satellites.
[0038] Step 3: Define the motion space including at least one of the following: the task adjustment action of the multi-axis robot (such as adjusting the current task priority of a robot), the material inbound / outbound control action of the automated warehouse (such as instructing the automated warehouse to release materials earlier or later), the delivery control action of the material delivery equipment (such as instructing the material delivery equipment to deliver materials earlier or later), the position adjustment action of the work-in-process satellite (such as temporarily moving a satellite workpiece to a buffer area, or moving it from the buffer area to the production line), and the standby filling action of the multi-axis robot (such as inserting a preset non-critical task during the robot's waiting period, such as self-inspection, fixture calibration, etc.).
[0039] Step 4: Define a reward function for the production line cycle time based on the satellite mixed-line production scenario.
[0040] In this step, the reward function (RewardR) is an indicator that comprehensively evaluates the effectiveness of scheduling decisions. The reward function can be defined based solely on the production line cycle time in the satellite mixed production scenario, or it can be defined based on the overall cycle time stability (low variance) of the production line.
[0041] Alternatively, in another possible implementation, a reward function is defined based on the production line cycle time of the satellite mixed-line production scenario, including: The stability of the production line cycle time is used as the first positive reward; the average utilization rate and / or on-time delivery rate of the multi-axis robot are used as the second positive reward; at least one of the following is used as the negative penalty: the maximum queue length at the entrance and exit of the automated warehouse, the idle waiting time of the multi-axis robot, and the number of material delivery timeouts of the material delivery equipment; the reward function is defined based on the first positive reward, the second positive reward, and the negative penalty.
[0042] The first positive reward, the second positive reward, and the negative penalty can be summed to obtain the reward function, and the weights of each item can be adjusted according to the actual production situation.
[0043] Thus, based on the state space, action space, and reward function, the initial model can be iteratively trained using reinforcement learning methods until it converges, resulting in a well-trained production scheduling model. The specific reinforcement learning training methods can refer to existing techniques, and this application does not impose any limitations.
[0044] In one example, the target scheduling action is selected based on the predicted conflict situation, including: Scenario A: If it is predicted that at least two types of high-priority target satellites under construction will use the same position on the production line within a future predetermined time window, then the scheduling strategy is determined to increase the processing speed of the pre-processing steps of the target satellites under construction, and the target scheduling action is selected according to the scheduling strategy to increase the priority of the multi-axis robot for the pre-processing steps of the target satellites under construction and / or move other satellites under construction at that position on the pre-processing steps to the buffer area.
[0045] For example, if it is predicted that two high-priority satellites will compete for the same test station within the next 30 minutes, the pre-assembly speed of one of the satellites will be adjusted in advance to stagger the test times.
[0046] Scenario B: If it is predicted that the target multi-axis robot will wait for the target material after a preset time, the scheduling strategy is determined to advance the arrival time of the target material, and the target scheduling action is selected according to the scheduling strategy to increase the outbound priority of the target material in the automated warehouse and / or increase the delivery priority of the material delivery equipment.
[0047] For example, if it is predicted that robot A will be waiting for material X at time T+5 within the next 30 minutes, it will immediately instruct the automated warehouse to raise the outbound priority of material X to the highest level and / or instruct the material delivery equipment to raise the delivery priority of material X to the highest level.
[0048] S103. The target scheduling action is converted into an industrial control command and sent to the underlying execution system of the satellite mixed production scenario to schedule the production of the corresponding equipment in the satellite mixed production scenario.
[0049] In this step, the target scheduling action can be converted into industrial control instructions, which are then sent to the MES / WCS system. The MES / WCS system can then be used to schedule the production of the corresponding equipment in the satellite mixed-line production scenario.
[0050] Furthermore, the production scheduling method provided in this application embodiment also includes: monitoring the production line operation data after the target scheduling action is executed, and performing online iterative optimization of the production scheduling model based on the production line operation data.
[0051] Among them, the fluctuation of production cycle time (such as cycle time variance) and the utilization rate of multi-axis robots can be determined based on the production line operation data, and then the production scheduling model can be iteratively optimized online.
[0052] In practice, the actual reward value of the reward function in the current step can be determined based on the production line operation data; then, the advantage function value (or timing difference error) can be determined based on the actual reward value in the current step, such as by combining the actual reward value and the state value evaluated by the value network to calculate the advantage function value.
[0053] When the advantage function value triggers a positive reward condition, such as when the advantage function value is greater than zero, it indicates that the actual benefit of the current target scheduling action is better than the model's historical expectations (i.e., it successfully avoids conflicts and improves the pace). In this case, the policy gradient is calculated and the policy network parameters are updated using the gradient ascent method to increase the probability that the production scheduling model outputs the target scheduling action under the same conditions. When the reward value triggers a negative reward condition, such as when the advantage function value is less than or equal to zero, it indicates that the actual benefit of the current target scheduling action is less than expected (e.g., it leads to new conflicts or equipment waiting). In this case, the error is backpropagated and the policy network parameters are corrected using the gradient descent method to reduce the probability that the production scheduling model outputs the target scheduling action under the same conditions.
[0054] The production scheduling method provided in this application can be implemented based on a production scheduling system. Please refer to... Figure 2 , Figure 2 This is a framework diagram of a production scheduling system based on reinforcement learning, provided as an embodiment of this application. Figure 2 As shown, the intelligent scheduling system is divided into a high-level planning layer and a low-level execution layer, which are connected and interact through a digital twin environment.
[0055] At the high-level planning level, satellite process data (BOM / SOP) and production priority constraints for each type of satellite are imported to provide basic process rules and production targets for scheduling. The core optimization targets for scheduling are clarified, including production line throughput, production cycle stability, utilization rate of high-value seven-axis robots, and on-time delivery rate, which serve as optimization directions for the reinforcement learning scheduling agent. The reinforcement learning scheduling agent completes scheduling decisions through a closed loop of perception-decision-execution-learning.
[0056] The process involves several key components: Perception: Real-time status feedback from the underlying execution layer collects comprehensive data on the seven-axis robot status, Automated Guided Vehicle (AGV) material status, and work-in-process (WIP) satellite production status to construct the current production state space. Decision-making: Based on the current state, a policy network, combined with a digital twin environment, simulates future production trends within a time window, predicting potential resource conflicts (such as seven-axis robot idleness, AGV queue congestion, and workstation competition). From a pre-defined action space (adjusting robot task priorities, regulating material outbound processes, caching WIP, inserting standby tasks, etc.), the optimal scheduling intervention action that maximizes the global optimization objective is output. Execution: The optimal scheduling action is converted into standard control commands and sent to the PLC / WCS system at the underlying execution layer. Learning: Real-world production line operation data after the execution of scheduling commands is collected. Based on a pre-defined reward function, the actual reward value is calculated. The scheduling strategy is iteratively optimized and updated online to continuously improve decision-making effectiveness.
[0057] In a digital twin environment, it is a complete mirror image of the physical production line, receiving scheduling instructions from the high-level planning layer and real-time status feedback from the low-level execution layer.
[0058] The underlying execution layer includes a seven-axis robot control unit and an automated warehousing / material handling unit, which are responsible for command execution and status feedback.
[0059] For the seven-axis robot control unit: PLC / WCS control layer: receives scheduling instructions from the higher-level planning layer and parses them into control signals that the robot can execute; Robot task queue: adjusts task priorities and inserts standby tasks (such as self-test and fixture calibration) according to scheduling instructions, and manages the operation sequence of the seven-axis robot; Tooling and fixture management: coordinates with robot tasks to complete the scheduling and switching of tooling and fixtures; Work-in-process satellite buffer: executes the scheduling action of "temporarily moving satellite workpieces to the buffer" to alleviate the pressure of workstation competition and material waiting; finally, it feeds back the real-time status of the seven-axis robot (task ID, remaining working hours, standby status, etc.) to the higher-level planning layer to complete the status closure loop.
[0060] For Automated Storage and Retrieval Systems (AS / RS) / AGV material handling units: PLC / WCS open queue: receives scheduling instructions, adjusts material outbound priority, and executes early / delayed outbound operations; AGV path planning and scheduling: optimizes AGV travel paths according to material delivery needs to ensure timely delivery of materials to workstations; material buffer: temporarily stores materials to be delivered, alleviating congestion at the entrance and exit queues of the automated warehouse; finally, real-time data such as automated warehouse queue length, AGV estimated arrival time (ETA), and material delivery status are fed back to the high-level planning layer.
[0061] This application provides a production scheduling method based on reinforcement learning. Through a production scheduling model trained using reinforcement learning, it predicts resource demand conflicts in satellite mixed-line production scenarios. Based on the predicted conflict situations, it selects and executes target scheduling actions to avoid future resource conflicts on the production line, thereby achieving globally optimal cycle time control, eliminating dynamic bottlenecks, ensuring stable output cycle time for multiple satellite models, and shortening the average manufacturing cycle. By inserting non-critical tasks in real time and optimizing timing, it improves resource utilization and production efficiency. Furthermore, it identifies and avoids timing mismatches between materials and robots in advance, reducing the risk of production interruptions due to material waiting or resource competition.
[0062] Please see Figure 3 , Figure 3 This is a schematic diagram of a reinforcement learning-based production scheduling system provided in an embodiment of this application. It is applied to a satellite mixed-line production scenario; the satellite mixed-line production scenario includes multi-axis robots, material delivery equipment, and an automated warehouse; as shown... Figure 3 As shown, the production scheduling system 300 includes: The environmental perception layer 310 is used to collect real-time status data of the satellite mixed-line production scenario; Action decision layer 320 is used to input the real-time status data into a pre-trained production scheduling model, predict the conflict of resource demand in the satellite mixed production scenario, and select the target scheduling action according to the predicted conflict; wherein, the production scheduling model is trained based on reinforcement learning. The execution layer 330 is used to convert the target scheduling action into industrial control instructions and send them to the underlying execution system of the satellite mixed production scenario to perform production scheduling on the corresponding equipment in the satellite mixed production scenario.
[0063] Furthermore, the production scheduling system 300 also includes: a feedback optimization layer ( Figure 3 (not shown in the image); the feedback optimization layer is used for: Monitor the production line operation data after the target scheduling action is executed, and perform online iterative optimization of the production scheduling model based on the production line operation data.
[0064] Furthermore, the production scheduling model includes a state prediction submodule and a reinforcement learning decision submodule; when the action decision layer 320 inputs the real-time state data into the pre-trained production scheduling model to predict the conflict of production line resource demand in the satellite mixed-line production scenario, and selects the target scheduling action based on the predicted conflict, the action decision layer 320 is used to: The real-time status data is input into the status prediction submodule, which then predicts the demand data of different types of satellites under construction for target production line resources in the satellite mixed production scenario within a future predetermined time window. Determine whether there are any conflicts between the demand data of the different types of satellites under construction and / or between the demand data of the different types of satellites under construction and the production line data of the mixed-line production scenario of satellites; If a conflict exists, an enhanced feature representation is generated based on the conflict situation and input into the reinforcement learning decision submodule. The reinforcement learning decision submodule then selects the scheduling action with the highest probability as the target scheduling action based on the probability distribution of each scheduling action in the action space output by the policy network.
[0065] Furthermore, the production scheduling system 300 also includes: a reinforcement learning training layer; when training the production scheduling model, the process specifications of the satellite to be produced, the constraints of each piece of equipment on the production line, and the initial storage location status of the automated warehouse are imported to construct the basic operating environment of the production scheduling model; The state space is defined to include at least one of the following: the multi-axis robot, the material delivery equipment, the automated warehouse, and the production status data of the satellite under construction; The defined action space includes at least one of the following: the task adjustment action of the multi-axis robot, the material inbound and outbound control action of the automated warehouse, the delivery control action of the material delivery equipment, the position adjustment action of the in-process satellite, and the standby filling action of the multi-axis robot. The production line cycle time is defined as a reward function based on the aforementioned satellite mixed-line production scenario.
[0066] Furthermore, when the reinforcement learning training layer is used to define the reward function for the production line cycle time based on the satellite mixed-line production scenario, the reinforcement learning training layer is used for: The stability of the production line cycle time is taken as the first positive reward item; The average utilization rate and / or on-time delivery rate of the multi-axis robot will be used as the second positive reward. The maximum queue length at the entrance and exit of the automated warehouse, the idle waiting time of the multi-axis robot, and the number of times the material delivery equipment times out are used as negative penalty items. The reward function is defined based on the first positive reward term, the second positive reward term, and the negative penalty term.
[0067] Furthermore, when the feedback optimization layer is used to perform online iterative optimization of the production scheduling model based on the production line operation data, the feedback optimization layer is used to: The actual reward value for the current step is determined based on the production line operation data. The advantage function value is determined based on the actual reward value of the current step; When the advantage function value triggers the positive reward condition, the policy gradient is calculated based on the advantage function value, and the policy network parameters of the production scheduling model are updated using the gradient ascent method to increase the probability that the production scheduling model outputs the target scheduling action under the same conditions. When the advantage function value triggers a negative reward condition, the policy gradient is calculated based on the advantage function value, and the policy network parameters of the production scheduling model are corrected using the gradient descent method to reduce the probability that the production scheduling model outputs the target scheduling action under the same conditions.
[0068] Furthermore, when selecting a target scheduling action based on the predicted conflict situation, the action decision layer 320 is used to: If it is predicted that at least two types of high-priority target satellites under construction will use the same position on the production line within a future predetermined time window, then the scheduling strategy is determined to increase the processing speed of the pre-processing steps of the target satellites under construction, and the target scheduling action is selected according to the scheduling strategy to increase the priority of the multi-axis robot for the pre-processing steps of the target satellites under construction and / or move other satellites under construction at that position on the pre-processing steps to the buffer area. If it is predicted that the target multi-axis robot will wait for the target material after a preset time, the scheduling strategy is determined to advance the arrival time of the target material, and the target scheduling action is selected according to the scheduling strategy to increase the outbound priority of the target material in the automated warehouse and / or increase the delivery priority of the material delivery equipment.
[0069] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 4 As shown, the electronic device 400 includes a processor 410, a memory 420, and a bus 430.
[0070] The memory 420 stores machine-readable instructions that can be executed by the processor 410. When the electronic device 400 is running, the processor 410 and the memory 420 communicate through the bus 430. When the machine-readable instructions are executed by the processor 410, the steps of the production scheduling method based on reinforcement learning in the above method embodiment can be executed. For specific implementation, please refer to the method embodiment, which will not be repeated here.
[0071] This application also provides a computer-readable storage medium storing a computer program. When the computer program is run by a processor, it can execute the steps of the production scheduling method based on reinforcement learning as described in the above method embodiments. For specific implementation details, please refer to the method embodiments, which will not be repeated here.
[0072] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0073] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.
[0074] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0075] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0076] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0077] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The scope of protection of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A production scheduling method based on reinforcement learning, characterized in that, The method is applied to a satellite mixed-line production scenario, which includes multi-axis robots, material handling equipment, and automated warehouses. Collect real-time status data of the satellite mixed-line production scenario; The real-time status data is input into a pre-trained production scheduling model to predict resource demand conflicts in the satellite mixed-line production scenario, and target scheduling actions are selected based on the predicted conflict situation; wherein, the production scheduling model is trained based on reinforcement learning. The target scheduling action is converted into industrial control instructions and sent to the underlying execution system of the satellite mixed production scenario to schedule the production of the corresponding equipment in the satellite mixed production scenario. The production scheduling model includes a state prediction submodule and a reinforcement learning decision-making submodule. The real-time state data is input into the pre-trained production scheduling model to predict conflicts in production line resource demands during the satellite mixed-line production scenario, and target scheduling actions are selected based on the predicted conflict situations, including: The real-time status data is input into the status prediction submodule, which then predicts the demand data of different types of satellites under construction for target production line resources in the mixed-line production scenario of satellites within a predetermined future time window. Determine whether there are any conflicts between the demand data of the different types of satellites under construction and / or between the demand data of the different types of satellites under construction and the production line data of the mixed-line production scenario of satellites; If a conflict exists, an enhanced feature representation is generated based on the conflict situation and input into the reinforcement learning decision submodule. The reinforcement learning decision submodule then selects the scheduling action with the highest probability as the target scheduling action based on the probability distribution of each scheduling action in the action space output by the policy network. Select target scheduling actions based on the predicted conflict situation, including: If it is predicted that at least two types of high-priority target satellites under construction will use the same position on the production line within a future predetermined time window, then the scheduling strategy is determined to increase the processing speed of the pre-processing steps of the target satellites under construction, and the target scheduling action is selected according to the scheduling strategy to increase the priority of the multi-axis robot for the pre-processing steps of the target satellites under construction and / or move other satellites under construction at that position on the pre-processing steps to the buffer area. If it is predicted that the target multi-axis robot will wait for the target material after a preset time, the scheduling strategy is determined to advance the arrival time of the target material, and the target scheduling action is selected according to the scheduling strategy to increase the outbound priority of the target material in the automated warehouse and / or increase the delivery priority of the material delivery equipment.
2. The method according to claim 1, characterized in that, The method further includes: Monitor the production line operation data after the target scheduling action is executed, and perform online iterative optimization of the production scheduling model based on the production line operation data.
3. The method according to claim 1, characterized in that, The method further includes: when training the production scheduling model, importing the process specifications of the satellite to be produced, the constraints of each piece of equipment on the production line, and the initial storage location status of the automated warehouse to construct the basic operating environment of the production scheduling model; The state space is defined to include at least one of the following: the multi-axis robot, the material delivery equipment, the automated warehouse, and the production status data of the satellite under construction; The defined action space includes at least one of the following: the task adjustment action of the multi-axis robot, the material inbound and outbound control action of the automated warehouse, the delivery control action of the material delivery equipment, the position adjustment action of the in-process satellite, and the standby filling action of the multi-axis robot. The production line cycle time is defined as a reward function based on the aforementioned satellite mixed-line production scenario.
4. The method according to claim 3, characterized in that, The production line cycle time definition reward function based on the aforementioned satellite mixed-line production scenario includes: The stability of the production line cycle time is taken as the first positive reward item; The average utilization rate and / or on-time delivery rate of the multi-axis robot will be used as the second positive reward. The maximum queue length at the entrance and exit of the automated warehouse, the idle waiting time of the multi-axis robot, and the number of times the material delivery equipment times out are used as negative penalty items. The reward function is defined based on the first positive reward term, the second positive reward term, and the negative penalty term.
5. The method according to claim 2, characterized in that, The production scheduling model is iteratively optimized online based on the production line operation data, including: The actual reward value for the current step is determined based on the production line operation data. The advantage function value is determined based on the actual reward value of the current step; When the advantage function value triggers the positive reward condition, the policy gradient is calculated based on the advantage function value, and the policy network parameters of the production scheduling model are updated using the gradient ascent method to increase the probability that the production scheduling model outputs the target scheduling action under the same conditions. When the advantage function value triggers a negative reward condition, the policy gradient is calculated based on the advantage function value, and the policy network parameters of the production scheduling model are corrected using the gradient descent method to reduce the probability that the production scheduling model outputs the target scheduling action under the same conditions.
6. A production scheduling system based on reinforcement learning, characterized in that, This system is applied to satellite mixed-line production scenarios; these scenarios include multi-axis robots, material handling equipment, and automated warehouses; the production scheduling system includes: The environmental perception layer is used to collect real-time status data of the satellite mixed-line production scenario; The action decision layer is used to input the real-time status data into a pre-trained production scheduling model, predict resource demand conflicts in the satellite mixed-line production scenario, and select target scheduling actions based on the predicted conflict situation; wherein, the production scheduling model is trained based on reinforcement learning. The execution layer is used to convert the target scheduling action into industrial control instructions and send them to the underlying execution system of the satellite mixed production scenario to perform production scheduling on the corresponding equipment in the satellite mixed production scenario. The production scheduling model includes a state prediction submodule and a reinforcement learning decision-making submodule. When the action decision layer inputs the real-time state data into the pre-trained production scheduling model to predict conflicts in production line resource demands during the satellite mixed-line production scenario, and selects a target scheduling action based on the predicted conflict situation, the action decision layer is used for: The real-time status data is input into the status prediction submodule, which then predicts the demand data of different types of satellites under construction for target production line resources in the mixed-line production scenario of satellites within a predetermined future time window. Determine whether there are any conflicts between the demand data of the different types of satellites under construction and / or between the demand data of the different types of satellites under construction and the production line data of the mixed-line production scenario of satellites; If a conflict exists, an enhanced feature representation is generated based on the conflict situation and input into the reinforcement learning decision submodule. The reinforcement learning decision submodule then selects the scheduling action with the highest probability as the target scheduling action based on the probability distribution of each scheduling action in the action space output by the policy network. When selecting a target scheduling action based on predicted conflict conditions, the action decision layer is used to: If it is predicted that at least two types of high-priority target satellites under construction will use the same position on the production line within a future predetermined time window, then the scheduling strategy is determined to increase the processing speed of the pre-processing steps of the target satellites under construction, and the target scheduling action is selected according to the scheduling strategy to increase the priority of the multi-axis robot for the pre-processing steps of the target satellites under construction and / or move other satellites under construction at that position on the pre-processing steps to the buffer area. If it is predicted that the target multi-axis robot will wait for the target material after a preset time, the scheduling strategy is determined to advance the arrival time of the target material, and the target scheduling action is selected according to the scheduling strategy to increase the outbound priority of the target material in the automated warehouse and / or increase the delivery priority of the material delivery equipment.
7. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. The machine-readable instructions are executed by the processor to perform the steps of the reinforcement learning-based production scheduling method as described in any one of claims 1 to 5.
8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the production scheduling method based on reinforcement learning as described in any one of claims 1 to 5.
Citation Information
Patent Citations
Workshop dynamic bottleneck prediction scheduling optimization method based on graph neural network
CN116739155A
