Tobacco production scheduling optimization method and system based on reinforcement learning

Abstract

The invention discloses a reinforcement learning-based tobacco production scheduling optimization method and system. The method comprises the steps of constructing a multi-dimensional value vector and a sparse vectorization Q table combined with process constraints; collecting production data in real time, and calculating a reward vector coupled with multi-target information; updating the Q table by using a vector Bellman equation; and screening out an available action set based on process constraints, calculating a vector sum of a value vector corresponding to each action, selecting and executing an optimal action based on the vector sum, collecting feedback and implementing loop optimization. According to the method, a multi-dimensional vector is used for replacing a traditional single scalar to represent a state-action value, and a coupling reward function is designed, so that the system can automatically learn the tradeoff among multiple targets, the dependence on a fixed artificial weight is abandoned, and a sparse Q table structure ensures the process compliance of decision making. The technical problems that manual parameter adjustment cost is high, multi-target information is lost and a static strategy cannot adapt to a dynamic environment in production scheduling of traditional reinforcement learning are effectively solved.

Description

A Reinforcement Learning-Based Method and System for Optimizing Tobacco Production Scheduling Technical Field

[0001] This invention belongs to the field of intelligent manufacturing technology, specifically, it relates to a method and system for optimizing tobacco production scheduling based on reinforcement learning. Background Technology

[0002] In intelligent manufacturing, the production scheduling system is the core of ensuring production efficiency, cost control, and product quality. Q-Learning algorithms, due to their model independence and online learning capabilities, have become a common solution.

[0003] Typical existing technical solutions typically adopt the system structure shown in Figure 1: the system collects production environment status information in real time, calculates a single-value reward through a reward function with manually adjusted weight parameters, updates the single scalar Q value stored in the Q table, and finally makes scheduling decisions by comparing the Q values ​​of different actions.

[0004] However, traditional Q-Learning production scheduling schemes, due to their scalar reward design, static human weighting, and single value representation, reveal the following significant shortcomings when facing complex, dynamic, and multi-objective modern production environments:

[0005] (1) The reward function of the existing system is as follows: In the form of, , , These represent key indicators such as time, cost, and loss. , , The weighting coefficients are manually set, corresponding to the relative importance of time, cost, and loss. Because the weights are fixed and rely on the experience of domain experts, the system cannot adjust the weighting parameters in real time when situations such as equipment aging or emergency orders occur. Optimization can take as long as 2 to 4 weeks, during which time the system remains in a suboptimal state for an extended period, resulting in continuous production losses.

[0006] (2) The multi-dimensional production goals are compressed into a single scalar reward Q[s][a] through linear weighting, resulting in the complete loss of independent information and trade-offs between the dimensions, making it impossible to distinguish the essential differences such as "fast but expensive" and "slow but economical". Summary of the Invention

[0007] The purpose of this invention is to propose a tobacco production scheduling optimization method and system based on reinforcement learning. This method maintains the integrity of multi-objective information such as time, load, and energy consumption through a three-dimensional vector Q; it constructs a sparse Q-table by combining process constraints to fundamentally avoid process violation decisions; the three-dimensional reward vector is calculated in real time according to the production status, adapting to environmental changes; the current state-action value vector is updated using a vector-form Bellman update equation, with the learning rate and discount factor automatically calculated without manual weight setting; and the optimal action is selected using vectors and the maximum strategy, achieving intelligent trade-offs among multiple objectives.

[0008] The present invention is implemented using the following technical solutions:

[0009] A reinforcement learning-based method for optimizing tobacco production scheduling is proposed, including:

[0010] Initialize the state space S, action space A, and vectorized Q table of the tobacco production system; wherein, the state space S is defined based on the combination of workpiece number and process number; the action space A is defined based on the set of available production equipment; the vectorized Q table is composed of value vectors corresponding to state-action, each value vector includes multi-dimensional components, and each dimension component corresponds to a type of production data;

[0011] Real-time collection of production data and calculation of reward vector; the reward vector is a multi-dimensional vector with the same dimension as the value vector, wherein the calculation of at least one component is coupled with information from other components;

[0012] The vectorized Q-table is updated using the improved Bellman equation;

[0013] Based on process constraints, a set of available actions is selected, and the sum of the value vectors corresponding to each action in the set of available actions is calculated. The optimal action is selected based on the maximum sum of the vectors.

[0014] Execute the optimal action and collect actual production data to calculate the actual reward vector in the new state, thus completing the state transition; feed back the actual reward vector and the new state information to the system.

[0015] In some embodiments of the present invention, the vectorized Q-table is constructed as a sparse structure based on manufacturing process constraints, and each state is associated only with the available actions allowed by the process.

[0016] In some embodiments of the present invention, the reward vector includes a time component, a load component, and a power consumption component; wherein, the time component is the reciprocal of the expected completion time T coupled with the device operating power P, and the load component is the reciprocal of the total load rate CT.

[0017] In some embodiments of the present invention, the improved Bellman equation is as follows:

[0018] ;in, For state Select action The value vector, For the reward vector, For learning rate, As a discount factor, For the next state The value vector of the next action selection. A set of available actions; Number the components of the vector. Let be the dimension of the vector.

[0019] In some embodiments of the present invention, the learning rate Calculated based on the adaptive decay strategy, including:

[0020] Automatically calculate the learning effect of the most recent n iterations;

[0021] Calculate the rate of change of Q. ;

[0022] Calculate the optimal learning rate Among them, setting boundary constraints: ensuring ;

[0023] Every The calculation is automatically recalculated in the next iteration.

[0024] In some embodiments of the present invention, the discount factor Determined based on average production cycle and product value, including:

[0025] Automatically calculate average production cycle ;

[0026] Automatically identify the product type based on the current work order;

[0027] Calculate the optimal discount rate , The product value coefficient; where, the intelligence boundary is defined: high-value products Ordinary products ;make sure .

[0028] In some embodiments of the present invention, the method further includes:

[0029] Real-time monitoring of changes in the production environment; when the reward component of a key performance indicator exceeds a threshold, the value vector of the relevant state-action pair is adjusted; the reward vector is calculated based on the adjusted value vector; and the optimal action is selected based on the new reward vector.

[0030] In some embodiments of the present invention, the adjustment of the value vector includes:

[0031] Adjust the positive / negative proportions for components that exceed the threshold.

[0032] A tobacco production scheduling system based on reinforcement learning is proposed, including:

[0033] The initialization module is used to initialize the state space S, action space A, and vectorized Q table of the tobacco production system. The state space S is defined based on the combination of workpiece number and process number; the action space A is defined based on the set of available production equipment; the vectorized Q table consists of value vectors corresponding to state and action, and each value vector includes multi-dimensional components, with each component corresponding to a type of production data.

[0034] The data acquisition and reward calculation module is used to collect production data in real time and calculate the reward vector; the reward vector is a multi-dimensional vector with the same dimension as the value vector, wherein the calculation of at least one component is coupled with the information of other components;

[0035] The Q-table update module is used to update the vectorized Q-table using the improved Bellman equation;

[0036] The decision-making module is used to filter out the set of available actions based on process constraints, calculate the vector sum of the value vectors corresponding to each action in the set of available actions, and select the optimal action based on the maximum vector sum.

[0037] The execution and feedback module is used to execute the optimal action, collect actual production data to calculate the actual reward vector in the new state, and complete the state transition; it feeds back the actual reward vector and the new state information to the system.

[0038] In some embodiments of the present invention, the system further includes:

[0039] The adaptive adjustment module is used to monitor changes in the production environment in real time. When it detects that the reward component of a key performance indicator exceeds the threshold, it triggers the adjustment of the value vector of the relevant state-action pair, calculates the reward vector based on the adjusted value vector, and selects the optimal action based on the new reward vector.

[0040] Compared with the prior art, the advantages and positive effects of the present invention are:

[0041] (1) This invention optimizes the traditional single scalar Q-value into a three-dimensional vector structure, maintaining the integrity of multi-objective optimization information such as time, load, and energy consumption. By designing a reward vector that couples multi-objective information and a vectorized value representation, the system can automatically grasp the trade-off relationship between multiple objectives such as time, load, and energy consumption during the learning process. This eliminates the reliance on experts for fixed manual weight tuning, transforms manual parameter tuning into an online adaptive learning process, and shortens the production cycle.

[0042] (2) When making decisions, the selection is based on the maximum sum of the three-dimensional vectors, so that the system can clearly know the specific differences in the speed of time, load and energy consumption of different actions, so as to find the optimal balance between time, load and energy consumption when multiple objectives conflict, rather than a simple single value comparison.

[0043] (3) A sparse Q-value table is constructed by combining process constraints. Each state is only associated with the actions allowed by the process, which fundamentally eliminates the possibility of selecting non-compliant equipment.

[0044] (4) Through real-time calculation of reward vector and adaptive update of value vector, adjustments can be automatically triggered when equipment performance deteriorates, orders fluctuate or emergency orders are inserted, thus achieving a technological upgrade from passive response to proactive prevention.

[0045] (5) Extend the traditional scalar Bellman equation into a vector form to ensure the independence and integrity of three-dimensional information during the update process, and achieve the goal of meeting the needs of different products.

[0046] (6) The learning rate and discount factor are automatically calculated and determined by the system, and the optimal strategy can be learned automatically without manual parameter tuning.

[0047] Other features and advantages of the present invention will become clearer after reading the detailed description of the embodiments of the present invention in conjunction with the accompanying drawings. Attached Figure Description

[0048] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 shows the system architecture for optimizing production scheduling using reinforcement learning in existing intelligent manufacturing processes.

[0050] Figure 2 shows the system structure of the tobacco production scheduling optimization system based on reinforcement learning proposed in this invention;

[0051] Figure 3 illustrates the execution flow of the tobacco production scheduling optimization method based on reinforcement learning proposed in this invention. Detailed Implementation

[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] This invention aims to optimize the tobacco production scheduling system by applying improved reinforcement learning. It upgrades the single scalar optimization based on manual weight adjustment to a multi-dimensional vector automatic balancing optimization that eliminates the need for manual parameter adjustment. This allows the reward vector to be calculated in real time based on the current production status and to automatically respond to environmental changes, enabling the system to learn to find the optimal balance point among time, load, and energy consumption.

[0054] Specifically, referring to Figures 2 and 3, the production organization optimization system and its optimization method based on reinforcement learning proposed in this invention are implemented as follows:

[0055] 1. Initialize the module.

[0056] The state space, action space, and Q-table are initialized to establish a complete learning framework, ensuring that the system starts the adaptive learning process from zero state.

[0057] State space initialization: Define all possible states of production scheduling, represented by coupling number S(s1,s2), where s1 is the workpiece number and s2 is the process number, forming an n×m dimensional state space matrix.

[0058] Action space initialization: Establish a set of available equipment M (M1, M2, ..., Mm), define the selection space for m parallel production equipment, and provide the basis for equipment selection for each process.

[0059] Vectorized Q-table initialization: Creating a three-dimensional vector Q-table, this invention employs [ , , A three-dimensional vector structure replaces the traditional scalar Q-value, maintaining the integrity of multi-objective optimization information such as time, load, and energy consumption, thereby achieving true multi-objective balance. Furthermore, in this invention, a sparse Q-table is established in conjunction with process constraints, which can fundamentally avoid process violation decisions, thus ensuring process compliance. During initialization, each Q-value is initialized to [0, 0, 0], corresponding to the time dimension, load dimension, and energy consumption dimension, respectively, providing an initial benchmark for subsequent learning.

[0060] In this embodiment of the invention, it is assumed that there are three production batches (workpieces) that need to be processed:

[0061] The workpieces are defined as: workpiece 1 (grade 1), workpiece 2 (grade 2), and workpiece 3 (grade 3).

[0062] The standard tobacco processing technology includes five steps: leaf processing (raw material pretreatment), tobacco drying, tobacco storage and balancing (intermediate storage), blending and flavoring (formula mixing), and packaging and finished product inspection (final packaging).

[0063] State space S(s1, s2): s1∈(1,2,3) (workpiece number), s2∈(1,2,3,4,5) (process number). Specific state representation: S(1,1): Grade 1 - Leaf processing, S(1,2): Grade 1 - Tobacco drying, S(1,3): Grade 1 - Tobacco storage and balancing, S(1,4)……S(3,5), a total of 15 states.

[0064] Taking the selection of 5 different devices as an example, the available device set M = (M1, M2, M3, M4, M5): M1: blade processing line A (A line); M2: blade processing line B (B line); M3: drying line D (D line); M4: drying line E (E line); M5: blending and flavoring line X (X line); then the action space is initialized as: A∈[1,2,3,4,5,].

[0065] Each Q value in the three-dimensional vector Q table is a three-dimensional vector of [time, load, energy], and its structure is Q[state][action] = [ , , The process constraints include: leaf processing - only update M1 and M2, drying - only update M3 and M4, blending and flavoring - only update M5, storage - no update (no option), packaging - currently not selectable.

[0066] All-zero initialization is performed using the sparse Q-table shown in Table 1 below:

[0067] Table 1 Initialization of the sparse Q-table with process constraints.

[0068]

[0069] In the Q table above, "-" indicates that the process is unavailable. The system only updates and makes decisions on the Q values ​​of available actions. In this case, the sparse Q table structure truly reflects the process constraints in the yarn making workshop.

[0070] 2. Data Acquisition Module: Collects key performance indicator data from the production site in real time, providing real-time input for the calculation of the three-dimensional reward vector, ensuring that the calculation of the reward vector is based on the latest production status and achieving dynamic environmental adaptability.

[0071] (1) Time data acquisition. The estimated completion time T of the current process is obtained in real time through the MES system, including the time consumption statistics of the current process and the calculation of the estimated completion time, with time accuracy reaching the hour level.

[0072] (2) Collect load data. Collect the total load rate (CT) of the production line, calculate the equipment utilization rate and process section load, and reflect the busyness of the production line and the resource utilization status in real time.

[0073] (3) Collect energy consumption data. Monitor the operating power P of the equipment, and obtain the actual energy consumption level of each process point through real-time power monitoring and energy intensity calculation.

[0074] For example, suppose the current state is S(1,2): workpiece 1 is in the wire drying process. In terms of time data, the estimated completion time of the current wire drying process is T=2.5 hours, 1.2 hours have been consumed so far, and 1.3 hours are expected to remain. In terms of equipment utilization (load), line D is 80% (currently running), line E is 70% (alternate), and CT=0.75 (total load rate 75%). In terms of power data, the current power of line D is P=150kW, and the standby power of line E is 30kW.

[0075] 3. Reward Calculation Module: Calculates a three-dimensional reward vector based on real-time data to achieve integrated evaluation of multi-objective optimization.

[0076] Time dimension calculation adopts Where T is the completion time. This is a power correction factor that couples time and energy consumption, encouraging rapid and energy-efficient production choices. Compared to traditional solutions... or The linear negative reward is addressed by the present invention through coupling power correction factor. This allows time-based rewards to simultaneously reflect the dual goals of efficiency and energy conservation.

[0077] Load dimension calculation uses , The reward is based on the overall load factor; the lower the load, the higher the reward, thus promoting load balancing across devices. Compared to traditional solutions, this will... As a constraint or weighting factor, this invention treats it as an independent reward dimension to directly optimize equipment utilization and avoid local optima traps such as "fast but high load".

[0078] In energy consumption calculations, power consumption On the one hand, it is treated as a separate reward dimension; on the other hand, it is addressed through a power correction factor. By coupling with the time dimension, intelligent association of reward-related components is achieved. Compared to traditional solutions that treat energy consumption as an independent dimension (which risks conflicting with time objectives), this invention, through a coupled design, enables the power correction factor to... It simultaneously serves time and energy consumption optimization, eliminating interference between dimensions.

[0079] For example, for a three-dimensional reward vector =[Time dimension, Load dimension, Energy consumption dimension]:

[0080] Time dimension: Among them, the power correction factor Directly using power The numerical part.

[0081] Load dimension: .

[0082] Energy consumption dimension: and through power correction factor Coupled in the time dimension.

[0083] Final reward vector =[0.00267, 1.33, 150].

[0084] This invention eliminates the need for manual weight parameter setting in existing technologies by designing a multi-dimensional reward vector. It automatically finds the optimal balance between time, load, and energy consumption through Q-learning, significantly shortening the product launch cycle.

[0085] 4. Q-table update module: The vectorized Q-table is updated using an improved vector Bellman equation.

[0086] The improved vector Bellman equation of this invention is as follows:

[0087] ;in, For state Select action The value vector, For the reward vector, For learning rate, As a discount factor, For the next state Select action The value vector, A set of available actions; Number the components of the vector. Let be the dimension of the vector.

[0088] The system performs a Bellman vector update for each state-action pair to ensure the integrity and consistency of the Q-table. In one learning iteration, the Q-values ​​of all state-action pairs are updated, avoiding decision bias caused by partial updates.

[0089] In the improved vector Bellman equation described above, the learning rate The discount factor is controlled within the range of 0.01-0.1 and determined using an adaptive attenuation strategy. The value is set in the range of 0.8-0.95, determined based on the characteristics of the production cycle.

[0090] Specifically, regarding the learning rate The adaptive decay strategies include:

[0091] (1) Automatically calculate the learning effect of the most recent n iterations;

[0092] (2) Calculate the rate of change of Q value ;

[0093] (3) Optimal learning rate ;

[0094] (4) Boundary constraints: ensure ;

[0095] (5) Each The calculation is automatically recalculated in the next iteration.

[0096] For example, in the 500th iteration, (Statistics automatically compiled by the system) The system output will be... .

[0097] Specifically, regarding the discount factor Determining production adaptability includes:

[0098] (1) Automatically calculate the average production cycle ;

[0099] (2) Automatically identify the product type based on the current work order;

[0100] (3) Optimal discount rate , This is the product value coefficient.

[0101] (4) Setting intelligent boundaries: high-value products Ordinary products ;

[0102] (5) Ensure .

[0103] For example, if the current product workpiece 1 (brand 1) is detected as a high-value product, its average cycle time is calculated. Hours, value coefficient is ,but Adjust the output to .

[0104] The following is a complete example of updating the Q table:

[0105] Assumptions: Current state S(1,2) - Grade 1 drying wire, parameters automatically calculated by the system. , The next state is S(1,3) - tobacco storage, and the available actions are: only M3 and M4.

[0106] The improved vector Bellman equation is then updated as follows:

[0107] Action M3 (D line):

[0108]

[0109] Action M4 (E line):

[0110]

[0111] The updated Q[S(1,2)]=( ,

[0112] ).

[0113] 5. Decision Module: Based on process constraints, select the set of available actions, calculate the vector sum of the value vectors corresponding to each action in the set of available actions, and select the optimal action based on the maximum vector sum to achieve intelligent trade-offs among multiple objectives.

[0114] For the set of available actions For each available action, calculate the sum of its Q-value vectors: .

[0115] Vector and maximization selection is performed using a maximization strategy under process constraints:

[0116] .

[0117] by Probability-based random exploration, in order to Probabilistic selection of the current optimal action: when the random number is less than At that time, an action is randomly selected from the set of available actions for exploration. When the random number is greater than or equal to When that happens, a vector sum maximization selection is performed on the available action set.

[0118] For example: State S(1,2) (Grade 1 - Drying wire), only M3 and M4 are available (process constraints). Conflict scenario: M3 has a short time but high energy consumption, while M4 has a long time but low energy consumption.

[0119] For the traditional scalar (manually preset weights) scheme, the Q value (implying a fixed weight) is manually set: Q[S(1,2)][M3]=15 (human preference for time), Q[S(1,2)][M4]=14 (human preference for energy consumption), and the decision is max(15,14), choosing M3. The drawback of the traditional scalar method is that if brand 1 is a high-end cigarette, choosing M3 happens to be correct, but if it is a regular cigarette brand 2, choosing M4 is more appropriate. Therefore, the traditional scalar value method cannot adapt to different products.

[0120] The three-dimensional vector scheme of this invention calculates the three-dimensional Q values ​​as follows: Q[S(1,2)][M3]=[9.8,5.2,2.1] corresponds to optimal time, moderate load, and poor energy consumption; Q[S(1,2)][M4]= [6.1,5.8,8.9] corresponds to poor time, optimal load, and low energy consumption; the vector sum calculation is Score[M3]=9.8+5.2+2.1=17.1, Score[M3]=6.1+5.8+8.9=20.8, the decision is max(17.1,20.8), and M4 is selected. That is, the system automatically finds the optimal balance point of time, load, and energy consumption. The whole process does not require manual preset weights, and the contribution of each dimension is clearly visible.

[0121] The intelligent trade-off achieved by the method of this invention breaks the rigidity of human weighting, eliminates reliance on experts, realizes zero human intervention in decision-making, and makes the decision-making process transparent and clear, finding the mathematically optimal balance point between time, load and energy consumption.

[0122] Compared to traditional single scalars, the three-dimensional vector Q-value fully preserves the performance profile, optimizing precisely to each dimension (single scalars can only distinguish between good and bad) and clearly indicating which dimension needs optimization, allowing for precision down to each dimension during learning.

[0123] 6. Execution and Feedback Module: Executes the selected optimal action, collects actual production data to calculate the actual reward vector in the new state, and completes the state transition; feeds back the actual reward vector and the new state information to the system.

[0124] Execution: Select the optimal production equipment based on the decision results, start the production process, monitor the entire execution process, and ensure the effective implementation of the action.

[0125] Environmental feedback: Obtaining new system state Calculate the actual reward vector value and record complete performance indicators including time, load, and energy consumption.

[0126] State transition: Completed The state transition updates the process progress information, preparing for the next round of decision-making.

[0127] 3D Vector Feedback: Unlike traditional scalar feedback, this invention acquires a complete 3D performance vector. =[Actual time deviation, actual load rate deviation, actual energy consumption deviation], forming a complete decision-making-execution-feedback closed loop to ensure the effectiveness and practicality of the learning process.

[0128] For example: Current state S (1, 2), select action M3, planned duration 2.5 hours, start at 08:00, actual duration 2.3 hours (completed 0.2 hours ahead of schedule), real-time collection of equipment load rate and electricity meter data (energy consumption).

[0129] The three-dimensional vector feedback acquisition is as follows: the planned indicator three-dimensional vector is [2.5h, 85%, 150kWh], the actual three-dimensional vector is [2.3h, 82%, 145kWh], and the performance vector is r=[+0.2h, -3%, -5kWh]=[0.2, -0.03, -5].

[0130] State transition: Original state S(1,2), current state S(1,3), transition: 08:00-10:18.

[0131] 7. Adaptive Adjustment Module: Monitors changes in the production environment in real time. When the reward component of a key performance indicator exceeds the threshold, it triggers an adjustment of the value vector of the relevant state-action pair. It calculates the reward vector based on the adjusted value vector and selects the optimal action based on the new reward vector.

[0132] In this embodiment of the invention, environmental change detection includes, but is not limited to, real-time monitoring of dynamic changes such as equipment aging (processing time T gradually increases), order fluctuations (sudden changes in CT load), and emergency order insertions (insertion of new processes). For example, the original reward (expressed in performance) r = [0.2h, 2%, -5kWh] (good time, medium load, good energy saving), and the detected change rˊ (expressed in performance) = [0.8h, -5%, 8kWh] (time difference, low load, high energy consumption). If the anomaly judgment criteria are: time dimension Δ > 0.5h, energy consumption dimension Δ > 10kWh, then an adjustment is triggered.

[0133] This invention achieves automatic optimization of decision-making strategies without manual intervention by dynamically updating the Q-table to adapt to new conditions through real-time recalculation of the reward vector based on changes. It can display three-dimensional performance indicators in real time, provide automatic early warnings of abnormal states, and intelligently push maintenance suggestions, thus enabling predictive maintenance.

[0134] For example, under normal initial conditions: the reward for M3 executing S(1,2) is [0.2h, 2%, -5kWh]. If the equipment ages, after 5 consecutive executions, the updated reward becomes [0.8h, -5%, 8kWh]. The system then detects... The system triggers an adjustment, with the adjustment strategy being Q[S(1,2)][M3][time]×0.8 (a 20% reduction, with M4's relative advantage automatically increasing). The new decision score is Score[M3]=8.12, Scor[M4]=9.45, and the system automatically switches to the more stable M4.

[0135] 8. Implement the entire optimization process in a loop and terminate it normally.

[0136] Based on three-dimensional performance indicators, the termination conditions are determined, and the optimization results containing complete data on time, load, and energy consumption are output, forming a complete closed loop of decision-making, execution, feedback, adjustment, and termination. This ensures the orderly progress of the optimization process and the complete output of results, providing comprehensive decision support for production management.

[0137] This invention achieves the transformation from human experience-based decision-making to mathematical intelligent optimization through the above steps. It realizes automatic trade-offs between time, load, and energy consumption through three-dimensional vector Bellman equations, ensures zero-violation decisions by combining process constraint sparse Q-tables, and combines adaptive calculation of α and γ parameters with real-time detection and feedback of three-dimensional changes to ultimately achieve intelligent production organization optimization driven by data, algorithmic decision-making, and zero manual parameter tuning.

[0138] It should be noted that, in the specific implementation process, the above-mentioned methods can be implemented by a hardware processor executing computer-executable instructions in software form stored in memory, which will not be elaborated here. The programs corresponding to the actions executed can all be stored in the computer-readable storage medium of the system in software form, so that the processor can call and execute the operations corresponding to the above modules.

[0139] The computer-readable storage media mentioned above may include volatile memory, such as random access memory; may also include non-volatile memory, such as read-only memory, flash memory, hard disk or solid-state drive; and may also include combinations of the above types of memory.

[0140] The term "processor" as mentioned above can also refer to a collective of multiple processing elements. For example, a processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor, and it can also be a special-purpose processor.

[0141] It should be noted that the above description is not intended to limit the present invention, and the present invention is not limited to the examples given above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A tobacco production scheduling optimization method based on reinforcement learning, characterized in that, include: The state space S, action space A, and vectorized Q-table of the tobacco production system are initialized. The state space S is defined based on a combination of workpiece and process numbers; the action space A is defined based on the set of available production equipment; the vectorized Q-table consists of value vectors corresponding to states and actions, each value vector including multi-dimensional components, each component corresponding to a type of production data; production data is collected in real time, and a reward vector is calculated; the reward vector is a multi-dimensional vector with the same dimensions as the value vectors, where the calculation of at least one component is coupled with information from other components; the vectorized Q-table is updated using an improved Bellman equation; a set of available actions is selected based on process constraints, and the vector sum of the value vectors corresponding to each action in the set is calculated; the optimal action is selected based on the maximum vector sum; the optimal action is executed, and actual production data is collected to calculate the actual reward vector in the new state, completing the state transition; the actual reward vector and the new state information are fed back to the system.

2. The tobacco production scheduling optimization method based on reinforcement learning according to claim 1, characterized in that, The vectorized Q-table is constructed as a sparse structure based on production process constraints, with each state associated only with the available actions allowed by the process.

3. The tobacco production scheduling optimization method based on reinforcement learning according to claim 1, characterized in that, The reward vector includes a time component, a load component, and a power consumption component; wherein, the time component is the reciprocal of the expected completion time T coupled with the equipment operating power P, and the load component is the reciprocal of the total load rate CT.

4. The tobacco production scheduling optimization method based on reinforcement learning according to claim 1, characterized in that, The improved Bellman equation is as follows: ;in, For state Select action The value vector, For the reward vector, For learning rate, As a discount factor, For the next state Select action The value vector, The set of available actions; Number the components of the vector. Let be the dimension of the vector.

5. The tobacco production scheduling optimization method based on reinforcement learning according to claim 4, characterized in that, The learning rate The calculation is based on an adaptive decay strategy, including: automatically calculating the learning effect of the most recent n iterations; and calculating the rate of change of the Q-value. ; Calculate the optimal learning rate Among them, setting boundary constraints: ensuring ;Every The calculation is automatically recalculated in the next iteration.

6. The tobacco production scheduling optimization method based on reinforcement learning according to claim 4, characterized in that, The discount factor Determined based on average production cycle and product value, including: automatically calculating the average production cycle. Automatically identify the product type based on the current work order; calculate the optimal discount rate. ， The product value coefficient; where, the intelligence boundary is defined: high-value products Ordinary products ;make sure 。 7. The tobacco production scheduling optimization method based on reinforcement learning according to claim 1, characterized in that, Its features are, The method further includes: real-time monitoring of changes in the production environment; when the reward component of a key performance indicator is detected to exceed a threshold, triggering an adjustment of the value vector of the relevant state-action pair; calculating a reward vector based on the adjusted value vector; and selecting the optimal action based on the new reward vector.

8. The tobacco production scheduling optimization method based on reinforcement learning according to claim 7, characterized in that, The adjustment of the value vector includes adjusting the positive / negative proportions of components that exceed the threshold.

9. A tobacco production scheduling system based on reinforcement learning, characterized in that, include: An initialization module is used to initialize the state space S, action space A, and vectorized Q-table of the tobacco production system. The state space S is defined based on a combination of workpiece number and process number; the action space A is defined based on the set of available production equipment; the vectorized Q-table consists of value vectors corresponding to states and actions, each value vector including multi-dimensional components, each component corresponding to a type of production data. A data acquisition and reward calculation module is used to acquire production data in real time and calculate reward vectors; the reward vector is a multi-dimensional vector with the same dimensions as the value vectors, where the calculation of at least one component is coupled with information from other components. A Q-table update module is used to update the vectorized Q-table using an improved Bellman equation. The decision-making module is used to select the set of available actions based on process constraints, calculate the vector sum of the value vectors corresponding to each action in the set of available actions, and select the optimal action based on the maximum vector sum; the execution and feedback module is used to execute the optimal action, collect actual production data to calculate the actual reward vector in the new state, complete the state transition, and feed back the actual reward vector and the new state information to the system.

10. The tobacco production scheduling system based on reinforcement learning according to claim 9, characterized in that, The system also includes an adaptive adjustment module, which monitors changes in the production environment in real time. When the reward component of a key performance indicator is detected to exceed a threshold, the module triggers an adjustment of the value vector of the relevant state-action pair, calculates the reward vector based on the adjusted value vector, and selects the optimal action based on the new reward vector.

Images