Intelligent coffee machine running state monitoring method and system based on reinforcement learning

Abstract

The invention discloses an intelligent coffee machine operation state monitoring method and system based on reinforcement learning, and the method comprises the following steps: S1, collecting and preprocessing data, and forming an operation parameter sequence; s2, carrying out dimension reduction processing on the operation parameters, and constructing an operation state sequence; s3, performing multi-step prediction on the running state sequence through a DLinear model, and generating a state prediction sequence by adopting a linear predictor; s4, calculating an instant reward value of each regulation and control behavior according to the running state sequence and the state prediction sequence; s5, an A3C algorithm is adopted, and a regulation and control instruction is generated and executed according to the instant reward value; s6, establishing a state transition group and writing the state transition group into an empirical data set; and S7, updating the DLinear model and A3C algorithm parameters according to the empirical data set. According to the method, the Markov model, the principal component analysis, the DLinear model and the A3C algorithm are fused, and the method has the advantages of being high in adaptability, high in prediction precision and good in stability.

Description

Technical Field

[0001] This invention relates to the field of equipment condition monitoring technology, and in particular to a method and system for monitoring the operating status of an intelligent coffee machine based on reinforcement learning. Background Technology

[0002] With the continuous development of the Internet of Things and embedded smart devices, smart coffee machines, as typical smart devices for consumer terminals, have gradually acquired multi-functional, multi-mode, and multi-parameter adjustable operating characteristics. In actual use, the operational stability and control precision of smart coffee machines have a significant impact on the quality of the finished product, thus placing higher demands on the real-time monitoring and intelligent optimization of the operating status of smart coffee machines.

[0003] Currently, commonly used operational status monitoring methods mainly rely on threshold judgments, fixed rule bases, or strategy templates developed based on human experience. These methods typically make static judgments based on a single parameter or a small number of related parameters, lacking the ability to perceive trends in operational status changes and failing to provide detailed modeling of fluctuations in equipment performance, environmental disturbances, or minor degradation caused by long-term operation. Furthermore, due to the static nature of the strategies and the lag in feedback, the system is prone to misjudgments, missed judgments, or response delays in complex multi-state switching scenarios, making it difficult to achieve adaptive control of operational behavior.

[0004] In some high-end equipment, some studies have attempted to introduce predictive models or control algorithms to perform forward estimation and rule-triggered control of the equipment's operating state. However, most methods remain at the level of setpoint regression or short-term window modeling, failing to capture the long-term evolution trend of the operating state. Furthermore, existing methods generally lack reward-driven feedback loop mechanisms and the ability to dynamically update state monitoring strategies.

[0005] Reinforcement learning, as a reward-driven decision optimization method, possesses strong adaptive capabilities in environment modeling, action selection, and policy updating, providing a new technical path for solving the real-time, flexibility, and long-term adaptability issues in the monitoring of intelligent coffee machine operation status. However, existing technologies lack a complete solution that integrates multidimensional data dimensionality reduction analysis, time-series trend modeling, and reward-driven intelligent policy optimization methods.

[0006] Therefore, how to provide a method and system for monitoring the operating status of intelligent coffee machines based on reinforcement learning is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0007] One objective of this invention is to propose a method and system for monitoring the operating status of an intelligent coffee machine based on reinforcement learning. This invention fully integrates Markov state modeling, principal component analysis, DLinear prediction model, and A3C reinforcement learning algorithm to construct an intelligent state monitoring process with the capabilities of state modeling, trend prediction, reward evaluation, and policy self-updating. It describes in detail the entire process from multi-dimensional operating data acquisition, state extraction, trend prediction, reward function construction to control policy generation and feedback optimization, and has the advantages of strong adaptability, high prediction accuracy, and good long-term stability.

[0008] A method for monitoring the operating status of a smart coffee machine based on reinforcement learning according to an embodiment of the present invention includes the following steps:

[0009] S1. Collect and preprocess multi-dimensional operating data of the smart coffee machine at a preset frequency to form an operating parameter sequence;

[0010] S2. In the Markov model, principal component analysis is used to reduce the dimensionality of the running parameters at each time step and extract the principal component features to construct the running state sequence.

[0011] S3. Perform multi-step prediction operations on the running state sequence through the DLinear model, construct a sliding time window to extract long-term trend components and short-term disturbance components, and use a linear predictor to generate the state prediction sequence.

[0012] S4. Based on the feature difference between the running state sequence and the state prediction sequence, construct the state reward function and calculate the instantaneous reward value of each control behavior in the preset behavior library.

[0013] S5. Using the A3C algorithm, the control instruction selection operation is performed based on the instant reward value. The control instructions for each time step are generated and executed through the policy network.

[0014] S6. Combine the running state sequence, control instructions, reward value and the running state sequence generated in real time after the control instructions are executed into a state transition group, and perform time difference error calculation and priority binding operation, and write the binding result into the existing experience data set.

[0015] S7. Select a training sample set from the experience data set according to the priority experience replay mechanism, and update the parameters of the DLinear model and the A3C algorithm through the backpropagation mechanism.

[0016] Optionally, the multidimensional operating data includes temperature, fluid pressure, water flow rate, current value, and operating duration; the preprocessing includes time alignment, noise filtering, and numerical normalization operations; the sliding time window represents a continuous time segment that is slidably extracted in the operating state sequence with a fixed step size and length; the preset behavior library represents a predefined set of candidate behaviors for regulating the operating state of the coffee machine; and the empirical data set represents a set of state transition groups containing priority weights that are continuously accumulated during operation.

[0017] Optionally, S2 specifically includes:

[0018] S21. In the Markov model, the sequence of operating parameters is expanded in chronological order, and the operating parameters corresponding to each time step are organized into parameter vectors. A state transition diagram is constructed based on the parameter vectors between adjacent time steps, specifically including:

[0019] The sequence of operating parameters is divided into multiple time segments in chronological order, and each time segment contains a set of operating parameter values ​​arranged according to preset indicators;

[0020] The values ​​of the running parameters in each time segment are combined into a parameter vector in a fixed order, forming a parameter vector sequence arranged by time step;

[0021] In a Markov model, parameter vectors from adjacent time steps are paired, and the state transition relationship between each pair is recorded. Specifically, this includes:

[0022] Pair up two adjacent parameter vectors in the parameter vector sequence to construct a set of state transition pairs;

[0023] Each state transition pair is numbered and assigned a directed connection label, which represents the unidirectional evolution relationship between the state transition pairs;

[0024] In the state transition pair, the parameter vector of the previous time step is marked as the current state node, and the parameter vector of the next time step is marked as the target state node.

[0025] Based on the directed connection markers between each current state node and its corresponding target state node, the resulting state transition relationships are recorded;

[0026] Use all parameter vectors as state nodes and construct a state transition graph by combining the corresponding state transition relationships.

[0027] S22. Using principal component analysis, the parameter vectors are subjected to mean removal, and the correlation matrix between the mean-removed parameter vectors is calculated. The set of principal directions is extracted based on the correlation matrix.

[0028] S23. Project the parameter vectors corresponding to each time step according to the principal direction set, and extract the principal component features of each time step.

[0029] S24. Associate the principal component features generated at each time step in sequence according to the state transition diagram to construct a sequence of running states arranged by time step.

[0030] Optionally, the extraction process of the principal component features specifically includes:

[0031] The parameter vector sequence is constructed into a two-dimensional parameter matrix according to the parameter dimension, and each row of the parameter matrix corresponds to the parameter vector of a time step;

[0032] Perform mean calculation on each column of the parameter matrix to obtain the mean vector of each parameter dimension at all time steps;

[0033] Subtract the corresponding mean vector from the parameter vector in each row of the parameter matrix to generate a zero-mean matrix;

[0034] The covariance matrix between each parameter dimension is calculated based on the zero-mean matrix, and the elements in the covariance matrix represent the degree of co-variance between two parameter dimensions.

[0035] Perform eigenvalue decomposition on the covariance matrix to extract all eigenvalues ​​and the corresponding set of eigenvectors;

[0036] Sort all feature values ​​in descending order, and select the feature vectors corresponding to the top feature values ​​in the sorting results according to the preset cumulative proportion to construct the main direction set;

[0037] By projecting each parameter vector onto the principal direction set, the principal component features corresponding to each time step are obtained.

[0038] Optionally, S3 specifically includes:

[0039] S31. Construct a sliding time window of fixed length based on the running state sequence, and divide the running state sequence into multiple state segments with a preset overlapping range. The state segment represents the evolution process of the smart coffee machine's running state within the time period.

[0040] S32. Input each state segment into the DLinear model, perform trend modeling and disturbance modeling operations along the time dimension, and extract the long-term trend component and short-term disturbance component respectively.

[0041] S33. The long-term trend component and short-term disturbance component of each state segment are spliced ​​together in the feature dimension to form a fusion feature matrix, which represents the comprehensive evolution characteristics within the state segment.

[0042] S34. Input the fused feature matrix into the linear predictor to perform multi-step prediction operations, and encode the predictions for future time. Establish a time mapping relationship between the encoding results and the time steps in the running state sequence to generate a state prediction sequence, specifically including:

[0043] The fusion feature matrix is ​​divided into multiple sub-windows, each sub-window containing a fixed number of fusion feature vectors for consecutive time steps;

[0044] Each sub-window is input into the linear predictor. Based on the overall changing trend of the fused feature vectors in the sub-windows, a state prediction operation with a set step size is performed to generate a set of predicted state vectors consistent with the step size.

[0045] For each predicted state vector, a future time step encoding information is appended, wherein the time step encoding information represents the predicted time index corresponding to the predicted state;

[0046] All predicted state vectors with time step encodings are arranged in time index order and a time mapping relationship is established with the time steps in the running state sequence to generate a state prediction sequence with the same structure as the running state sequence.

[0047] Optionally, S32 specifically includes:

[0048] S321. Arrange the running state vectors in the state segment in chronological order to construct a state segment matrix. Each row of the state segment matrix corresponds to the running state vector of a time step, and each column corresponds to a different feature dimension in the running state vector.

[0049] S322. Treat each column of the state segment matrix as a time evolution sequence of the corresponding feature dimension. In the trend modeling channel of the DLinear model, perform a least-squares fitting operation on the state segment matrix column by column to extract the trend estimation sequence corresponding to the time evolution sequence of each feature dimension, and concatenate them column by column to generate a long-term trend matrix. Specifically, this includes:

[0050] Extract each column from the state fragment matrix and construct a time evolution sequence from the state feature values ​​corresponding to that column in chronological order;

[0051] For each time evolution sequence, a corresponding time index sequence is constructed. Each time index is paired with the corresponding state feature value to form a coordinate pair, and a fitting sample set for that feature dimension is constructed.

[0052] Based on the fitted sample set, the slope and intercept values ​​are calculated using the least squares method. Then, based on the slope and intercept values, the trend fitting value for each time step is calculated to generate a trend estimation sequence, specifically including:

[0053] For each feature dimension, calculate the mean of the time index and the mean of the state feature in the corresponding fitted sample set;

[0054] Subtract the mean state feature value from the state feature value of each time step to obtain the corresponding state feature deviation value. At the same time, subtract the mean time index from the time index of each time step to obtain the corresponding time index deviation value.

[0055] Multiply the state feature deviation value corresponding to each time step with the time index deviation value, and calculate the average of all product results to obtain the covariance between the feature dimension and the time index.

[0056] The variance of the time index is obtained by averaging the squared results of all time index deviations.

[0057] Divide the covariance by the variance to obtain the slope value of that feature dimension;

[0058] The intercept value of this feature dimension is obtained by subtracting the product of the slope value and the time index mean from the mean of the state feature.

[0059] Multiply the time index by the slope value and add it to the intercept value to obtain the trend fit value for the corresponding time step;

[0060] Arrange all trend fit values ​​in chronological order to form a trend estimation sequence for this feature dimension;

[0061] By concatenating the trend estimation sequences of all feature dimensions column by column, a long-term trend matrix with the same structure as the state fragment matrix is ​​constructed.

[0062] S323. In the disturbance modeling channel, the difference between the state segment matrix and the long-term trend matrix is ​​calculated according to the element position to obtain the residual change value of each time step in each feature dimension to construct the short-term disturbance matrix.

[0063] S324. Perform structural reorganization operations on the long-term trend matrix and the short-term disturbance matrix at time steps, and mark the source of the feature dimensions respectively to form the long-term trend component and the short-term disturbance component.

[0064] Optionally, S4 specifically includes:

[0065] S41. Perform time alignment processing on the running state sequence and the state prediction sequence according to the time mapping relationship to construct a state aligned sequence;

[0066] S42. Perform feature difference calculation operation on each set of running state vectors and predicted state vectors in the state alignment sequence to generate a feature difference sequence;

[0067] S43. According to the preset mapping rules, each feature difference vector is mapped to a state offset score value, wherein the state offset score value represents the degree of deviation between the running state vector and the predicted state vector.

[0068] S44. Construct a state reward function based on the state offset score, wherein the state reward function includes a dense reward component and a sparse reward component, specifically including:

[0069] The continuous rate of change and the magnitude of the prediction error are calculated based on the state offset score at each time step to form a dense reward component.

[0070] Identify abrupt time steps in the state offset score value, mark abrupt time steps whose state offset score value exceeds the set reference threshold as high-risk time steps based on the identification results, and set fixed reward incentive or penalty value in high-risk time steps to form sparse reward components.

[0071] The dense reward component and the sparse reward component are superimposed and fused at time steps to construct the state reward function;

[0072] S45. Based on the state reward function, calculate the instantaneous reward value of each control behavior in the preset behavior library at each time step.

[0073] Optionally, S5 specifically includes:

[0074] S51. Use the A3C algorithm to initialize the parameters of the policy network;

[0075] S52. Perform behavior probability calculation on the running state vector at each time step in the policy network to generate the behavior probability distribution corresponding to each time step. Perform weighted processing on the behavior probability distribution based on the immediate reward value, and select the control behavior based on the weighted result to generate a set of control instructions, specifically including:

[0076] In the policy network, a forward propagation operation is performed on the running state vector at each time step to obtain the probability scores of all regulatory behaviors at the current time step and construct the behavior probability distribution.

[0077] The instant reward value is used as a weighting factor, and the probability scores of the corresponding time step control behavior in the behavior probability distribution are weighted according to the preset weighting mechanism.

[0078] The regulation behavior with the highest probability score in the weighted behavior probability distribution is selected as the optimal regulation behavior for that time step.

[0079] Extract the behavior description content corresponding to the optimal control behavior from the preset behavior library, construct the control instructions for this time step, and summarize all control instructions into a control instruction set;

[0080] S53. Execute the set of control instructions in the smart coffee machine in chronological order, and trigger the corresponding control behavior at each time step.

[0081] Optionally, S6 specifically includes:

[0082] S61. Real-time acquisition of multi-dimensional operation data corresponding to the time step of the state prediction sequence, and generation of the operation state sequence after the execution of the control command as a state comparison sequence;

[0083] S62. Combine the running state sequence before the execution of the control command, the set of control commands, all instant reward values ​​and the state comparison sequence into a state transition group;

[0084] S63. Based on the instantaneous reward value and state comparison sequence in the state transition group, perform time difference error calculation operation to generate time difference sequence;

[0085] S64. Perform Min-Max normalization on the time difference sequence to obtain the priority score value as the priority weight, and bind it to the corresponding state transition group. Write the state transition group containing the priority weight into the empirical data set.

[0086] According to an embodiment of the present invention, a smart coffee machine operation status monitoring system based on reinforcement learning includes:

[0087] The data acquisition module is used to collect and preprocess multi-dimensional operating data of the smart coffee machine at a preset frequency to form a sequence of operating parameters;

[0088] The feature extraction module is used to perform dimensionality reduction on the running parameters at each time step in the Markov model using principal component analysis, and to extract principal component features to construct the running state sequence.

[0089] The state prediction module is used to perform multi-step prediction operations on the running state sequence through the DLinear model, construct a sliding time window to extract long-term trend components and short-term disturbance components, and use a linear predictor to generate the state prediction sequence.

[0090] The reward calculation module is used to construct a state reward function based on the feature difference between the running state sequence and the state prediction sequence, and to calculate the instantaneous reward value of each control behavior in the preset behavior library.

[0091] The instruction generation module is used to perform the control instruction selection operation based on the real-time reward value using the A3C algorithm, and to generate and execute the control instructions for each time step through the policy network.

[0092] The data storage module is used to form a state transition group by combining the running state sequence, control instructions, reward value and the running state sequence generated in real time after the control instructions are executed, and to perform time difference error calculation and priority binding operation, and write the binding result into the existing experience data set.

[0093] The feedback update module is used to select a training sample set from the experience data set according to the priority experience replay mechanism, and update the parameters of the DLinear model and the A3C algorithm through the backpropagation mechanism.

[0094] The beneficial effects of this invention are:

[0095] First, this invention extracts key feature information from the multidimensional operating data of a smart coffee machine using principal component analysis, and constructs a state transition diagram under the Markov state modeling framework. This enables continuous modeling, feature compression, and temporal expression of the equipment's operation process, effectively enhancing the sensitivity, descriptive ability, and modeling stability to changes in operating states.

[0096] Secondly, this invention utilizes the sliding time prediction structure constructed by the DLinear model to effectively separate the long-term trend component and the short-term disturbance component. It combines the feature difference between the running state and the predicted state to construct a state reward function, thereby completing the quantitative evaluation of the value of candidate regulatory behaviors. The A3C algorithm is used to realize the optimal policy generation, dynamic policy adjustment and parallel training optimization under multi-state switching conditions.

[0097] Finally, the time difference error calculation and priority weight binding mechanism designed in this invention supports continuous updates to the policy network in the DLinear model and A3C algorithm. The constructed experience data set can continuously accumulate key state transition information in a dynamic operating environment, realize priority learning of core samples and focused memorization of important experiences, and significantly improve the self-learning ability, convergence speed and long-term robustness of the monitoring policy. Attached Figure Description

[0098] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0099] Figure 1 This is a flowchart of a method for monitoring the operating status of an intelligent coffee machine based on reinforcement learning, as proposed in this invention.

[0100] Figure 2 This is a flowchart illustrating the strategy generation and feedback update process of a reinforcement learning-based intelligent coffee machine operation status monitoring method proposed in this invention.

[0101] Figure 3 This is a block diagram of the module structure of an intelligent coffee machine operation status monitoring system based on reinforcement learning proposed in this invention. Detailed Implementation

[0102] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0103] refer to Figure 1-2 A method for monitoring the operating status of an intelligent coffee machine based on reinforcement learning includes the following steps:

[0104] S1. Collect and preprocess multi-dimensional operating data of the smart coffee machine at a preset frequency to form an operating parameter sequence;

[0105] S2. In the Markov model, principal component analysis is used to reduce the dimensionality of the running parameters at each time step and extract the principal component features to construct the running state sequence.

[0106] S3. Perform multi-step prediction operations on the running state sequence through the DLinear model, construct a sliding time window to extract long-term trend components and short-term disturbance components, and use a linear predictor to generate the state prediction sequence.

[0107] S4. Based on the feature difference between the running state sequence and the state prediction sequence, construct the state reward function and calculate the instantaneous reward value of each control behavior in the preset behavior library.

[0108] S5. Using the A3C algorithm, the control instruction selection operation is performed based on the instant reward value. The control instructions for each time step are generated and executed through the policy network.

[0109] S6. Combine the running state sequence, control instructions, reward value and the running state sequence generated in real time after the control instructions are executed into a state transition group, and perform time difference error calculation and priority binding operation, and write the binding result into the existing experience data set.

[0110] S7. Select a training sample set from the experience data set according to the priority experience replay mechanism, and update the parameters of the DLinear model and the A3C algorithm through the backpropagation mechanism.

[0111] In this embodiment, the multidimensional operating data includes temperature, fluid pressure, water flow rate, current value, and operating duration. The preprocessing includes time alignment, noise filtering, and numerical normalization operations. The sliding time window represents a continuous time segment that is slidably extracted in the operating state sequence with a fixed step size and length. The preset behavior library represents a predefined set of candidate behaviors for regulating the operating state of the coffee machine. The experience data set represents a set of state transition groups containing priority weights that are continuously accumulated during operation.

[0112] In this embodiment, S2 specifically includes:

[0113] S21. In the Markov model, the sequence of operating parameters is expanded in chronological order, and the operating parameters corresponding to each time step are organized into parameter vectors. A state transition diagram is constructed based on the parameter vectors between adjacent time steps, specifically including:

[0114] The sequence of operating parameters is divided into multiple time segments in chronological order, and each time segment contains a set of operating parameter values ​​arranged according to preset indicators;

[0115] The values ​​of the running parameters in each time segment are combined into a parameter vector in a fixed order, forming a parameter vector sequence arranged by time step;

[0116] In a Markov model, parameter vectors from adjacent time steps are paired, and the state transition relationship between each pair is recorded. Specifically, this includes:

[0117] Pair up two adjacent parameter vectors in the parameter vector sequence to construct a set of state transition pairs;

[0118] Each state transition pair is numbered and assigned a directed connection label, which represents the unidirectional evolution relationship between the state transition pairs;

[0119] In the state transition pair, the parameter vector of the previous time step is marked as the current state node, and the parameter vector of the next time step is marked as the target state node.

[0120] Based on the directed connection markers between each current state node and its corresponding target state node, the resulting state transition relationships are recorded;

[0121] Use all parameter vectors as state nodes and construct a state transition graph by combining the corresponding state transition relationships.

[0122] S22. Using principal component analysis, the parameter vectors are subjected to mean removal, and the correlation matrix between the mean-removed parameter vectors is calculated. The set of principal directions is extracted based on the correlation matrix.

[0123] S23. Project the parameter vectors corresponding to each time step according to the principal direction set, and extract the principal component features of each time step.

[0124] S24. Associate the principal component features generated at each time step in sequence according to the state transition diagram to construct a sequence of running states arranged by time step.

[0125] In this embodiment, the extraction process of the principal component features specifically includes:

[0126] The parameter vector sequence is constructed into a two-dimensional parameter matrix according to the parameter dimension, and each row of the parameter matrix corresponds to the parameter vector of a time step;

[0127] Perform mean calculation on each column of the parameter matrix to obtain the mean vector of each parameter dimension at all time steps;

[0128] Subtract the corresponding mean vector from the parameter vector in each row of the parameter matrix to generate a zero-mean matrix;

[0129] The covariance matrix between each parameter dimension is calculated based on the zero-mean matrix, and the elements in the covariance matrix represent the degree of co-variance between two parameter dimensions.

[0130] Perform eigenvalue decomposition on the covariance matrix to extract all eigenvalues ​​and the corresponding set of eigenvectors;

[0131] Sort all feature values ​​in descending order, and select the feature vectors corresponding to the top feature values ​​in the sorting results according to the preset cumulative proportion to construct the main direction set;

[0132] By projecting each parameter vector onto the principal direction set, the principal component features corresponding to each time step are obtained.

[0133] In this embodiment, the calculation process of the covariance matrix specifically includes:

[0134] Treat each column of the zero-mean matrix as a time series of the corresponding parameter dimension;

[0135] For any two parameter dimensions, multiply the corresponding two time series step by step to obtain the corresponding product sequence;

[0136] Perform a summation operation on the product sequence of each pair of parameter dimensions, and divide the summation result by the total number of time steps to obtain the corresponding covariance value;

[0137] Construct a covariance matrix by filling the covariance values ​​between all pairs of parameters into the corresponding positions according to the row and column combination method.

[0138] In this embodiment, S3 specifically includes:

[0139] S31. Construct a sliding time window of fixed length based on the running state sequence, and divide the running state sequence into multiple state segments with a preset overlapping range. The state segment represents the evolution process of the smart coffee machine's running state within the time period.

[0140] S32. Input each state segment into the DLinear model, perform trend modeling and disturbance modeling operations along the time dimension, and extract the long-term trend component and short-term disturbance component respectively.

[0141] S33. The long-term trend component and short-term disturbance component of each state segment are spliced ​​together in the feature dimension to form a fusion feature matrix, which represents the comprehensive evolution characteristics within the state segment.

[0142] S34. Input the fused feature matrix into the linear predictor to perform multi-step prediction operations, and encode the predictions for future time. Establish a time mapping relationship between the encoding results and the time steps in the running state sequence to generate a state prediction sequence, specifically including:

[0143] The fusion feature matrix is ​​divided into multiple sub-windows, each sub-window containing a fixed number of fusion feature vectors for consecutive time steps;

[0144] Each sub-window is input into the linear predictor. Based on the overall changing trend of the fused feature vectors in the sub-windows, a state prediction operation with a set step size is performed to generate a set of predicted state vectors consistent with the step size.

[0145] For each predicted state vector, a future time step encoding information is appended, wherein the time step encoding information represents the predicted time index corresponding to the predicted state;

[0146] All predicted state vectors with time step encodings are arranged in time index order and a time mapping relationship is established with the time steps in the running state sequence to generate a state prediction sequence with the same structure as the running state sequence.

[0147] In this embodiment, S32 specifically includes:

[0148] S321. Arrange the running state vectors in the state segment in chronological order to construct a state segment matrix. Each row of the state segment matrix corresponds to the running state vector of a time step, and each column corresponds to a different feature dimension in the running state vector.

[0149] S322. Treat each column of the state segment matrix as a time evolution sequence of the corresponding feature dimension. In the trend modeling channel of the DLinear model, perform a least-squares fitting operation on the state segment matrix column by column to extract the trend estimation sequence corresponding to the time evolution sequence of each feature dimension, and concatenate them column by column to generate a long-term trend matrix. Specifically, this includes:

[0150] Extract each column from the state fragment matrix and construct a time evolution sequence from the state feature values ​​corresponding to that column in chronological order;

[0151] For each time evolution sequence, a corresponding time index sequence is constructed. Each time index is paired with the corresponding state feature value to form a coordinate pair, and a fitting sample set for that feature dimension is constructed.

[0152] Based on the fitted sample set, the slope and intercept values ​​are calculated using the least squares method. Then, based on the slope and intercept values, the trend fitting value for each time step is calculated to generate a trend estimation sequence, specifically including:

[0153] For each feature dimension, calculate the mean of the time index and the mean of the state feature in the corresponding fitted sample set;

[0154] Subtract the mean state feature value from the state feature value of each time step to obtain the corresponding state feature deviation value. At the same time, subtract the mean time index from the time index of each time step to obtain the corresponding time index deviation value.

[0155] Multiply the state feature deviation value corresponding to each time step with the time index deviation value, and calculate the average of all product results to obtain the covariance between the feature dimension and the time index.

[0156] The variance of the time index is obtained by averaging the squared results of all time index deviations.

[0157] Divide the covariance by the variance to obtain the slope value of that feature dimension;

[0158] The intercept value of this feature dimension is obtained by subtracting the product of the slope value and the time index mean from the mean of the state feature.

[0159] Multiply the time index by the slope value and add it to the intercept value to obtain the trend fit value for the corresponding time step;

[0160] Arrange all trend fit values ​​in chronological order to form a trend estimation sequence for this feature dimension;

[0161] By concatenating the trend estimation sequences of all feature dimensions column by column, a long-term trend matrix with the same structure as the state fragment matrix is ​​constructed.

[0162] S323. In the disturbance modeling channel, the difference between the state segment matrix and the long-term trend matrix is ​​calculated according to the element position to obtain the residual change value of each time step in each feature dimension to construct the short-term disturbance matrix.

[0163] S324. Perform structural reorganization operations on the long-term trend matrix and the short-term disturbance matrix at time steps, and mark the source of the feature dimensions respectively to form the long-term trend component and the short-term disturbance component.

[0164] In this embodiment, S324 specifically includes:

[0165] S3241. Take each row in the long-term trend matrix and the corresponding row in the short-term disturbance matrix as a data segment at the same time step.

[0166] S3242. For each data segment at a time step, extract the trend value and perturbation value of that time step in all feature dimensions, construct the trend vector and perturbation vector respectively, and attach a source label, wherein the source label is used to identify the feature dimension corresponding to each element;

[0167] S3243. Construct all trend vectors and disturbance vectors into long-term trend components and short-term disturbance components in chronological order.

[0168] In this embodiment, S4 specifically includes:

[0169] S41. Perform time alignment processing on the running state sequence and the state prediction sequence according to the time mapping relationship to construct a state aligned sequence;

[0170] S42. Perform feature difference calculation operation on each set of running state vectors and predicted state vectors in the state alignment sequence to generate a feature difference sequence;

[0171] S43. According to the preset mapping rules, each feature difference vector is mapped to a state offset score value, wherein the state offset score value represents the degree of deviation between the running state vector and the predicted state vector.

[0172] S44. Construct a state reward function based on the state offset score, wherein the state reward function includes a dense reward component and a sparse reward component, specifically including:

[0173] The continuous rate of change and the magnitude of the prediction error are calculated based on the state offset score at each time step to form a dense reward component.

[0174] Identify abrupt time steps in the state offset score value, mark abrupt time steps whose state offset score value exceeds the set reference threshold as high-risk time steps based on the identification results, and set fixed reward incentive or penalty value in high-risk time steps to form sparse reward components.

[0175] The dense reward component and the sparse reward component are superimposed and fused at time steps to construct the state reward function;

[0176] S45. Based on the state reward function, calculate the instantaneous reward value of each control behavior in the preset behavior library at each time step.

[0177] In this embodiment, S5 specifically includes:

[0178] S51. Use the A3C algorithm to initialize the parameters of the policy network;

[0179] S52. Perform behavior probability calculation on the running state vector at each time step in the policy network to generate the behavior probability distribution corresponding to each time step. Perform weighted processing on the behavior probability distribution based on the immediate reward value, and select the control behavior based on the weighted result to generate a set of control instructions, specifically including:

[0180] In the policy network, a forward propagation operation is performed on the running state vector at each time step to obtain the probability scores of all regulatory behaviors at the current time step and construct the behavior probability distribution.

[0181] The instant reward value is used as a weighting factor, and the probability scores of the corresponding time step control behavior in the behavior probability distribution are weighted according to the preset weighting mechanism.

[0182] The regulation behavior with the highest probability score in the weighted behavior probability distribution is selected as the optimal regulation behavior for that time step.

[0183] Extract the behavior description content corresponding to the optimal control behavior from the preset behavior library, construct the control instructions for this time step, and summarize all control instructions into a control instruction set;

[0184] S53. Execute the set of control instructions in the smart coffee machine in chronological order, and trigger the corresponding control behavior at each time step.

[0185] In this embodiment, S6 specifically includes:

[0186] S61. Real-time acquisition of multi-dimensional operation data corresponding to the time step of the state prediction sequence, and generation of the operation state sequence after the execution of the control command as a state comparison sequence;

[0187] S62. Combine the running state sequence before the execution of the control command, the set of control commands, all instant reward values ​​and the state comparison sequence into a state transition group;

[0188] S63. Based on the instantaneous reward value and state comparison sequence in the state transition group, perform time difference error calculation operation to generate time difference sequence;

[0189] S64. Perform Min-Max normalization on the time difference sequence to obtain the priority score value as the priority weight, and bind it to the corresponding state transition group. Write the state transition group containing the priority weight into the empirical data set.

[0190] In this embodiment, S7 specifically includes:

[0191] S71. Sort the state transition groups in the empirical data set according to priority weights, and select the training sample set from the sorting results according to the preset sampling ratio using the priority empirical replay mechanism.

[0192] S72. Structurally decompose each state transition group in the training sample set to construct a running state set, a control instruction set, an instant reward set, and a state comparison set respectively.

[0193] S73. Construct a supervision signal based on the set of running states and the set of state comparisons, and update the parameters of the DLinear model using the backpropagation mechanism;

[0194] S74. Based on the set of immediate rewards and the set of control instructions, the parameter update operation is performed on the policy network in the A3C algorithm through the backpropagation mechanism.

[0195] refer to Figure 3 A smart coffee machine operation status monitoring system based on reinforcement learning, comprising:

[0196] The data acquisition module is used to collect and preprocess multi-dimensional operating data of the smart coffee machine at a preset frequency to form a sequence of operating parameters;

[0197] The feature extraction module is used to perform dimensionality reduction on the running parameters at each time step in the Markov model using principal component analysis, and to extract principal component features to construct the running state sequence.

[0198] The state prediction module is used to perform multi-step prediction operations on the running state sequence through the DLinear model, construct a sliding time window to extract long-term trend components and short-term disturbance components, and use a linear predictor to generate the state prediction sequence.

[0199] The reward calculation module is used to construct a state reward function based on the feature difference between the running state sequence and the state prediction sequence, and to calculate the instantaneous reward value of each control behavior in the preset behavior library.

[0200] The instruction generation module is used to perform the control instruction selection operation based on the real-time reward value using the A3C algorithm, and to generate and execute the control instructions for each time step through the policy network.

[0201] The data storage module is used to form a state transition group by combining the running state sequence, control instructions, reward value and the running state sequence generated in real time after the control instructions are executed, and to perform time difference error calculation and priority binding operation, and write the binding result into the existing experience data set.

[0202] The feedback update module is used to select a training sample set from the experience data set according to the priority experience replay mechanism, and update the parameters of the DLinear model and the A3C algorithm through the backpropagation mechanism.

[0203] Example 1:

[0204] To verify the feasibility of this invention in practice, it was applied to the operation monitoring and strategy optimization process of a smart coffee device. Multidimensional sensor data of the device under different operating states were collected, modeled, and trained. An adaptive state monitoring process based on reinforcement learning was constructed. The improvement performance of this invention in terms of state perception accuracy, prediction accuracy, strategy selection effect, and long-term operation stability was verified.

[0205] This smart coffee device has multiple operating modes, including preheating, extraction, rinsing, steaming, and standby. Each mode corresponds to different operating parameter variation patterns. In traditional methods, the identification of operating modes mainly relies on static thresholds and logical rules, which leads to a large number of misjudgments and false alarms. Especially when the mode switches frequently or the device has been working continuously for several hours, the monitoring strategy becomes significantly less effective due to unstable parameter boundaries caused by hardware fluctuations.

[0206] To address the aforementioned issues, this embodiment uses sensors to collect multidimensional data from the coffee equipment during continuous operation, including temperature, pressure, water flow rate, current, and duration. The sampling frequency is set to once per second, and 12 hours of continuous data collection is performed to ultimately form a raw parameter sequence containing 43,200 data sets.

[0207] First, in the data preprocessing stage, the raw data was normalized, time-aligned, and noise-filtered to construct a standardized sequence of operating parameters. Next, principal component analysis was used to reduce the dimensionality of the operating parameters at each time step, extracting three principal components to construct the operating state sequence, and establishing state transition relationships based on a Markov model. Subsequently, the operating state sequence was input into the DLinear model, and trend and disturbance components were extracted within a sliding time window to predict the state trajectory for the next 15 time steps.

[0208] During the policy generation phase, based on the A3C algorithm, the instantaneous reward value of each candidate control behavior under the predicted state is calculated through a reward function. In the policy network, the optimal control instruction is selected based on the reward-driven mechanism and applied to the device operation control interface in real time. Simultaneously, the results of each control behavior are recorded, forming a state transition group. After prioritization based on TD error, the group is written into the experience replay set for continuous use in subsequent policy optimization training.

[0209] To comprehensively evaluate the performance of the proposed method, it was compared with traditional threshold methods and state recognition methods based on static decision trees. The comparison included state recognition accuracy, average prediction error, strategy selection response time, and long-term monitoring stability. The table below shows the test results statistics:

[0210] Table 1. Comparison of the effects of different methods on the status monitoring of smart coffee equipment.

[0211] Indicator Name Traditional threshold method Static decision tree method Method of the present invention Status recognition accuracy (%) 84.7 89.2 96.5 State transition response delay (ms) 480 320 105 Average Error of Trend Forecast (RMSE) 0.154 0.118 0.062 Mutation identification F1 value 0.68 0.73 0.89 Strategy selection hit rate (%) none 81.6 94.3 Policy Convergence Rounds — — 14 False alarm rate over long-term operation (%) 7.4 5.1 1.6 Total inference time of the model (ms) 143 225 177

[0212] As shown in Table 1, the method of this invention significantly outperforms traditional methods in terms of operational state identification accuracy, achieving an accuracy of 96.5%, which is approximately 7.3 percentage points higher than that of static decision trees. In trend prediction, the DLinear model achieves high-precision modeling of future states, reducing the RMSE to 0.062 and effectively improving prediction stability. Regarding control strategies, the A3C algorithm achieves a strategy hit rate of 94.3% and converges the strategy within 14 rounds, demonstrating strong learning ability and environmental adaptability. Compared to the monitoring failure problem of traditional methods in scenarios with long running times and frequent state changes, this invention reduces the false alarm rate from 7.4% to 1.6%, significantly improving the long-term stability and reliability of the state monitoring system.

[0213] In summary, this invention constructs a complete closed-loop system for monitoring operational status by introducing Markov state modeling, principal component feature extraction, DLinear trend prediction, and A3C strategy optimization algorithm. This system can effectively address the shortcomings of traditional methods in multi-state regulation, prediction bias, and model degradation, and verifies the accuracy, adaptability, and long-term robustness of this invention in practical applications.

[0214] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for monitoring the operating status of an intelligent coffee machine based on reinforcement learning, characterized in that, Includes the following steps: S1. Collect and preprocess multi-dimensional operating data of the smart coffee machine at a preset frequency to form an operating parameter sequence; S2. In the Markov model, principal component analysis is used to reduce the dimensionality of the running parameters at each time step and extract the principal component features to construct the running state sequence. S3. Perform multi-step prediction operations on the running state sequence through the DLinear model, construct a sliding time window to extract long-term trend components and short-term disturbance components, and use a linear predictor to generate the state prediction sequence. S4. Based on the feature difference between the running state sequence and the state prediction sequence, construct the state reward function and calculate the instantaneous reward value of each control behavior in the preset behavior library. S5. Using the A3C algorithm, the control instruction selection operation is performed based on the instant reward value. The control instructions for each time step are generated and executed through the policy network. S6. Combine the running state sequence, control instructions, reward value and the running state sequence generated in real time after the control instructions are executed into a state transition group, and perform time difference error calculation and priority binding operation, and write the binding result into the existing experience data set. S7. Select a training sample set from the experience data set according to the priority experience replay mechanism, and update the parameters of the DLinear model and the A3C algorithm through the backpropagation mechanism.

2. The method for monitoring the operating status of an intelligent coffee machine based on reinforcement learning according to claim 1, characterized in that, The multidimensional operating data includes temperature, fluid pressure, water flow rate, current value, and operating duration. The preprocessing includes time alignment, noise filtering, and numerical normalization operations. The sliding time window represents a continuous time segment that is slidably extracted in the operating state sequence with a fixed step size and length. The preset behavior library represents a predefined set of candidate behaviors for regulating the operating state of the coffee machine. The empirical data set represents a set of state transition groups containing priority weights that are continuously accumulated during operation.

3. The method for monitoring the operating status of an intelligent coffee machine based on reinforcement learning according to claim 1, characterized in that, S2 specifically includes: S21. In the Markov model, the sequence of operating parameters is expanded in chronological order, and the operating parameters corresponding to each time step are organized into parameter vectors. A state transition diagram is constructed based on the parameter vectors between adjacent time steps, specifically including: The sequence of operating parameters is divided into multiple time segments in chronological order, and each time segment contains a set of operating parameter values ​​arranged according to preset indicators; The values ​​of the running parameters in each time segment are combined into a parameter vector in a fixed order, forming a parameter vector sequence arranged by time step; In a Markov model, parameter vectors from adjacent time steps are paired, and the state transition relationship between each pair is recorded. Specifically, this includes: Pair up two adjacent parameter vectors in the parameter vector sequence to construct a set of state transition pairs; Each state transition pair is numbered and assigned a directed connection label, which represents the unidirectional evolution relationship between the state transition pairs; In the state transition pair, the parameter vector of the previous time step is marked as the current state node, and the parameter vector of the next time step is marked as the target state node. Based on the directed connection markers between each current state node and its corresponding target state node, the resulting state transition relationships are recorded; Use all parameter vectors as state nodes and construct a state transition graph by combining the corresponding state transition relationships. S22. Using principal component analysis, the parameter vectors are subjected to mean removal, and the correlation matrix between the mean-removed parameter vectors is calculated. The set of principal directions is extracted based on the correlation matrix. S23. Project the parameter vectors corresponding to each time step according to the principal direction set, and extract the principal component features of each time step. S24. Associate the principal component features generated at each time step in sequence according to the state transition diagram to construct a sequence of running states arranged by time step.

4. The method for monitoring the operating status of an intelligent coffee machine based on reinforcement learning according to claim 3, characterized in that, The extraction process of the principal component features specifically includes: The parameter vector sequence is constructed into a two-dimensional parameter matrix according to the parameter dimension, and each row of the parameter matrix corresponds to the parameter vector of a time step; Perform mean calculation on each column of the parameter matrix to obtain the mean vector of each parameter dimension at all time steps; Subtract the corresponding mean vector from the parameter vector in each row of the parameter matrix to generate a zero-mean matrix; The covariance matrix between each parameter dimension is calculated based on the zero-mean matrix, and the elements in the covariance matrix represent the degree of co-variance between two parameter dimensions. Perform eigenvalue decomposition on the covariance matrix to extract all eigenvalues ​​and the corresponding set of eigenvectors; Sort all feature values ​​in descending order, and select the feature vectors corresponding to the top feature values ​​in the sorting results according to the preset cumulative proportion to construct the main direction set; By projecting each parameter vector onto the principal direction set, the principal component features corresponding to each time step are obtained.

5. The method for monitoring the operating status of an intelligent coffee machine based on reinforcement learning according to claim 1, characterized in that, S3 specifically includes: S31. Construct a sliding time window of fixed length based on the running state sequence, and divide the running state sequence into multiple state segments with a preset overlapping range. The state segment represents the evolution process of the smart coffee machine's running state within the time period. S32. Input each state segment into the DLinear model, perform trend modeling and disturbance modeling operations along the time dimension, and extract the long-term trend component and short-term disturbance component respectively. S33. The long-term trend component and short-term disturbance component of each state segment are spliced ​​together in the feature dimension to form a fusion feature matrix, which represents the comprehensive evolution characteristics within the state segment. S34. Input the fused feature matrix into the linear predictor to perform multi-step prediction operations, and encode the predictions for future time. Establish a time mapping relationship between the encoding results and the time steps in the running state sequence to generate a state prediction sequence, specifically including: The fusion feature matrix is ​​divided into multiple sub-windows, each sub-window containing a fixed number of fusion feature vectors for consecutive time steps; Each sub-window is input into the linear predictor. Based on the overall changing trend of the fused feature vectors in the sub-windows, a state prediction operation with a set step size is performed to generate a set of predicted state vectors consistent with the step size. For each predicted state vector, a future time step encoding information is appended, wherein the time step encoding information represents the predicted time index corresponding to the predicted state; All predicted state vectors with time step encodings are arranged in time index order and a time mapping relationship is established with the time steps in the running state sequence to generate a state prediction sequence with the same structure as the running state sequence.

6. The method for monitoring the operating status of an intelligent coffee machine based on reinforcement learning according to claim 5, characterized in that, Specifically, S32 includes: S321. Arrange the running state vectors in the state segment in chronological order to construct a state segment matrix. Each row of the state segment matrix corresponds to the running state vector of a time step, and each column corresponds to a different feature dimension in the running state vector. S322. Treat each column of the state segment matrix as a time evolution sequence of the corresponding feature dimension. In the trend modeling channel of the DLinear model, perform a least-squares fitting operation on the state segment matrix column by column to extract the trend estimation sequence corresponding to the time evolution sequence of each feature dimension, and concatenate them column by column to generate a long-term trend matrix. Specifically, this includes: Extract each column from the state fragment matrix and construct a time evolution sequence from the state feature values ​​corresponding to that column in chronological order; For each time evolution sequence, a corresponding time index sequence is constructed. Each time index is paired with the corresponding state feature value to form a coordinate pair, and a fitting sample set for that feature dimension is constructed. Based on the fitted sample set, the slope and intercept values ​​are calculated using the least squares method. Then, based on the slope and intercept values, the trend fitting value for each time step is calculated to generate a trend estimation sequence, specifically including: For each feature dimension, calculate the mean of the time index and the mean of the state feature in the corresponding fitted sample set; Subtract the mean state feature value from the state feature value of each time step to obtain the corresponding state feature deviation value. At the same time, subtract the mean time index from the time index of each time step to obtain the corresponding time index deviation value. Multiply the state feature deviation value corresponding to each time step with the time index deviation value, and calculate the average of all product results to obtain the covariance between the feature dimension and the time index. The variance of the time index is obtained by averaging the squared results of all time index deviations. Divide the covariance by the variance to obtain the slope value of that feature dimension; The intercept value of this feature dimension is obtained by subtracting the product of the slope value and the time index mean from the mean of the state feature. Multiply the time index by the slope value and add it to the intercept value to obtain the trend fit value for the corresponding time step; Arrange all trend fit values ​​in chronological order to form a trend estimation sequence for this feature dimension; By concatenating the trend estimation sequences of all feature dimensions column by column, a long-term trend matrix with the same structure as the state fragment matrix is ​​constructed. S323. In the disturbance modeling channel, the difference between the state segment matrix and the long-term trend matrix is ​​calculated according to the element position to obtain the residual change value of each time step in each feature dimension to construct the short-term disturbance matrix. S324. Perform structural reorganization operations on the long-term trend matrix and the short-term disturbance matrix at time steps, and mark the source of the feature dimensions respectively to form the long-term trend component and the short-term disturbance component.

7. The method for monitoring the operating status of an intelligent coffee machine based on reinforcement learning according to claim 1, characterized in that, S4 specifically includes: S41. Perform time alignment processing on the running state sequence and the state prediction sequence according to the time mapping relationship to construct a state aligned sequence; S42. Perform feature difference calculation operation on each set of running state vectors and predicted state vectors in the state alignment sequence to generate a feature difference sequence; S43. According to the preset mapping rules, each feature difference vector is mapped to a state offset score value, wherein the state offset score value represents the degree of deviation between the running state vector and the predicted state vector. S44. Construct a state reward function based on the state offset score, wherein the state reward function includes a dense reward component and a sparse reward component, specifically including: The continuous rate of change and the magnitude of the prediction error are calculated based on the state offset score at each time step to form a dense reward component. Identify abrupt time steps in the state offset score value, mark abrupt time steps whose state offset score value exceeds the set reference threshold as high-risk time steps based on the identification results, and set fixed reward incentive or penalty value in high-risk time steps to form sparse reward components. The dense reward component and the sparse reward component are superimposed and fused at time steps to construct the state reward function; S45. Based on the state reward function, calculate the instantaneous reward value of each control behavior in the preset behavior library at each time step.

8. The method for monitoring the operating status of an intelligent coffee machine based on reinforcement learning according to claim 1, characterized in that, S5 specifically includes: S51. Use the A3C algorithm to initialize the parameters of the policy network; S52. Perform behavior probability calculation on the running state vector at each time step in the policy network to generate the behavior probability distribution corresponding to each time step. Perform weighted processing on the behavior probability distribution based on the immediate reward value, and select the control behavior based on the weighted result to generate a set of control instructions, specifically including: In the policy network, a forward propagation operation is performed on the running state vector at each time step to obtain the probability scores of all regulatory behaviors at the current time step and construct the behavior probability distribution. The instant reward value is used as a weighting factor, and the probability scores of the corresponding time step control behavior in the behavior probability distribution are weighted according to the preset weighting mechanism. The regulation behavior with the highest probability score in the weighted behavior probability distribution is selected as the optimal regulation behavior for that time step. Extract the behavior description content corresponding to the optimal control behavior from the preset behavior library, construct the control instructions for this time step, and summarize all control instructions into a control instruction set; S53. Execute the set of control instructions in the smart coffee machine in chronological order, and trigger the corresponding control behavior at each time step.

9. The method for monitoring the operating status of an intelligent coffee machine based on reinforcement learning according to claim 1, characterized in that, S6 specifically includes: S61. Real-time acquisition of multi-dimensional operation data corresponding to the time step of the state prediction sequence, and generation of the operation state sequence after the execution of the control command as a state comparison sequence; S62. Combine the running state sequence before the execution of the control command, the set of control commands, all instant reward values ​​and the state comparison sequence into a state transition group; S63. Based on the instantaneous reward value and state comparison sequence in the state transition group, perform time difference error calculation operation to generate time difference sequence; S64. Perform Min-Max normalization on the time difference sequence to obtain the priority score value as the priority weight, and bind it to the corresponding state transition group. Write the state transition group containing the priority weight into the empirical data set.

10. A reinforcement learning-based intelligent coffee machine operation status monitoring system, executing the reinforcement learning-based intelligent coffee machine operation status monitoring method according to any one of claims 1 to 9, characterized in that, include: The data acquisition module is used to collect and preprocess multi-dimensional operating data of the smart coffee machine at a preset frequency to form a sequence of operating parameters; The feature extraction module is used to perform dimensionality reduction on the running parameters at each time step in the Markov model using principal component analysis, and to extract principal component features to construct the running state sequence. The state prediction module is used to perform multi-step prediction operations on the running state sequence through the DLinear model, construct a sliding time window to extract long-term trend components and short-term disturbance components, and use a linear predictor to generate the state prediction sequence. The reward calculation module is used to construct a state reward function based on the feature difference between the running state sequence and the state prediction sequence, and to calculate the instantaneous reward value of each control behavior in the preset behavior library. The instruction generation module is used to perform the control instruction selection operation based on the real-time reward value using the A3C algorithm, and to generate and execute the control instructions for each time step through the policy network. The data storage module is used to form a state transition group by combining the running state sequence, control instructions, reward value and the running state sequence generated in real time after the control instructions are executed, and to perform time difference error calculation and priority binding operation, and write the binding result into the existing experience data set. The feedback update module is used to select a training sample set from the experience data set according to the priority experience replay mechanism, and update the parameters of the DLinear model and the A3C algorithm through the backpropagation mechanism.

Images