Igniter robot production line control method and system based on deep reinforcement learning

Abstract

The invention provides an igniter robot production line control method and system based on deep reinforcement learning, and relates to the technical field of robot production line control, and the method comprises the steps: collecting the real-time state data of a production line station, extracting features through a multi-scale residual neural network and a spatial pyramid pooling module, and carrying out the feature extraction; channel attention and space attention modules are combined to perform feature enhancement, a cross-modal encoder is used to perform time sequence modeling, task decomposition is performed based on hierarchical reinforcement learning, soft constraint trajectory optimization and adaptive impedance control are used to generate a station control instruction, and a multi-target Pareto evaluation and priority dual experience pool is used to perform sample screening. Network parameters are optimized and updated through a near-end strategy, and a production line optimization decision scheme is output, so that the production line control precision can be improved, the system robustness can be enhanced, and the optimization of the production efficiency is realized.

Description

Technical Field

[0001] The present invention relates to the field of robot production line control technology, and in particular to an ignition tool robot production line control method and system based on deep reinforcement learning. Background Art

[0002] The ignition tool production line is a typical discrete manufacturing system, involving multiple processes such as assembly, testing, and packaging. With the development of Industry 4.0, the ignition tool production line has gradually realized intelligent and automated upgrades. The traditional ignition tool production line mainly relies on manual operation and simple PLC control systems for production management; In recent years, with the development of artificial intelligence technology, deep reinforcement learning methods have shown good application prospects in the field of manufacturing system control. Deep reinforcement learning continuously optimizes decision-making strategies through the interaction between intelligent agents and the environment, and can handle control problems in high-dimensional state spaces and complex action spaces. However, existing technologies still lack effective feature extraction and anomaly detection mechanisms, are difficult to effectively handle multi-task collaborative control problems in production lines, and lack systematic consideration of multi-objective optimization problems. Therefore, a solution is urgently needed to solve the problems existing in the prior art. Summary of the invention

[0003] The embodiments of the present invention provide an ignition tool robot production line control method and system based on deep reinforcement learning, which can at least solve some of the problems existing in the prior art.

[0004] A first aspect of an embodiment of the present invention provides an ignition tool robot production line control method based on deep reinforcement learning, comprising: Collect real-time status data on the production line workstation and add it to the multi-scale residual neural network, extract features through the spatial pyramid pooling module to obtain multi-scale features, enhance the multi-scale features through the channel attention module and the spatial attention module, and combine the cross-modal encoder to perform time series modeling on the enhanced features to obtain a high-dimensional feature vector, perform anomaly detection on the high-dimensional feature vector through an anomaly detection algorithm based on soft clustering, remove abnormal data in combination with the Mahalanobis distance and perform denoising on the retained data through wavelet transform to obtain a standardized feature vector and add it to the pre-set dual deep Q network, calculate the state value and action advantage in combination with the action optimization module and the value evaluation module, and perform importance weighting in combination with the attention mechanism to obtain the production line action value function; The production line action value function is added to the task decomposition module based on hierarchical reinforcement learning, and the production line control task is decomposed into multiple subtasks through a hierarchical option architecture. For each subtask, the corresponding policy network is trained through a hierarchical policy value network algorithm, and a subtask action sequence is generated and added to the trajectory optimization algorithm based on soft constraints. The station control instructions corresponding to each station are generated in combination with a predictive control method based on time series difference. The station control instructions are adjusted in real time through an adaptive impedance control algorithm. The dynamic characteristic parameters of the production line are identified online in combination with the recursive least squares method, and the identification results are added to a performance prediction model based on Gaussian process regression to obtain the predicted value of the production line performance index; The predicted value of the production line performance index is added to the evaluation module based on multi-objective Pareto, the non-dominated sorting algorithm is combined to solve the non-dominated solution set and the target weight value of each optimization target is determined by the fuzzy hierarchical analysis algorithm, the comprehensive evaluation index of the production line is generated and added to the sample screening module based on the priority dual experience pool, the importance sampling algorithm is combined to assign a priority to each experience sample, the high-value training samples are determined by the confidence upper bound search method and added to the parameter update module based on proximal strategy optimization, the policy network and the value network in the parameter update module are jointly optimized by the trust region optimization algorithm and the KL divergence constraint, the exploration strategy is optimized by the entropy regularization method combined with the adaptive temperature parameter, the updated network parameters are obtained and the production line optimization decision plan is generated.

[0005] In an optional embodiment, The real-time status data of the production line workstations is collected and added to the multi-scale residual neural network. The spatial pyramid pooling module is used to extract features to obtain multi-scale features. The multi-scale features are enhanced by the channel attention module and the spatial attention module. The enhanced features are modeled in time series by combining the cross-modal encoder to obtain a high-dimensional feature vector. The high-dimensional feature vector is detected by an anomaly detection algorithm based on soft clustering. The abnormal data is removed by combining the Mahalanobis distance and the retained data is subjected to denoising by wavelet transform to obtain a standardized feature vector and add it to the pre-set dual deep Q network. The state value and action advantage are calculated by combining the action optimization module and the value evaluation module. The importance is weighted by combining the attention mechanism to obtain the production line action value function, which includes: A data acquisition network based on a layered architecture is used to collect real-time status data on the production line workstations, wherein the data acquisition network includes a bottom-level acquisition node, a middle-level data concentrator, and a top-level central controller. The bottom-level acquisition node collects force data, motion parameter data, visual data, and quality inspection data according to a preset priority, and the middle-level data concentrator is used to perform signal conditioning on the collected data. The top-level central controller completes normalization processing through a distributed preprocessing structure. The normalized data of the top-level central controller is input into a multi-scale residual neural network, and features are extracted through multiple residual learning blocks in the backbone network. The large-scale convolution kernel, the medium-scale convolution kernel and the small-scale convolution kernel configured in the parallel branch network are combined to extract global semantic features, regional structural features and local texture features respectively. The extracted features are globally pooled, four-region pooled and nine-region pooled through the spatial pyramid pooling module, and the pooling results are fused to form multi-scale features through channel rearrangement. The preset channel attention module generates channel statistical features through global average pooling and maximum pooling, and generates channel attention weights through a multi-layer perceptron with shared weights. The spatial attention module combines position attention and channel attention to generate a spatial attention weight map, and enhances the multi-scale features based on the spatial attention weight map; The enhanced features are input into the cross-modal encoder, and the multimodal features are mapped to a unified feature space using the feature encoding layer. The long short-term memory network in the bidirectional processing layer is combined to capture the sequence dependency. The attention fusion layer integrates the temporal information through the multi-head self-attention mechanism and outputs a high-dimensional feature vector. In the central controller, an anomaly detection algorithm based on soft clustering is used to perform anomaly detection on the high-dimensional feature vector, the sample probability density is estimated by a Gaussian kernel function, the category center is optimized by an expectation maximization algorithm, the abnormal data is removed in combination with the Mahalanobis distance, and the retained data is subjected to denoising processing by wavelet transform to obtain a standardized feature vector; Input the standardized feature vector into a preset dual deep Q network, map the state feature into the state value using the regression network structure in the value evaluation module, and output the action advantage value using the classification network structure in the action optimization module; The central controller calculates the importance of the action based on the attention mechanism and converts it into a weight coefficient, weights the state value and the action advantage value by importance to obtain the production line action value function, and sends the action value function to the data concentrator.

[0006] In an optional embodiment, In the central controller, the high-dimensional feature vector is subjected to anomaly detection using an anomaly detection algorithm based on soft clustering, the sample probability density is estimated by a Gaussian kernel function, the category center is optimized by an expectation maximization algorithm, the abnormal data is removed by combining the Mahalanobis distance, and the retained data is subjected to denoising by wavelet transform to obtain a standardized feature vector including: In the central controller, the high-dimensional feature vector is preprocessed in segments by a sliding window, the sliding window moves continuously on the high-dimensional feature vector, and time domain features, frequency domain features and statistical features are calculated based on the sliding window, the time domain features include peak value, mean value, variance, kurtosis and skewness, the frequency domain features include main frequency component, frequency band energy ratio and spectrum entropy, and the statistical features include quartiles, zero crossing rate and waveform factor of data distribution; In the central controller, a data sample is projected into a high-dimensional feature space by a preset Gaussian kernel function, a similarity relationship between samples is calculated, a probability density distribution diagram is constructed, a plurality of category centers are initialized by the expectation maximization algorithm, the position of the category center and the category belonging probability of each data point are iteratively optimized, the probability distribution of each data point belonging to each category is calculated, and the position of the category center is updated based on the probability distribution until a preset maximum number of updates is reached, wherein the bandwidth parameter of the Gaussian kernel function is adaptively determined by a cross-validation method; In the central controller, the Mahalanobis distance between the category center and each data point is calculated based on the position of the category center, an adaptive threshold is set according to the historical data distribution, when the Mahalanobis distance of a data point to the nearest category center exceeds the adaptive threshold, it is marked as the abnormal data, the adjustment of the adaptive threshold is triggered based on the timing constraint, and the retained data is decomposed into wavelet coefficients of different frequency bands through the wavelet transform, the wavelet coefficients are denoised by a soft threshold method, and the denoised data is normalized by centering and scaling transformation to obtain the standardized feature vector; In the central controller, data processing tasks are distributed to multiple computing nodes through a distributed parallel computing framework, the processing results of the computing nodes are summarized through a shared memory mechanism, and the intermediate calculation results with access frequency greater than a preset access threshold are stored based on a multi-level cache strategy. The abnormal data is continuously collected and an abnormal pattern library is constructed, the abnormal detection results and their verification information are recorded, and the bandwidth parameter of the Gaussian kernel function, the number of category centers and the adaptive threshold of the Mahalanobis distance are updated based on the verification information.

[0007] In an optional embodiment, The production line action value function is added to the task decomposition module based on hierarchical reinforcement learning, and the production line control task is decomposed into multiple subtasks through a hierarchical option architecture. For each subtask, the corresponding policy network is trained through a hierarchical policy value network algorithm, and a subtask action sequence is generated and added to the trajectory optimization algorithm based on soft constraints. The station control instructions corresponding to each station are generated in combination with a predictive control method based on time series difference. The station control instructions are adjusted in real time through an adaptive impedance control algorithm. The dynamic characteristic parameters of the production line are identified online by combining the recursive least squares method, and the identification results are added to a performance prediction model based on Gaussian process regression. The predicted values ​​of the production line performance indicators include: Obtaining the action value function of the production line, optimizing the weight parameters in the action value function based on historical operation data, adjusting the weight coefficients of the production efficiency, product quality, energy consumption and equipment load indicators through online iteration, and inputting the optimized action value function into the task decomposition module based on hierarchical reinforcement learning; The hierarchical option architecture of the task decomposition module is used to decompose the production line control task into the top-level task of overall production line scheduling control, the middle-level task of workstation group collaborative control, and the bottom-level task of individual workstation fine control. An option linking mechanism is established to transfer the decision result of the top-level task to the middle-level task, and transfer the decision result of the middle-level task to the bottom-level task. For each underlying task, a hierarchical strategy value network is trained. Sensor data and process parameters are converted into subtask feature vectors through the state encoding layer. The action probability distribution is output based on the subtask feature vectors through the strategy generation layer. The long-term benefits of the action sequence are predicted in combination with the value evaluation layer. The training samples are stored in the experience playback mechanism. The parameters of the hierarchical strategy value network are optimized based on the temporal difference learning method. The action sequence output by the optimized hierarchical strategy value network is input into the trajectory optimization algorithm based on soft constraints, and the obstacle avoidance constraint is established in combination with the potential field function. The action sequence is iteratively optimized by the gradient descent method to generate an optimized trajectory that satisfies the kinematic constraints, dynamic constraints and process constraints, and the optimized trajectory is decomposed into the benchmark control instructions of each station; A two-layer architecture of a state prediction model and a controller is constructed based on the benchmark control instructions, the state prediction model is used to learn the dynamic characteristics of the workstation online and predict the state evolution trajectory, the workstation control instructions are generated based on the state evolution trajectory, a dual closed-loop structure of position control and force control is established, the parameters of the controller are adjusted in real time according to the feedback of the force sensor, a dynamic model of inertia parameters, damping parameters and stiffness parameters is constructed, the parameters of the dynamic model are updated based on the sensor data by the recursive least squares method, the updated parameters are input into the performance prediction model, a mapping relationship between system parameters and performance indicators is established, the performance prediction model is continuously optimized, and the predicted values ​​of the production line performance indicators are output; Based on the predicted values ​​of the production line performance indicators, state prediction, trajectory optimization and parameter identification are performed on a parallel computing architecture. The bottom-level data cache module is used to collect and preprocess sensor data. The middle-level data processing module is used to extract features and estimate states. Performance evaluation and optimization decisions are performed through the top-level data analysis module to achieve hierarchical data management.

[0008] In an optional embodiment, The action sequence output by the optimized hierarchical strategy value network is input into the trajectory optimization algorithm based on soft constraints, and the obstacle avoidance constraint is established in combination with the potential field function. The action sequence is iteratively optimized by the gradient descent method to generate an optimized trajectory that satisfies the kinematic constraints, dynamic constraints and process constraints, and the optimized trajectory is decomposed into the benchmark control instructions of each station, including: Receive the action sequence output by the hierarchical strategy value network, extract the position vector, velocity vector, acceleration vector and attitude quaternion of each control action point, use the sliding window method to identify abnormal state points, correct the abnormal state points based on the local weighted regression algorithm, use a low-pass filter to smooth the corrected state sequence, and generate an initial trajectory sequence; The initial trajectory sequence is divided into a plurality of control intervals, a control point is set in each control interval, a trajectory parameterization model is constructed using a cubic spline basis function, joint position limits, speed limits, acceleration limits, joint torque limits and process parameter limits of the actuator are extracted to construct a soft constraint evaluation function, and the initial trajectory sequence is substituted into the soft constraint evaluation function to calculate the constraint violation degree of each control interval; A grid potential field model is established in the workspace, a static repulsive potential field is calculated for fixed obstacles, a time-varying repulsive potential field is constructed for dynamic obstacles based on velocity prediction, a Gaussian function is used to describe the attenuation characteristics of the potential field intensity with distance, a sigmoid function is used to construct a potential field transition interval at the obstacle boundary, the initial trajectory sequence is discretely sampled, and the potential field potential energy of the trajectory points is calculated by interpolation; A weighted objective function is constructed based on the degree of constraint violation of each control interval and the potential energy of the potential field of the trajectory point, and the numerical gradient of the objective function to the state variable of the control point is calculated using the central difference method. The optimization step size is adaptively adjusted in combination with the line search criterion, and the control point state is iteratively updated in the opposite direction of the gradient. The objective function value is calculated in real time until its relative change rate is less than the convergence threshold; The continuous trajectory is reconstructed by performing cubic Hermite spline interpolation on the optimized control points. The reconstructed trajectory is substituted into the soft constraint evaluation function to calculate the constraint violation. The critical point is located by using the dichotomy method for the trajectory segment that violates the constraint. Random sampling and local optimization are performed in the neighborhood of the critical point, and it is iterated until all constraint requirements are met. The optimized trajectory that meets the constraints is uniformly sampled according to the control cycle to generate a discrete instruction sequence of position, velocity, acceleration and posture. The Cartesian space instructions are converted into joint space instructions based on the kinematic model of the actuator. A coordinate transformation matrix chain is established according to the workstation layout, and the global trajectory instructions are decomposed into benchmark control instructions for the actuators of each workstation.

[0009] In an optional embodiment, The predicted value of the production line performance index is added to the evaluation module based on multi-objective Pareto, the non-dominated sorting algorithm is combined to solve the non-dominated solution set and the target weight value of each optimization target is determined by the fuzzy hierarchical analysis algorithm, the comprehensive evaluation index of the production line is generated and added to the sample screening module based on the priority dual experience pool, the importance sampling algorithm is combined to assign a priority to each experience sample, the confidence upper bound search method is combined to determine the high-value training samples and add them to the parameter update module based on the proximal strategy optimization, the policy network and the value network in the parameter update module are jointly optimized by the trust region optimization algorithm and the KL divergence constraint, the exploration strategy is optimized by the entropy regularization method combined with the adaptive temperature parameter, the updated network parameters are obtained and the production line optimization decision plan is generated, including: The predicted value of the production line performance index is added to the evaluation module based on multi-objective Pareto, the domination relationship matrix is ​​calculated for the solution set in the evaluation module, the solution with a non-dominated count of zero is identified by combining the non-dominated sorting algorithm to construct a non-dominated solution set, the non-dominated solution set is removed from the solution set as the first layer, and the domination relationship of the remaining solutions is repeatedly calculated until all solutions are stratified to obtain a hierarchical performance index set; Based on the hierarchical performance indicator set, a hierarchical structure model is established for the optimization objectives in the non-dominated solution set, a fuzzy judgment matrix is ​​constructed through a fuzzy hierarchical analysis algorithm, a consistency check is performed on the fuzzy judgment matrix and judgments that do not meet the consistency are corrected, and a target weight value of each optimization objective is determined in combination with a fuzzy comprehensive evaluation; Combining the non-dominated solution set and the target weight value to generate a comprehensive evaluation index for the production line, and adding the comprehensive evaluation index for the production line to a sample screening module based on a priority dual experience pool, wherein the sample screening module based on a priority dual experience pool includes a long-term memory pool for storing historical high-quality samples and a short-term memory pool for dynamically updating interactive samples; For the samples in the sample screening module based on the priority dual experience pool, a priority is assigned to each experience sample according to an importance sampling algorithm, wherein the priority is jointly determined by the time recency, return rarity and comprehensive evaluation index of the production line of the sample, and a sampling probability distribution is constructed based on the priority; Based on the sampling probability distribution, samples are extracted from the priority-based dual experience pool according to the sampling probability distribution through a confidence upper bound search method, the expected return value and prediction uncertainty of the samples are calculated based on the value network, the expected return value and prediction uncertainty are weighted to construct a confidence upper bound evaluation index, and high-value training samples are determined according to the confidence upper bound evaluation index; The high-value training samples are added to the parameter update module based on proximal policy optimization, the KL divergence between the new and old distributions of the policy network is calculated by the trust region optimization algorithm and the deviation constraint is set, the policy network and the value network in the parameter update module are jointly optimized under the KL divergence constraint, and the knowledge transfer is completed through the shared feature extraction layer; The exploration strategy is optimized by combining the entropy regularization method with adaptive temperature parameters, an entropy regularization term is constructed based on the entropy value of the strategy distribution, the entropy regularization term is added to the parameter optimization objective, the temperature parameters are adaptively adjusted according to the degree of improvement of the optimization objective, a production line optimization decision plan is generated based on the updated network parameters, the production line optimization decision plan is deployed to the production line control system, new interaction data is collected to update the priority-based dual experience pool, and the optimization is repeated until convergence.

[0010] In an optional embodiment, The high-value training samples are added to the parameter update module based on proximal policy optimization, the KL divergence between the new and old distributions of the policy network is calculated by the trust region optimization algorithm and the deviation constraint is set, the policy network and the value network in the parameter update module are jointly optimized under the KL divergence constraint, and the knowledge transfer is completed through the shared feature extraction layer, including: Obtain high-value training samples and perform one-hot encoding conversion on discrete data in the high-value training samples, perform maximum and minimum value scaling on continuous data, obtain a standardized state vector and add it to a pre-set parameter update module based on proximal strategy optimization; Based on the standardized state vector, a shared feature extraction layer including a three-layer fully connected network is constructed, wherein the first layer of the shared feature extraction layer receives the standardized state vector, converts the standardized state vector into a first intermediate feature through a ReLU activation function, the first intermediate feature is passed through a second layer of the fully connected network to obtain a second intermediate feature, and the second intermediate feature is passed through a third layer of the fully connected network to obtain a low-dimensional feature vector; Based on the low-dimensional feature vector, the output layer of the policy network and the output layer of the value network are constructed. The output layer of the policy network maps the low-dimensional feature vector to a new probability distribution of the control action through a Softmax function. The output layer of the value network maps the low-dimensional feature vector to a state value through a linear transformation. The action probability distribution corresponding to the current policy network parameters is recorded as the old probability distribution. A training batch is extracted from the standardized state vector and input into the shared feature extraction layer. A new action probability distribution is output through the policy network. The KL divergence between the new action probability distribution and the old probability distribution is calculated based on a trust region optimization algorithm. The advantage value is calculated based on the difference between the state value output by the value network and the actual return, the parameters of the policy network are updated based on the advantage value, the parameters of the value network are updated in combination with the time series difference error, and the parameters of the policy network and the value network are updated alternately under the condition that the KL divergence satisfies the preset threshold constraint; The second intermediate feature of the policy network is input into the prediction layer to obtain the predicted state value, and the prediction error between the predicted state value and the actual state value is back-propagated to the shared feature extraction layer; the second intermediate feature of the value network is input into the reconstruction layer to obtain the reconstructed state vector, and the reconstruction error between the reconstructed state vector and the standardized state vector is back-propagated to the shared feature extraction layer, and the knowledge transfer between the policy network and the value network is completed through the back-propagation of the prediction error and the reconstruction error.

[0011] A second aspect of an embodiment of the present invention provides an ignition tool robot production line control system based on deep reinforcement learning, comprising: The first unit is used to collect real-time status data on the production line workstation and add it to the multi-scale residual neural network, extract features through the spatial pyramid pooling module to obtain multi-scale features, enhance the multi-scale features through the channel attention module and the spatial attention module, and perform time series modeling on the enhanced features in combination with the cross-modal encoder to obtain a high-dimensional feature vector, perform anomaly detection on the high-dimensional feature vector through an anomaly detection algorithm based on soft clustering, remove abnormal data in combination with the Mahalanobis distance and perform denoising on the retained data through wavelet transform, obtain a standardized feature vector and add it to a pre-set dual deep Q network, calculate the state value and action advantage in combination with the action optimization module and the value evaluation module, and perform importance weighting in combination with the attention mechanism to obtain the production line action value function; The second unit is used to add the production line action value function to the task decomposition module based on hierarchical reinforcement learning, decompose the production line control task into multiple subtasks through a hierarchical option architecture, and for each subtask, train the corresponding policy network through a hierarchical policy value network algorithm, generate a subtask action sequence and add it to the trajectory optimization algorithm based on soft constraints, generate the workstation control instructions corresponding to each workstation in combination with a predictive control method based on time series difference, adjust the workstation control instructions in real time through an adaptive impedance control algorithm, and identify the dynamic characteristic parameters of the production line online in combination with the recursive least squares method, and add the identification results to the performance prediction model based on Gaussian process regression to obtain the predicted value of the production line performance index; The third unit is used to add the predicted value of the production line performance index to the evaluation module based on multi-objective Pareto, solve the non-dominated solution set in combination with the non-dominated sorting algorithm and determine the target weight value of each optimization target through the fuzzy hierarchical analysis algorithm, generate a comprehensive evaluation index of the production line and add it to the sample screening module based on the priority dual experience pool, assign a priority to each experience sample in combination with the importance sampling algorithm, determine the high-value training samples in combination with the confidence upper bound search method and add them to the parameter update module based on proximal strategy optimization, jointly optimize the policy network and the value network in the parameter update module through the trust region optimization algorithm and the KL divergence constraint, optimize the exploration strategy through the entropy regularization method combined with the adaptive temperature parameter, obtain the updated network parameters and generate the production line optimization decision plan.

[0012] According to a third aspect of the embodiments of the present invention, An electronic device is provided, comprising: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0013] A fourth aspect of the embodiments of the present invention is: A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0014] In the present invention, feature extraction and enhancement are performed through a multi-scale residual neural network combined with a spatial pyramid pooling module and an attention mechanism, and a cross-modal encoder is used for time series modeling. Combined with anomaly detection based on soft clustering and wavelet denoising, the key feature information of the production line status can be effectively extracted, the data quality can be improved, and a reliable data basis can be provided for subsequent decision-making. A task decomposition method based on hierarchical reinforcement learning is adopted, and a complex control task is decomposed into multiple subtasks through a hierarchical option architecture. Combined with soft constraint trajectory optimization and temporal difference predictive control, accurate hierarchical execution of production line control is achieved, control accuracy and response speed are improved, and the robustness and adaptability of the system are enhanced. A sample screening mechanism based on multi-objective Pareto evaluation and priority dual experience pools is used, and network parameters are updated in combination with proximal strategy optimization and trust region constraints. The exploration strategy is optimized by an entropy regularization method of an adaptive temperature parameter, and continuous optimization and performance improvement of the control strategy are achieved, thereby improving the overall operating efficiency of the production line and the stability of product quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a flow chart of a method for controlling an ignition tool robot production line based on deep reinforcement learning according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of an ignition tool robot production line control system based on deep reinforcement learning in an embodiment of the present invention. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0017] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0018] Figure 1 FIG. 1 is a flow chart of a method for controlling an ignition tool robot production line based on deep reinforcement learning according to an embodiment of the present invention. Figure 1 As shown, the method includes: S1. Collect real-time status data on the production line workstation and add it to the multi-scale residual neural network, extract features through the spatial pyramid pooling module to obtain multi-scale features, enhance the multi-scale features through the channel attention module and the spatial attention module, combine the cross-modal encoder to perform time series modeling on the enhanced features, and obtain a high-dimensional feature vector, perform anomaly detection on the high-dimensional feature vector through an anomaly detection algorithm based on soft clustering, remove abnormal data in combination with the Mahalanobis distance and perform denoising on the retained data through wavelet transform, obtain a standardized feature vector and add it to the pre-set dual deep Q network, calculate the state value and action advantage in combination with the action optimization module and the value evaluation module, and perform importance weighting in combination with the attention mechanism to obtain the production line action value function; The spatial pyramid pooling module is a method for dividing an input feature map into regions of different sizes and performing a pooling operation on each region to generate a fixed-length feature vector with multi-scale information. The cross-modal encoder is a neural network module that can map data from different modalities into a shared representation space, and is used to achieve joint learning and fusion of cross-modal data. The anomaly detection algorithm based on soft clustering is a method that uses the probability of a data point belonging to multiple cluster centers to evaluate its degree of anomaly, which helps to capture anomalies in data distribution more carefully. The Mahalanobis distance is a statistical method for measuring the distance between data points, which can take into account the correlation and scale differences of data distribution. The dual deep Q network is a reinforcement learning algorithm that stabilizes the deep Q learning process and improves the policy optimization effect by introducing a target network and a behavior network. The action optimization module is a mechanism for selecting the optimal action according to the action value function under a given state, which is used in the decision-making process in reinforcement learning. The production line action value function is a mathematical function that quantifies the impact of different operations or strategies in a production line on the overall efficiency or revenue, and is used to optimize the operation of the production line.

[0019] In an optional embodiment, The real-time status data of the production line workstations is collected and added to the multi-scale residual neural network. The spatial pyramid pooling module is used to extract features to obtain multi-scale features. The multi-scale features are enhanced by the channel attention module and the spatial attention module. The enhanced features are modeled in time series by combining the cross-modal encoder to obtain a high-dimensional feature vector. The high-dimensional feature vector is detected by an anomaly detection algorithm based on soft clustering. The abnormal data is removed by combining the Mahalanobis distance and the retained data is subjected to denoising by wavelet transform to obtain a standardized feature vector and add it to the pre-set dual deep Q network. The state value and action advantage are calculated by combining the action optimization module and the value evaluation module. The importance is weighted by combining the attention mechanism to obtain the production line action value function, which includes: A data acquisition network based on a layered architecture is used to collect real-time status data on the production line workstations, wherein the data acquisition network includes a bottom-level acquisition node, a middle-level data concentrator, and a top-level central controller. The bottom-level acquisition node collects force data, motion parameter data, visual data, and quality inspection data according to a preset priority, and the middle-level data concentrator is used to perform signal conditioning on the collected data. The top-level central controller completes normalization processing through a distributed preprocessing structure. The normalized data of the top-level central controller is input into a multi-scale residual neural network, and features are extracted through multiple residual learning blocks in the backbone network. The large-scale convolution kernel, the medium-scale convolution kernel and the small-scale convolution kernel configured in the parallel branch network are combined to extract global semantic features, regional structural features and local texture features respectively. The extracted features are globally pooled, four-region pooled and nine-region pooled through the spatial pyramid pooling module, and the pooling results are fused to form multi-scale features through channel rearrangement. The preset channel attention module generates channel statistical features through global average pooling and maximum pooling, and generates channel attention weights through a multi-layer perceptron with shared weights. The spatial attention module combines position attention and channel attention to generate a spatial attention weight map, and enhances the multi-scale features based on the spatial attention weight map; The enhanced features are input into the cross-modal encoder, and the multimodal features are mapped to a unified feature space using the feature encoding layer. The long short-term memory network in the bidirectional processing layer is combined to capture the sequence dependency. The attention fusion layer integrates the temporal information through the multi-head self-attention mechanism and outputs a high-dimensional feature vector. In the central controller, an anomaly detection algorithm based on soft clustering is used to perform anomaly detection on the high-dimensional feature vector, the sample probability density is estimated by a Gaussian kernel function, the category center is optimized by an expectation maximization algorithm, the abnormal data is removed in combination with the Mahalanobis distance, and the retained data is subjected to denoising processing by wavelet transform to obtain a standardized feature vector; Input the standardized feature vector into a preset dual deep Q network, map the state feature into the state value using the regression network structure in the value evaluation module, and output the action advantage value using the classification network structure in the action optimization module; The central controller calculates the importance of the action based on the attention mechanism and converts it into a weight coefficient, weights the state value and the action advantage value by importance to obtain the production line action value function, and sends the action value function to the data concentrator.

[0020] The force data refers to the force and tactile information related to the contact between objects collected by sensors or measuring devices, including the magnitude, direction, distribution, etc. of the force, which is used to analyze physical interactions or achieve precise control. The parallel branch network is a design that uses multiple parallel branches in a neural network architecture to process different features or tasks, which improves the expressive power and computational efficiency of the model. The channel rearrangement is a method of rearranging the order of feature channels in a neural network to optimize feature representation and network performance. The expectation maximization algorithm is an iterative optimization algorithm that gradually approaches the data generation process by alternately estimating the distribution of latent variables and optimizing model parameters.

[0021] A data acquisition network based on a layered architecture is used to collect real-time status data on the production line workstations. The data acquisition network includes bottom-level acquisition nodes, middle-level data concentrators, and top-level central controllers. The bottom-level acquisition nodes collect force data, motion parameter data, visual data, and quality inspection data according to preset priorities. For example, force data includes cutting force and torque during workpiece processing; motion parameter data includes machine tool spindle speed, feed speed, etc.; visual data includes workpiece surface quality, processing accuracy, etc.; quality inspection data includes dimensional error, surface roughness, etc. The middle-level data concentrator performs signal conditioning on the collected data, including filtering, amplification, analog-to-digital conversion, and other processing. The top-level central controller completes normalization processing through a distributed preprocessing structure, unifying different types of data into the same numerical range for easy subsequent processing.

[0022] The normalized data is input into a multi-scale residual neural network, which consists of a backbone network and a parallel branch network. The backbone network contains multiple residual learning blocks, each of which consists of a convolution layer, a batch normalization layer, and an activation function layer to extract the basic features of the data. The parallel branch network is configured with large-scale convolution kernels (such as 7x7), medium-scale convolution kernels (such as 5x5), and small-scale convolution kernels (such as 3x3), which are used to extract global semantic features, regional structural features, and local texture features, respectively. For example, for visual data, a large-scale convolution kernel can capture the overall shape of the workpiece, a medium-scale convolution kernel can identify the structure of key parts, and a small-scale convolution kernel can detect subtle surface defects.

[0023] The extracted features are pooled at multiple scales through the spatial pyramid pooling module, which includes three levels: global pooling, four-region pooling, and nine-region pooling. Global pooling performs average pooling on the entire feature map to obtain a global descriptor; four-region pooling divides the feature map into four 2x2 regions and performs average pooling on each region; nine-region pooling divides the feature map into nine 3x3 regions and performs average pooling on each region. The pooling results are fused through channel rearrangement to form multi-scale features.

[0024] The channel attention module and the spatial attention module are introduced. The channel attention module generates channel statistical features through global average pooling and maximum pooling, and generates channel attention weights through a multi-layer perceptron with shared weights. The spatial attention module combines position attention and channel attention to generate a spatial attention weight map. Specifically, position attention generates attention weights by calculating the correlation between different positions in the feature map, while channel attention focuses on the importance between different channels. Based on the generated spatial attention weight map, multi-scale features are enhanced to highlight the information of important areas and channels.

[0025] The enhanced features are input into the cross-modal encoder, which includes a feature encoding layer, a bidirectional processing layer, and an attention fusion layer. The feature encoding layer uses a fully connected network to map multimodal features to a unified feature space to solve the heterogeneity problem between different modal data. The bidirectional processing layer adopts a long short-term memory network (LSTM) structure, taking into account both forward and reverse temporal information to capture sequence dependencies. The attention fusion layer integrates temporal information through a multi-head self-attention mechanism, where each attention head focuses on a different feature subspace and finally outputs a high-dimensional feature vector.

[0026] In order to detect abnormal data, the high-dimensional feature vector is processed by the anomaly detection algorithm based on soft clustering in the central controller, the sample probability density is estimated by the Gaussian kernel function, and then the category center is optimized using the expectation maximization algorithm. Specifically, multiple category centers are initialized, the probability of each sample belonging to each category is iteratively calculated, and the category center is updated until convergence. The Mahalanobis distance is combined to remove abnormal data, and the Mahalanobis distance from each sample to the category center is calculated. Samples whose distance exceeds the threshold are marked as abnormal and removed. The retained data is denoised by wavelet transform, and the appropriate wavelet basis function and decomposition layer number are selected. The signal is decomposed at multiple scales, the high-frequency noise component is removed, and the standardized feature vector is reconstructed.

[0027] The standardized feature vector is input into the pre-set dual deep Q network, which includes a value assessment module and an action optimization module. The value assessment module uses a regression network structure to map state features to state values. For example, a multi-layer fully connected network is used, and nonlinear activation functions (such as ReLU) are added between each layer. The last layer outputs a scalar value to represent the state value. The action optimization module uses a classification network structure to output the action advantage value. This module uses a multi-layer convolutional network to extract features and outputs the advantage value of each possible action through a fully connected layer.

[0028] The central controller calculates the importance of actions based on the attention mechanism and converts it into a weight coefficient. Specifically, the self-attention mechanism is used to calculate the correlation between different actions, generate an attention weight matrix, and multiply the attention weight by the action advantage value to obtain the weighted action advantage value. The state value and the weighted action advantage value are added to obtain the final production line action value function. The central controller sends the action value function to the data concentrator to guide the optimization control of the production line.

[0029] In this embodiment, multi-scale feature extraction of production line status data is realized through a multi-scale residual neural network and a spatial pyramid pooling module, which can simultaneously capture global semantic information, regional structural features and local texture details, improve the feature expression ability and the generalization performance of the model, introduce a channel attention module and a spatial attention module to enhance the features, highlight the information of important areas and channels, improve the model's perception of key features, and help improve the accuracy and sensitivity of anomaly detection. A cross-modal encoder is used for time series modeling, combined with a long short-term memory network and a multi-head self-attention mechanism, which effectively captures the temporal dependency between different modal data, improves the model's modeling ability for complex time series patterns, and helps to more accurately predict the future state of the production line.

[0030] In an optional embodiment, In the central controller, the high-dimensional feature vector is subjected to anomaly detection using an anomaly detection algorithm based on soft clustering, the sample probability density is estimated by a Gaussian kernel function, the category center is optimized by an expectation maximization algorithm, the abnormal data is removed by combining the Mahalanobis distance, and the retained data is subjected to denoising by wavelet transform to obtain a standardized feature vector including: In the central controller, the high-dimensional feature vector is preprocessed in segments by a sliding window, the sliding window moves continuously on the high-dimensional feature vector, and time domain features, frequency domain features and statistical features are calculated based on the sliding window, the time domain features include peak value, mean value, variance, kurtosis and skewness, the frequency domain features include main frequency component, frequency band energy ratio and spectrum entropy, and the statistical features include quartiles, zero crossing rate and waveform factor of data distribution; In the central controller, a data sample is projected into a high-dimensional feature space by a preset Gaussian kernel function, a similarity relationship between samples is calculated, a probability density distribution diagram is constructed, a plurality of category centers are initialized by the expectation maximization algorithm, the position of the category center and the category belonging probability of each data point are iteratively optimized, the probability distribution of each data point belonging to each category is calculated, and the position of the category center is updated based on the probability distribution until a preset maximum number of updates is reached, wherein the bandwidth parameter of the Gaussian kernel function is adaptively determined by a cross-validation method; In the central controller, the Mahalanobis distance between the category center and each data point is calculated based on the position of the category center, an adaptive threshold is set according to the historical data distribution, when the Mahalanobis distance of a data point to the nearest category center exceeds the adaptive threshold, it is marked as the abnormal data, the adjustment of the adaptive threshold is triggered based on the timing constraint, and the retained data is decomposed into wavelet coefficients of different frequency bands through the wavelet transform, the wavelet coefficients are denoised by a soft threshold method, and the denoised data is normalized by centering and scaling transformation to obtain the standardized feature vector; In the central controller, data processing tasks are distributed to multiple computing nodes through a distributed parallel computing framework, the processing results of the computing nodes are summarized through a shared memory mechanism, and the intermediate calculation results with access frequency greater than a preset access threshold are stored based on a multi-level cache strategy. The abnormal data is continuously collected and an abnormal pattern library is constructed, the abnormal detection results and their verification information are recorded, and the bandwidth parameter of the Gaussian kernel function, the number of category centers and the adaptive threshold of the Mahalanobis distance are updated based on the verification information.

[0031] The category center refers to the cluster center of each category in the classification task in the feature space, which is used to measure the relationship between samples and categories. The category belonging probability is a numerical value that measures the possibility that a sample belongs to a specific category, which is used as a decision basis for classification problems.

[0032] The anomaly detection algorithm based on soft clustering preprocesses the input high-dimensional feature vector. The data is segmented by setting a sliding window of 1000 data points and moving the window 100 data points each time. Time domain features are extracted in each window, including calculating the peak, mean, variance, kurtosis and skewness of the signal; frequency domain features are extracted at the same time, the main frequency component is obtained through fast Fourier transform, the energy proportion of different frequency bands is calculated, and the spectrum entropy value is calculated; in addition, statistical features are extracted, including the quartile distribution of the data, the frequency of the signal crossing the zero point, and the waveform factor.

[0033] After feature extraction, the Gaussian kernel function is used for similarity calculation. The bandwidth parameter of the kernel function is determined by grid search combined with five-fold cross validation. The search range is set to 0.1 to 10, and the step size is 0.1. After projecting the data samples into the high-dimensional feature space, a probability density distribution diagram between samples is constructed. Five category centers are initially set, and the category center position and sample belonging probability are iteratively optimized by the expectation maximization algorithm. The number of iterations is set to 100. In each iteration, the probability distribution of each data point belonging to each category is calculated, and the category center position is updated based on the probability distribution.

[0034] After determining the category center, calculate the Mahalanobis distance of each data point to the nearest category center. Based on the distance distribution of historical data, set the initial threshold to the distance mean plus two standard deviations. When the Mahalanobis distance of a data point exceeds the threshold, mark it as an outlier. Update the threshold based on the newly added data every 1,000 data points, and use the db4 wavelet basis function to perform a three-layer decomposition on the retained data to obtain the wavelet coefficients of different frequency bands. Use the soft threshold method to reduce the noise of the wavelet coefficients, and set the threshold to 0.6 times the standard deviation of the wavelet coefficients. Subtract the mean from the denoised data and divide it by the standard deviation for standardization.

[0035] In the distributed computing implementation, the Spark distributed framework is used to divide the data into 10MB blocks and distribute them to different computing nodes. The calculation results of each node are summarized through shared memory. For intermediate calculation results with an access frequency of more than 10 times per second, multi-level cache is used for storage. The system continuously collects abnormal data to build an abnormal pattern library and records verification indicators such as the accuracy and recall rate of anomaly detection. When the verification indicator is lower than the preset threshold, the parameter update mechanism is triggered, including adjusting the kernel function bandwidth, the number of categories, and the distance threshold.

[0036] In this embodiment, the feature expression ability is improved through sliding window segmentation and multi-feature extraction, the Gaussian kernel function is combined to achieve accurate modeling of data distribution, the kernel function parameters are adaptively determined to improve the model generalization performance, the category center position is optimized based on the expectation maximization algorithm, the Mahalanobis distance and adaptive threshold are combined to achieve dynamic adjustment of anomaly detection, the quality of feature vectors is improved through wavelet transform denoising, the processing efficiency is improved by distributed parallel computing, the data access performance is optimized by a multi-level cache mechanism, and the continuous update and verification mechanism of the abnormal pattern library ensures the reliability of the detection results and the continuous optimization of the model.

[0037] S2. Add the production line action value function to the task decomposition module based on hierarchical reinforcement learning, decompose the production line control task into multiple subtasks through a hierarchical option architecture, and for each subtask, train the corresponding policy network through a hierarchical policy value network algorithm, generate a subtask action sequence and add it to the trajectory optimization algorithm based on soft constraints, and generate the workstation control instructions corresponding to each workstation in combination with a predictive control method based on time series difference, adjust the workstation control instructions in real time through an adaptive impedance control algorithm, and identify the dynamic characteristic parameters of the production line online in combination with the recursive least squares method, and add the identification results to the performance prediction model based on Gaussian process regression to obtain the predicted value of the production line performance index; The hierarchical option architecture is a design framework for hierarchically organizing options, which improves the flexibility and efficiency of the system by decomposing decisions into different levels. The soft-constraint-based trajectory optimization algorithm is a path planning method that balances trajectory performance and constraint feasibility by introducing flexible constraints in the optimization process. The temporal difference-based predictive control method is an algorithm that combines temporal difference estimation and model predictive control, and is used to optimize the dynamic evolution of system states in real time. The adaptive impedance control algorithm is a method for dynamically adjusting system impedance parameters, which is used to enhance the flexibility and stability of the interaction between a robot or a robotic arm and the environment. The recursive least squares method is a parameter estimation algorithm based on recursive updates, which can quickly process parameter changes and data flows of dynamic systems.

[0038] In an optional embodiment, The production line action value function is added to the task decomposition module based on hierarchical reinforcement learning, and the production line control task is decomposed into multiple subtasks through a hierarchical option architecture. For each subtask, the corresponding policy network is trained through a hierarchical policy value network algorithm, and a subtask action sequence is generated and added to the trajectory optimization algorithm based on soft constraints. The station control instructions corresponding to each station are generated in combination with a predictive control method based on time series difference. The station control instructions are adjusted in real time through an adaptive impedance control algorithm. The dynamic characteristic parameters of the production line are identified online by combining the recursive least squares method, and the identification results are added to a performance prediction model based on Gaussian process regression. The predicted values ​​of the production line performance indicators include: Obtaining the action value function of the production line, optimizing the weight parameters in the action value function based on historical operation data, adjusting the weight coefficients of the production efficiency, product quality, energy consumption and equipment load indicators through online iteration, and inputting the optimized action value function into the task decomposition module based on hierarchical reinforcement learning; The hierarchical option architecture of the task decomposition module is used to decompose the production line control task into the top-level task of overall production line scheduling control, the middle-level task of workstation group collaborative control, and the bottom-level task of individual workstation fine control. An option linking mechanism is established to transfer the decision result of the top-level task to the middle-level task, and transfer the decision result of the middle-level task to the bottom-level task. For each underlying task, a hierarchical strategy value network is trained. Sensor data and process parameters are converted into subtask feature vectors through the state encoding layer. The action probability distribution is output based on the subtask feature vectors through the strategy generation layer. The long-term benefits of the action sequence are predicted in combination with the value evaluation layer. The training samples are stored in the experience playback mechanism. The parameters of the hierarchical strategy value network are optimized based on the temporal difference learning method. The action sequence output by the optimized hierarchical strategy value network is input into the trajectory optimization algorithm based on soft constraints, and the obstacle avoidance constraint is established in combination with the potential field function. The action sequence is iteratively optimized by the gradient descent method to generate an optimized trajectory that satisfies the kinematic constraints, dynamic constraints and process constraints, and the optimized trajectory is decomposed into the benchmark control instructions of each station; A two-layer architecture of a state prediction model and a controller is constructed based on the benchmark control instructions, the state prediction model is used to learn the dynamic characteristics of the workstation online and predict the state evolution trajectory, the workstation control instructions are generated based on the state evolution trajectory, a dual closed-loop structure of position control and force control is established, the parameters of the controller are adjusted in real time according to the feedback of the force sensor, a dynamic model of inertia parameters, damping parameters and stiffness parameters is constructed, the parameters of the dynamic model are updated based on the sensor data by the recursive least squares method, the updated parameters are input into the performance prediction model, a mapping relationship between system parameters and performance indicators is established, the performance prediction model is continuously optimized, and the predicted values ​​of the production line performance indicators are output; Based on the predicted values ​​of the production line performance indicators, state prediction, trajectory optimization and parameter identification are performed on a parallel computing architecture. The bottom-level data cache module is used to collect and preprocess sensor data. The middle-level data processing module is used to extract features and estimate states. Performance evaluation and optimization decisions are performed through the top-level data analysis module to achieve hierarchical data management.

[0039] The state encoding layer is a module in a neural network or algorithm model that encodes input state information into a specific feature representation for easy subsequent processing and analysis. The benchmark control instruction is an instruction signal used as a reference standard in the system to guide actual control execution and compare deviations. The dual closed-loop structure is a control system design that combines an inner loop and an outer loop to enhance stability and dynamic performance.

[0040] The action value function of the production line is obtained, and the weight parameters in the function are optimized based on the historical operation data. The system collects historical data of multiple indicators including production efficiency, product quality, energy consumption and equipment load. The weight coefficients of the indicators are continuously adjusted through online iteration. For example, the weights of production efficiency, product quality, energy consumption and equipment load can be set to 0.3, 0.3, 0.2 and 0.2 respectively at the beginning. As the production line runs, these weights are dynamically adjusted according to the actual situation. Assume that after a certain iteration, the weights are adjusted to 0.25, 0.35, 0.2 and 0.2, indicating that the system pays more attention to product quality. The optimized action value function will serve as an important basis for the subsequent task decomposition.

[0041] The optimized action value function is input into the task decomposition module based on hierarchical reinforcement learning. The task decomposition module adopts a hierarchical option architecture to decompose the production line control task into three levels: the top-level production line overall scheduling control task, the middle-level workstation group collaborative control task, and the bottom-level single workstation fine control task. In this process, the system establishes an option linking mechanism to ensure that the decision results can be passed from the top level to the middle level, and then from the middle level to the bottom level. For example, the top-level task may determine the production plan within a certain period of time, and affect the specific operating parameters of each workstation in the bottom-level task by affecting the collaborative strategy of each workstation group in the middle-level task.

[0042] For each underlying task, the system trains a hierarchical policy value network, which contains three key components: state encoding layer, policy generation layer, and value assessment layer. The state encoding layer is responsible for converting sensor data and process parameters into subtask feature vectors. For example, for a welding station, sensor data may include parameters such as temperature, pressure, and current, and process parameters may include welding speed, welding time, etc. After encoding, a 128-dimensional feature vector is formed. The policy generation layer outputs the action probability distribution based on this feature vector. For example, the probability distribution of adjusting the welding current is: the probability of increasing by 10% is 0.2, the probability of remaining unchanged is 0.6, and the probability of decreasing by 10% is 0.2. The value assessment layer predicts the long-term benefits of the action sequence, such as predicting the improvement in product qualification rate that may be brought about by the current action sequence.

[0043] The system stores the training samples in the experience replay mechanism and uses the temporal difference learning method to optimize the parameters of the hierarchical policy value network. The experience replay mechanism can store the most recent 10,000 training samples and randomly select 1,000 for batch learning each time. The temporal difference learning method continuously adjusts the network parameters by comparing the predicted value with the actual reward obtained, thereby improving the prediction accuracy.

[0044] The action sequence output by the optimized hierarchical strategy value network is input into the trajectory optimization algorithm based on soft constraints. Obstacle avoidance constraints are established in combination with the potential field function to ensure that the generated trajectory can avoid obstacles in the workspace. Through the gradient descent method, the system iteratively optimizes the action sequence to generate an optimized trajectory that satisfies kinematic constraints, dynamic constraints, and process constraints. For example, for a handling robot, the optimized trajectory must not only meet the range of motion limitations of the robot joints, but also consider the limitations of acceleration and torque, while also meeting process requirements such as stability and smoothness. Finally, the system decomposes the optimized trajectory into benchmark control instructions for each station.

[0045] Based on the benchmark control instructions, the system builds a two-layer architecture of state prediction model and controller. The state prediction model grasps the dynamic characteristics of the workstation through online learning and predicts the state evolution trajectory. For example, for a spraying workstation, the model can predict the change trend of coating thickness over time under the current spraying parameters. The controller generates specific workstation control instructions based on the predicted state evolution trajectory. The system adopts a dual closed-loop structure of position control and force control, and adjusts the controller parameters according to the real-time feedback of the force sensor.

[0046] The system builds a dynamic model including inertia parameters, damping parameters, and stiffness parameters. Through recursive least squares, the system continuously updates the parameters of these dynamic models based on sensor data. For example, for a robotic arm, the system may find that the stiffness parameter of one of its joints decreases slightly with the increase of usage time, so as to adjust the control strategy in time.

[0047] The updated parameters are input into the performance prediction model to establish a mapping relationship between system parameters and performance indicators. Through continuous optimization, the performance prediction model can accurately output the predicted values ​​of the production line's performance indicators, such as the product qualification rate and energy consumption within the next 8 hours.

[0048] Based on the predicted values ​​of production line performance indicators, state prediction, trajectory optimization and parameter identification are performed simultaneously on the parallel computing architecture. Data management adopts a three-layer structure: the bottom-level data cache module is responsible for the collection and preprocessing of sensor data, such as filtering and normalizing the original data; the middle-level data processing module performs feature extraction and state estimation, such as extracting key features from preprocessed data and estimating the current system state; the top-level data analysis module is responsible for performance evaluation and optimization decisions, such as evaluating system performance based on current status and historical data, and making optimization decisions.

[0049] In this embodiment, by decomposing complex control tasks into sub-tasks at different levels and training a dedicated strategy network for each sub-task, the accuracy and adaptability of the control strategy are greatly improved. A dynamic weight adjustment mechanism and online parameter identification technology are adopted to enable the control system to adapt to changes in the production environment in real time. By continuously optimizing the weight parameters of the action value function, the system can flexibly adjust the importance of various performance indicators according to actual production needs. The application of recursive least squares method enables the system to capture the dynamic changes of equipment characteristics in a timely manner, ensuring the timeliness and accuracy of the control strategy, which not only improves the control accuracy of a single workstation, but also realizes the coordinated optimization of the entire production line, significantly improving production efficiency and product quality, while effectively reducing energy consumption and equipment load.

[0050] In an optional embodiment, The action sequence output by the optimized hierarchical strategy value network is input into the trajectory optimization algorithm based on soft constraints, and the obstacle avoidance constraint is established in combination with the potential field function. The action sequence is iteratively optimized by the gradient descent method to generate an optimized trajectory that satisfies the kinematic constraints, dynamic constraints and process constraints, and the optimized trajectory is decomposed into the benchmark control instructions of each station, including: Receive the action sequence output by the hierarchical strategy value network, extract the position vector, velocity vector, acceleration vector and attitude quaternion of each control action point, use the sliding window method to identify abnormal state points, correct the abnormal state points based on the local weighted regression algorithm, use a low-pass filter to smooth the corrected state sequence, and generate an initial trajectory sequence; The initial trajectory sequence is divided into a plurality of control intervals, a control point is set in each control interval, a trajectory parameterization model is constructed using a cubic spline basis function, joint position limits, speed limits, acceleration limits, joint torque limits and process parameter limits of the actuator are extracted to construct a soft constraint evaluation function, and the initial trajectory sequence is substituted into the soft constraint evaluation function to calculate the constraint violation degree of each control interval; A grid potential field model is established in the workspace, a static repulsive potential field is calculated for fixed obstacles, a time-varying repulsive potential field is constructed for dynamic obstacles based on velocity prediction, a Gaussian function is used to describe the attenuation characteristics of the potential field intensity with distance, a sigmoid function is used to construct a potential field transition interval at the obstacle boundary, the initial trajectory sequence is discretely sampled, and the potential field potential energy of the trajectory points is calculated by interpolation; A weighted objective function is constructed based on the degree of constraint violation of each control interval and the potential energy of the potential field of the trajectory point, and the numerical gradient of the objective function to the state variable of the control point is calculated using the central difference method. The optimization step size is adaptively adjusted in combination with the line search criterion, and the control point state is iteratively updated in the opposite direction of the gradient. The objective function value is calculated in real time until its relative change rate is less than the convergence threshold; The continuous trajectory is reconstructed by performing cubic Hermite spline interpolation on the optimized control points. The reconstructed trajectory is substituted into the soft constraint evaluation function to calculate the constraint violation. The critical point is located by using the dichotomy method for the trajectory segment that violates the constraint. Random sampling and local optimization are performed in the neighborhood of the critical point, and it is iterated until all constraint requirements are met. The optimized trajectory that meets the constraints is uniformly sampled according to the control cycle to generate a discrete instruction sequence of position, velocity, acceleration and posture. The Cartesian space instructions are converted into joint space instructions based on the kinematic model of the actuator. A coordinate transformation matrix chain is established according to the workstation layout, and the global trajectory instructions are decomposed into benchmark control instructions for the actuators of each workstation.

[0051] The cubic spline basis function is a basis function for constructing a smooth curve, which is defined by a piecewise cubic polynomial and has a continuous second-order derivative. The soft constraint evaluation function is a function used to penalize or evaluate solutions that do not strictly meet the constraints in an optimization problem, taking into account flexibility and solution quality. The cubic Hermite spline interpolation is an interpolation method for constructing a smooth curve based on known point positions and derivative information, and is used for high-precision path fitting. The Cartesian space instruction refers to a control or motion instruction defined directly in the Cartesian coordinate system, which is commonly used for path planning at the end of a robot. The coordinate transformation matrix chain is a series of matrix products used to describe multi-level transformation relationships from one coordinate system to another.

[0052] Preprocess the action sequence output by the hierarchical policy value network. Receive a control action point sequence containing position vector, velocity vector, acceleration vector and attitude quaternion. Use a sliding window of length 5 to identify abnormal points by calculating the statistical characteristics of the state quantity. For the identified abnormal points, use a local weighted regression algorithm with a Gaussian kernel function to correct them, and the weight decays exponentially with the distance. Then use a Butterworth low-pass filter with a cutoff frequency of 10Hz to smooth the corrected sequence and generate the initial trajectory sequence.

[0053] The trajectory sequence is divided into several control intervals according to time. The length of each interval is 0.5 seconds, and control points are set at the endpoints of the intervals. The trajectory parameterization model is constructed using the cubic spline basis function, and the node interval of the basis function is 0.1 seconds. The joint limit information of the robot is extracted, including the position limit of plus or minus 90 degrees, the speed limit of plus or minus 180 degrees per second, the acceleration limit of plus or minus 360 degrees per square second, the torque limit of plus or minus 100 Nm, and the process parameter limit. The soft constraint evaluation function is constructed, and the initial trajectory is discretely sampled to calculate the degree of constraint violation.

[0054] A three-dimensional potential field model with a grid resolution of 0.1 meters is established. For fixed obstacles, a static repulsive potential field with an intensity of 100 and the center of the obstacle as the origin is calculated. For dynamic obstacles, a linear prediction is performed based on the current velocity to construct a time-varying repulsive potential field with a prediction time domain of 2 seconds. The attenuation of the potential field intensity with distance is described by a Gaussian function with a standard deviation of 0.5 meters. A potential field transition interval is set within 0.2 meters of the obstacle boundary, and a sigmoid function is used to achieve a smooth transition of the potential field. The trajectory sequence is discretely sampled at intervals of 0.1 seconds, and the potential energy of the potential field at the sampling point is calculated by cubic spline interpolation.

[0055] The objective function is constructed by combining the degree of constraint violation and the potential energy of the potential field, with the constraint term weight of 1.0 and the potential field term weight of 0.5. The numerical gradient of the objective function to the control point state variable is calculated using the central difference method with a step size of 0.001. The Armijo criterion is used to adaptively adjust the optimization step size, with an initial step size of 0.1. The control point state is iteratively updated in the reverse direction of the gradient, and convergence is determined when the relative change rate of the objective function is less than 0.001.

[0056] The optimized control point sequence is reconstructed by using cubic Hermite spline interpolation with an interpolation interval of 0.01 seconds. The reconstructed trajectory is substituted into the constraint evaluation function. If a trajectory segment violating the constraint is found, the critical point is located by using the dichotomy method. 100 random samplings are performed within 0.5 seconds before and after the critical point, and local gradient descent optimization is performed. This process is repeated until all constraint requirements are met.

[0057] The trajectory is decomposed, and the optimized trajectory is uniformly sampled according to the control cycle of 0.001 seconds to generate a discrete instruction sequence of position, velocity, acceleration and posture. Based on the DH parameters and Jacobian matrix of the robot, the Cartesian space instructions are converted into joint space instructions. According to the relative position and posture of the workstation, a coordinate transformation matrix chain is established to decompose the global trajectory instructions into the actuators of each workstation.

[0058] In this embodiment, the stability and continuity of the initial trajectory are significantly improved by sliding window identification and local weighted regression correction of abnormal points, combined with low-pass filtering and smoothing processing, which provides a good foundation for subsequent optimization. The cubic spline parameterized model and soft constraint evaluation function are combined with the obstacle avoidance constraint based on the potential field to construct a complete trajectory optimization framework, which can effectively balance the motion performance and safety requirements, and generate a smooth and continuous optimized trajectory that meets multiple constraints. The constraint correction mechanism of binary positioning and random sampling, combined with the precise trajectory decomposition strategy, ensures the actual executability of the optimized trajectory, improves the accuracy and efficiency of multi-station collaborative operations, and has good engineering application value.

[0059] S3. Add the predicted value of the production line performance index to the evaluation module based on multi-objective Pareto, solve the non-dominated solution set in combination with the non-dominated sorting algorithm and determine the target weight value of each optimization target through the fuzzy hierarchical analysis algorithm, generate the comprehensive evaluation index of the production line and add it to the sample screening module based on the priority dual experience pool, assign priority to each experience sample in combination with the importance sampling algorithm, determine the high-value training samples in combination with the confidence upper bound search method and add them to the parameter update module based on proximal strategy optimization, jointly optimize the policy network and the value network in the parameter update module through the trust region optimization algorithm and the KL divergence constraint, optimize the exploration strategy through the entropy regularization method combined with the adaptive temperature parameter, obtain the updated network parameters and generate the production line optimization decision plan.

[0060] The multi-objective Pareto-based evaluation module is a module that analyzes multi-objective performance by constructing a Pareto frontier, and is used to find a set of optimal solutions in multi-objective optimization. The non-dominated sorting algorithm is a sorting method that divides multi-objective solutions into multiple levels according to dominance relationships, and is often used to evaluate the pros and cons and diversity of solutions. The fuzzy hierarchical analysis algorithm is a multi-criteria decision-making method based on fuzzy logic, which supports complex decisions by constructing a fuzzy judgment matrix and weight calculation. The sample screening module based on priority dual experience pool is a screening mechanism that combines historical priority and dynamic experience pool, and is used to optimize the sample selection process and improve model training efficiency. The importance sampling algorithm is a technology that redistributes samples according to sample importance in a probability distribution, and is used to accelerate the estimation of sparse events. The confidence upper bound search method is a search algorithm that optimizes the objective function by balancing exploration and utilization, and is widely used in reinforcement learning and optimization problems. The trust region optimization algorithm is an algorithm that dynamically adjusts the search range during the optimization process to ensure that the optimized solution converges within a credible range. The entropy regularization method combined with an adaptive temperature parameter is a technology that introduces an entropy regularization term in optimization and dynamically adjusts the temperature parameter, and is used to balance model stability and diversity.

[0061] In an optional embodiment, The predicted value of the production line performance index is added to the evaluation module based on multi-objective Pareto, the non-dominated sorting algorithm is combined to solve the non-dominated solution set and the target weight value of each optimization target is determined by the fuzzy hierarchical analysis algorithm, the comprehensive evaluation index of the production line is generated and added to the sample screening module based on the priority dual experience pool, the importance sampling algorithm is combined to assign a priority to each experience sample, the confidence upper bound search method is combined to determine the high-value training samples and add them to the parameter update module based on the proximal strategy optimization, the policy network and the value network in the parameter update module are jointly optimized by the trust region optimization algorithm and the KL divergence constraint, the exploration strategy is optimized by the entropy regularization method combined with the adaptive temperature parameter, the updated network parameters are obtained and the production line optimization decision plan is generated, including: The predicted value of the production line performance index is added to the evaluation module based on multi-objective Pareto, the domination relationship matrix is ​​calculated for the solution set in the evaluation module, the solution with a non-dominated count of zero is identified by combining the non-dominated sorting algorithm to construct a non-dominated solution set, the non-dominated solution set is removed from the solution set as the first layer, and the domination relationship of the remaining solutions is repeatedly calculated until all solutions are stratified to obtain a hierarchical performance index set; Based on the hierarchical performance indicator set, a hierarchical structure model is established for the optimization objectives in the non-dominated solution set, a fuzzy judgment matrix is ​​constructed through a fuzzy hierarchical analysis algorithm, a consistency check is performed on the fuzzy judgment matrix and judgments that do not meet the consistency are corrected, and a target weight value of each optimization objective is determined in combination with a fuzzy comprehensive evaluation; Combining the non-dominated solution set and the target weight value to generate a comprehensive evaluation index for the production line, and adding the comprehensive evaluation index for the production line to a sample screening module based on a priority dual experience pool, wherein the sample screening module based on a priority dual experience pool includes a long-term memory pool for storing historical high-quality samples and a short-term memory pool for dynamically updating interactive samples; For the samples in the sample screening module based on the priority dual experience pool, a priority is assigned to each experience sample according to an importance sampling algorithm, wherein the priority is jointly determined by the time recency, return rarity and comprehensive evaluation index of the production line of the sample, and a sampling probability distribution is constructed based on the priority; Based on the sampling probability distribution, samples are extracted from the priority-based dual experience pool according to the sampling probability distribution through a confidence upper bound search method, the expected return value and prediction uncertainty of the samples are calculated based on the value network, the expected return value and prediction uncertainty are weighted to construct a confidence upper bound evaluation index, and high-value training samples are determined according to the confidence upper bound evaluation index; The high-value training samples are added to the parameter update module based on proximal policy optimization, the KL divergence between the new and old distributions of the policy network is calculated by the trust region optimization algorithm and the deviation constraint is set, the policy network and the value network in the parameter update module are jointly optimized under the KL divergence constraint, and the knowledge transfer is completed through the shared feature extraction layer; The exploration strategy is optimized by combining the entropy regularization method with adaptive temperature parameters, an entropy regularization term is constructed based on the entropy value of the strategy distribution, the entropy regularization term is added to the parameter optimization objective, the temperature parameters are adaptively adjusted according to the degree of improvement of the optimization objective, a production line optimization decision plan is generated based on the updated network parameters, the production line optimization decision plan is deployed to the production line control system, new interaction data is collected to update the priority-based dual experience pool, and the optimization is repeated until convergence.

[0062] The dominance relationship matrix is ​​a matrix used to represent the dominance relationship between solutions in multi-objective optimization, and is used to analyze and screen solution sets. The fuzzy judgment matrix is ​​a matrix that represents the relative importance of different factors through fuzzy logic, and is used for hierarchical analysis and weight calculation.

[0063] Collect production line operation data, including equipment operation parameters, product quality data, production efficiency indicators, etc. Process the data through deep learning models to obtain performance indicator predictions including capacity utilization, product qualification rate, energy consumption, etc.

[0064] Construct a dominance relationship matrix. Taking capacity utilization and energy consumption as an example, when the capacity utilization of plan A is 85% and the energy consumption is 100kWh, and the capacity utilization of plan B is 80% and the energy consumption is 120kWh, plan A is better than plan B in both objectives, and A dominates B. In this way, all non-dominated solutions are identified to form the first-level solution set. For example, in the case of 100 optimization solutions, the first level may contain 15 non-dominated solutions, which constitute the Pareto frontier.

[0065] For the determined non-dominated solution set, the system establishes a hierarchical model. Taking production line optimization as an example, the objectives can be divided into three levels: production efficiency, product quality, and resource consumption. The weights are determined through fuzzy hierarchical analysis, such as production efficiency weight 0.4, product quality weight 0.35, and resource consumption weight 0.25.

[0066] The system maintains a dual experience pool structure. The long-term memory pool can be set to 10,000 samples to store the best historical samples. The short-term memory pool has a capacity of 1,000 samples and dynamically updates the latest interaction data. The priority is calculated for each sample. For example, if the time recency of a sample is 0.9, the reward rarity is 0.8, and the comprehensive evaluation index is 0.85, then its priority may be 0.85.

[0067] Based on the sampling probability distribution of priority, the system uses the confidence upper bound search method to select training samples. Assuming that 1,000 samples are drawn from the experience pool, the expected return mean predicted by the value network is 0.75, and the prediction uncertainty is 0.1, then the confidence upper bound evaluation index is 0.85. Samples with higher evaluation indicators are selected for training.

[0068] The system jointly optimizes the policy network and the value network. By setting the trust region radius to 0.2, the network parameters are updated under the constraint that the policy distribution does not deviate from this range. At the same time, entropy regularization is introduced. When the policy improves significantly, the temperature parameter is lowered to increase utilization. When the improvement is slow, the temperature parameter is increased to increase exploration. Finally, the production line optimization decision plan is output.

[0069] In this embodiment, by combining multi-objective Pareto evaluation and fuzzy hierarchical analysis, multiple optimization objectives can be fully considered, and the importance of each objective can be flexibly adjusted according to actual production needs, so that the optimization results are more in line with the needs of actual application scenarios. A priority-based dual experience pool structure is adopted, combined with importance sampling and confidence upper bound search methods, which can effectively preserve and utilize high-value samples, improve training efficiency, accelerate optimization convergence speed, and maintain continuous accumulation of knowledge. Through the entropy regularization method of trust region constraints and adaptive temperature parameters, a dynamic balance between exploration and utilization is achieved while ensuring stable strategy updates, thereby improving the robustness and adaptability of the optimization algorithm and obtaining a more reliable production line optimization solution.

[0070] In an optional embodiment, The high-value training samples are added to the parameter update module based on proximal policy optimization, the KL divergence between the new and old distributions of the policy network is calculated by the trust region optimization algorithm and the deviation constraint is set, the policy network and the value network in the parameter update module are jointly optimized under the KL divergence constraint, and the knowledge transfer is completed through the shared feature extraction layer, including: Obtain high-value training samples and perform one-hot encoding conversion on discrete data in the high-value training samples, perform maximum and minimum value scaling on continuous data, obtain a standardized state vector and add it to a pre-set parameter update module based on proximal strategy optimization; Based on the standardized state vector, a shared feature extraction layer including a three-layer fully connected network is constructed, wherein the first layer of the shared feature extraction layer receives the standardized state vector, converts the standardized state vector into a first intermediate feature through a ReLU activation function, the first intermediate feature is passed through a second layer of the fully connected network to obtain a second intermediate feature, and the second intermediate feature is passed through a third layer of the fully connected network to obtain a low-dimensional feature vector; Based on the low-dimensional feature vector, the output layer of the policy network and the output layer of the value network are constructed. The output layer of the policy network maps the low-dimensional feature vector to a new probability distribution of the control action through a Softmax function. The output layer of the value network maps the low-dimensional feature vector to a state value through a linear transformation. The action probability distribution corresponding to the current policy network parameters is recorded as the old probability distribution. A training batch is extracted from the standardized state vector and input into the shared feature extraction layer. A new action probability distribution is output through the policy network. The KL divergence between the new action probability distribution and the old probability distribution is calculated based on a trust region optimization algorithm. The advantage value is calculated based on the difference between the state value output by the value network and the actual return, the parameters of the policy network are updated based on the advantage value, the parameters of the value network are updated in combination with the time series difference error, and the parameters of the policy network and the value network are updated alternately under the condition that the KL divergence satisfies the preset threshold constraint; The second intermediate feature of the policy network is input into the prediction layer to obtain the predicted state value, and the prediction error between the predicted state value and the actual state value is back-propagated to the shared feature extraction layer; the second intermediate feature of the value network is input into the reconstruction layer to obtain the reconstructed state vector, and the reconstruction error between the reconstructed state vector and the standardized state vector is back-propagated to the shared feature extraction layer, and the knowledge transfer between the policy network and the value network is completed through the back-propagation of the prediction error and the reconstruction error.

[0071] The one-hot encoding conversion is a technology that converts categorical data into sparse vector representation, which is used for standardized processing of model input. The shared feature extraction layer is a design that extracts common features of different tasks through a shared network layer in multi-task learning, which can improve the parameter efficiency and generalization ability of the model.

[0072] Perform data preprocessing on the acquired high-value training samples. For discrete feature data, one-hot encoding is used for conversion, and each discrete value is mapped to a binary vector, where only one position in the vector is 1 and the rest are 0. For example, for discrete features with 3 possible values, convert them into a three-dimensional vector, such as value 1 is converted to [1, 0, 0], value 2 is converted to [0, 1, 0], and value 3 is converted to [0, 0, 1]. For continuous feature data, the maximum and minimum value normalization method is used for scaling, and the values ​​are mapped to the range of 0 to 1. For example, for the original value x, the normalized value is obtained by calculating (x-minimum value) / (maximum value-minimum value). The standardized state vector is obtained through the above preprocessing.

[0073] A three-layer fully connected network is constructed as a shared feature extraction layer. The first layer contains 256 neurons, and the input is a standardized state vector. The first intermediate feature is obtained through the ReLU activation function. The second layer contains 128 neurons, which converts the first intermediate feature into the second intermediate feature. The third layer contains 64 neurons, which compresses the second intermediate feature into a low-dimensional feature vector.

[0074] In the policy network output layer, the low-dimensional feature vector is converted to an action probability distribution through the Softmax function. For example, for a scene with 4 optional actions, the output layer will generate a probability distribution of the form [0.2, 0.3, 0.4, 0.1]. In the value network output layer, the low-dimensional feature vector is mapped to a scalar state value through a linear transformation of a neuron. The action probability distribution corresponding to the current policy network parameters is recorded as the old distribution, and 32 samples are randomly selected from the standardized state vector to form a training batch, which is input into the shared feature extraction layer to obtain the new action probability distribution. The KL divergence between the new and old distributions is calculated based on the trust region optimization method.

[0075] The advantage value is calculated based on the difference between the state value output by the value network and the actual cumulative return. When the advantage value is positive, it indicates that the action is better than the average level, increasing its probability of occurrence; when the advantage value is negative, it reduces the probability of occurrence of the action. In each round of iteration, the prediction error of the value network is calculated in combination with the temporal difference method. The KL divergence threshold is set to 0.01, and the parameters of the policy network and the value network are updated alternately under this constraint.

[0076] To achieve knowledge transfer, a prediction layer and a reconstruction layer are added. The prediction layer contains one neuron, which inputs the second intermediate feature of the policy network and outputs the predicted state value. The number of neurons in the reconstruction layer is the same as the dimension of the state vector, which inputs the second intermediate feature of the value network and outputs the reconstructed state vector. The prediction error and reconstruction error are back-propagated to the shared feature extraction layer to achieve knowledge sharing between the policy network and the value network.

[0077] In this embodiment, by sharing the feature extraction layer and the bidirectional knowledge transfer mechanism, the learning efficiency of the policy network and the value network is significantly improved, the training convergence speed is accelerated, and the parameter update method of the trust region constraint is adopted to effectively prevent the strategy from deviating excessively, ensure the stability of the training process, and avoid the occurrence of strategy collapse. Combined with the dual constraints of the prediction layer and the reconstruction layer, the robustness of feature extraction is enhanced, and the generalization ability and adaptability of the model in complex environments are improved.

[0078] Figure 2 FIG. 1 is a schematic diagram of the structure of the ignition tool robot production line control system based on deep reinforcement learning according to an embodiment of the present invention. Figure 2 As shown, the system comprises: The first unit is used to collect real-time status data on the production line workstation and add it to the multi-scale residual neural network, extract features through the spatial pyramid pooling module to obtain multi-scale features, enhance the multi-scale features through the channel attention module and the spatial attention module, and perform time series modeling on the enhanced features in combination with the cross-modal encoder to obtain a high-dimensional feature vector, perform anomaly detection on the high-dimensional feature vector through an anomaly detection algorithm based on soft clustering, remove abnormal data in combination with the Mahalanobis distance and perform denoising on the retained data through wavelet transform, obtain a standardized feature vector and add it to a pre-set dual deep Q network, calculate the state value and action advantage in combination with the action optimization module and the value evaluation module, and perform importance weighting in combination with the attention mechanism to obtain the production line action value function; The second unit is used to add the production line action value function to the task decomposition module based on hierarchical reinforcement learning, decompose the production line control task into multiple subtasks through a hierarchical option architecture, and for each subtask, train the corresponding policy network through a hierarchical policy value network algorithm, generate a subtask action sequence and add it to the trajectory optimization algorithm based on soft constraints, generate the workstation control instructions corresponding to each workstation in combination with a predictive control method based on time series difference, adjust the workstation control instructions in real time through an adaptive impedance control algorithm, and identify the dynamic characteristic parameters of the production line online in combination with the recursive least squares method, and add the identification results to the performance prediction model based on Gaussian process regression to obtain the predicted value of the production line performance index; The third unit is used to add the predicted value of the production line performance index to the evaluation module based on multi-objective Pareto, solve the non-dominated solution set in combination with the non-dominated sorting algorithm and determine the target weight value of each optimization target through the fuzzy hierarchical analysis algorithm, generate a comprehensive evaluation index of the production line and add it to the sample screening module based on the priority dual experience pool, assign a priority to each experience sample in combination with the importance sampling algorithm, determine the high-value training samples in combination with the confidence upper bound search method and add them to the parameter update module based on proximal strategy optimization, jointly optimize the policy network and the value network in the parameter update module through the trust region optimization algorithm and the KL divergence constraint, optimize the exploration strategy through the entropy regularization method combined with the adaptive temperature parameter, obtain the updated network parameters and generate the production line optimization decision plan.

[0079] According to a third aspect of the embodiments of the present invention, An electronic device is provided, comprising: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0080] A fourth aspect of the embodiments of the present invention is: A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0081] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0082] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A control method for an ignition tool robot production line based on deep reinforcement learning, characterized in that: include: Collect real-time status data on the production line workstation and add it to the multi-scale residual neural network, extract features through the spatial pyramid pooling module to obtain multi-scale features, enhance the multi-scale features through the channel attention module and the spatial attention module, and combine the cross-modal encoder to perform time series modeling on the enhanced features to obtain a high-dimensional feature vector, perform anomaly detection on the high-dimensional feature vector through an anomaly detection algorithm based on soft clustering, remove abnormal data in combination with the Mahalanobis distance and perform denoising on the retained data through wavelet transform to obtain a standardized feature vector and add it to the pre-set dual deep Q network, calculate the state value and action advantage in combination with the action optimization module and the value evaluation module, and perform importance weighting in combination with the attention mechanism to obtain the production line action value function; The production line action value function is added to the task decomposition module based on hierarchical reinforcement learning, and the production line control task is decomposed into multiple subtasks through a hierarchical option architecture. For each subtask, the corresponding policy network is trained through a hierarchical policy value network algorithm, and a subtask action sequence is generated and added to the trajectory optimization algorithm based on soft constraints. The station control instructions corresponding to each station are generated in combination with a predictive control method based on time series difference. The station control instructions are adjusted in real time through an adaptive impedance control algorithm. The dynamic characteristic parameters of the production line are identified online in combination with the recursive least squares method, and the identification results are added to a performance prediction model based on Gaussian process regression to obtain the predicted value of the production line performance index; The predicted value of the production line performance index is added to the evaluation module based on multi-objective Pareto, the non-dominated sorting algorithm is combined to solve the non-dominated solution set and the target weight value of each optimization target is determined by the fuzzy hierarchical analysis algorithm, the comprehensive evaluation index of the production line is generated and added to the sample screening module based on the priority dual experience pool, the importance sampling algorithm is combined to assign a priority to each experience sample, the high-value training samples are determined by the confidence upper bound search method and added to the parameter update module based on proximal strategy optimization, the policy network and the value network in the parameter update module are jointly optimized by the trust region optimization algorithm and the KL divergence constraint, the exploration strategy is optimized by the entropy regularization method combined with the adaptive temperature parameter, the updated network parameters are obtained and the production line optimization decision plan is generated.

2. The method according to claim 1, characterized in that The real-time status data of the production line workstations is collected and added to the multi-scale residual neural network. The spatial pyramid pooling module is used to extract features to obtain multi-scale features. The multi-scale features are enhanced by the channel attention module and the spatial attention module. The enhanced features are modeled in time series by combining the cross-modal encoder to obtain a high-dimensional feature vector. The high-dimensional feature vector is detected by an anomaly detection algorithm based on soft clustering. The abnormal data is removed by combining the Mahalanobis distance and the retained data is subjected to denoising by wavelet transform to obtain a standardized feature vector and add it to the pre-set dual deep Q network. The state value and action advantage are calculated by combining the action optimization module and the value evaluation module. The importance is weighted by combining the attention mechanism to obtain the production line action value function, which includes: A data acquisition network based on a layered architecture is used to collect real-time status data on the production line workstations, wherein the data acquisition network includes a bottom-level acquisition node, a middle-level data concentrator, and a top-level central controller. The bottom-level acquisition node collects force data, motion parameter data, visual data, and quality inspection data according to a preset priority, and the middle-level data concentrator is used to perform signal conditioning on the collected data. The top-level central controller completes normalization processing through a distributed preprocessing structure. The normalized data of the top-level central controller is input into a multi-scale residual neural network, and features are extracted through multiple residual learning blocks in the backbone network. The large-scale convolution kernel, the medium-scale convolution kernel and the small-scale convolution kernel configured in the parallel branch network are combined to extract global semantic features, regional structural features and local texture features respectively. The extracted features are globally pooled, four-region pooled and nine-region pooled through the spatial pyramid pooling module, and the pooling results are fused to form multi-scale features through channel rearrangement. The preset channel attention module generates channel statistical features through global average pooling and maximum pooling, and generates channel attention weights through a multi-layer perceptron with shared weights. The spatial attention module combines position attention and channel attention to generate a spatial attention weight map, and enhances the multi-scale features based on the spatial attention weight map; The enhanced features are input into the cross-modal encoder, and the multimodal features are mapped to a unified feature space using the feature encoding layer. The long short-term memory network in the bidirectional processing layer is combined to capture the sequence dependency. The attention fusion layer integrates the temporal information through the multi-head self-attention mechanism and outputs a high-dimensional feature vector. In the central controller, an anomaly detection algorithm based on soft clustering is used to perform anomaly detection on the high-dimensional feature vector, the sample probability density is estimated by a Gaussian kernel function, the category center is optimized by an expectation maximization algorithm, the abnormal data is removed in combination with the Mahalanobis distance, and the retained data is subjected to denoising processing by wavelet transform to obtain a standardized feature vector; Input the standardized feature vector into a preset dual deep Q network, map the state feature into the state value using the regression network structure in the value evaluation module, and output the action advantage value using the classification network structure in the action optimization module; The central controller calculates the importance of the action based on the attention mechanism and converts it into a weight coefficient, weights the state value and the action advantage value by importance to obtain the production line action value function, and sends the action value function to the data concentrator.

3. The method according to claim 2, characterized in that In the central controller, the high-dimensional feature vector is subjected to anomaly detection using an anomaly detection algorithm based on soft clustering, the sample probability density is estimated by a Gaussian kernel function, the category center is optimized by an expectation maximization algorithm, the abnormal data is removed by combining the Mahalanobis distance, and the retained data is subjected to denoising by wavelet transform to obtain a standardized feature vector including: In the central controller, the high-dimensional feature vector is preprocessed in segments by a sliding window, the sliding window moves continuously on the high-dimensional feature vector, and time domain features, frequency domain features and statistical features are calculated based on the sliding window, the time domain features include peak value, mean value, variance, kurtosis and skewness, the frequency domain features include main frequency component, frequency band energy ratio and spectrum entropy, and the statistical features include quartiles, zero crossing rate and waveform factor of data distribution; In the central controller, a data sample is projected into a high-dimensional feature space by a preset Gaussian kernel function, a similarity relationship between samples is calculated, a probability density distribution diagram is constructed, a plurality of category centers are initialized by the expectation maximization algorithm, the position of the category center and the category belonging probability of each data point are iteratively optimized, the probability distribution of each data point belonging to each category is calculated, and the position of the category center is updated based on the probability distribution until a preset maximum number of updates is reached, wherein the bandwidth parameter of the Gaussian kernel function is adaptively determined by a cross-validation method; In the central controller, the Mahalanobis distance between the category center and each data point is calculated based on the position of the category center, an adaptive threshold is set according to the historical data distribution, when the Mahalanobis distance of a data point to the nearest category center exceeds the adaptive threshold, it is marked as the abnormal data, the adjustment of the adaptive threshold is triggered based on the timing constraint, and the retained data is decomposed into wavelet coefficients of different frequency bands through the wavelet transform, the wavelet coefficients are denoised by a soft threshold method, and the denoised data is normalized by centering and scaling transformation to obtain the standardized feature vector; In the central controller, data processing tasks are distributed to multiple computing nodes through a distributed parallel computing framework, the processing results of the computing nodes are summarized through a shared memory mechanism, and the intermediate calculation results with access frequency greater than a preset access threshold are stored based on a multi-level cache strategy. The abnormal data is continuously collected and an abnormal pattern library is constructed, the abnormal detection results and their verification information are recorded, and the bandwidth parameter of the Gaussian kernel function, the number of category centers and the adaptive threshold of the Mahalanobis distance are updated based on the verification information.

4. The method according to claim 1, characterized in that: The production line action value function is added to the task decomposition module based on hierarchical reinforcement learning, and the production line control task is decomposed into multiple subtasks through a hierarchical option architecture. For each subtask, the corresponding policy network is trained through a hierarchical policy value network algorithm, and a subtask action sequence is generated and added to the trajectory optimization algorithm based on soft constraints. The station control instructions corresponding to each station are generated in combination with a predictive control method based on time series difference. The station control instructions are adjusted in real time through an adaptive impedance control algorithm. The dynamic characteristic parameters of the production line are identified online by combining the recursive least squares method, and the identification results are added to a performance prediction model based on Gaussian process regression. The predicted values ​​of the production line performance indicators include: Obtaining the action value function of the production line, optimizing the weight parameters in the action value function based on historical operation data, adjusting the weight coefficients of the production efficiency, product quality, energy consumption and equipment load indicators through online iteration, and inputting the optimized action value function into the task decomposition module based on hierarchical reinforcement learning; The hierarchical option architecture of the task decomposition module is used to decompose the production line control task into the top-level task of overall production line scheduling control, the middle-level task of workstation group collaborative control, and the bottom-level task of individual workstation fine control. An option linking mechanism is established to transfer the decision result of the top-level task to the middle-level task, and transfer the decision result of the middle-level task to the bottom-level task. For each underlying task, a hierarchical strategy value network is trained. Sensor data and process parameters are converted into subtask feature vectors through the state encoding layer. The action probability distribution is output based on the subtask feature vectors through the strategy generation layer. The long-term benefits of the action sequence are predicted in combination with the value evaluation layer. The training samples are stored in the experience playback mechanism. The parameters of the hierarchical strategy value network are optimized based on the temporal difference learning method. The action sequence output by the optimized hierarchical strategy value network is input into the trajectory optimization algorithm based on soft constraints, and the obstacle avoidance constraint is established in combination with the potential field function. The action sequence is iteratively optimized by the gradient descent method to generate an optimized trajectory that satisfies the kinematic constraints, dynamic constraints and process constraints, and the optimized trajectory is decomposed into the benchmark control instructions of each station; A two-layer architecture of a state prediction model and a controller is constructed based on the benchmark control instructions, the state prediction model is used to learn the dynamic characteristics of the workstation online and predict the state evolution trajectory, the workstation control instructions are generated based on the state evolution trajectory, a dual closed-loop structure of position control and force control is established, the parameters of the controller are adjusted in real time according to the feedback of the force sensor, a dynamic model of inertia parameters, damping parameters and stiffness parameters is constructed, the parameters of the dynamic model are updated based on the sensor data by the recursive least squares method, the updated parameters are input into the performance prediction model, a mapping relationship between system parameters and performance indicators is established, the performance prediction model is continuously optimized, and the predicted values ​​of the production line performance indicators are output; Based on the predicted values ​​of the production line performance indicators, state prediction, trajectory optimization and parameter identification are performed on a parallel computing architecture. The bottom-level data cache module is used to collect and preprocess sensor data. The middle-level data processing module is used to extract features and estimate states. Performance evaluation and optimization decisions are performed through the top-level data analysis module to achieve hierarchical data management.

5. The method according to claim 4, characterized in that The action sequence output by the optimized hierarchical strategy value network is input into the trajectory optimization algorithm based on soft constraints, and the obstacle avoidance constraint is established in combination with the potential field function. The action sequence is iteratively optimized by the gradient descent method to generate an optimized trajectory that satisfies the kinematic constraints, dynamic constraints and process constraints, and the optimized trajectory is decomposed into the benchmark control instructions of each station, including: Receive the action sequence output by the hierarchical strategy value network, extract the position vector, velocity vector, acceleration vector and attitude quaternion of each control action point, use the sliding window method to identify abnormal state points, correct the abnormal state points based on the local weighted regression algorithm, use a low-pass filter to smooth the corrected state sequence, and generate an initial trajectory sequence; The initial trajectory sequence is divided into a plurality of control intervals, a control point is set in each control interval, a trajectory parameterization model is constructed using a cubic spline basis function, joint position limits, speed limits, acceleration limits, joint torque limits and process parameter limits of the actuator are extracted to construct a soft constraint evaluation function, and the initial trajectory sequence is substituted into the soft constraint evaluation function to calculate the constraint violation degree of each control interval; A grid potential field model is established in the workspace, a static repulsive potential field is calculated for fixed obstacles, a time-varying repulsive potential field is constructed for dynamic obstacles based on velocity prediction, a Gaussian function is used to describe the attenuation characteristics of the potential field intensity with distance, a sigmoid function is used to construct a potential field transition interval at the obstacle boundary, the initial trajectory sequence is discretely sampled, and the potential field potential energy of the trajectory points is calculated by interpolation; A weighted objective function is constructed based on the degree of constraint violation of each control interval and the potential energy of the potential field of the trajectory point, and the numerical gradient of the objective function to the state variable of the control point is calculated using the central difference method. The optimization step size is adaptively adjusted in combination with the line search criterion, and the control point state is iteratively updated in the opposite direction of the gradient. The objective function value is calculated in real time until its relative change rate is less than the convergence threshold; The continuous trajectory is reconstructed by performing cubic Hermite spline interpolation on the optimized control points. The reconstructed trajectory is substituted into the soft constraint evaluation function to calculate the constraint violation. The critical point is located by using the dichotomy method for the trajectory segment that violates the constraint. Random sampling and local optimization are performed in the neighborhood of the critical point, and it is iterated until all constraint requirements are met. The optimized trajectory that meets the constraints is uniformly sampled according to the control cycle to generate a discrete instruction sequence of position, velocity, acceleration and posture. The Cartesian space instructions are converted into joint space instructions based on the kinematic model of the actuator. A coordinate transformation matrix chain is established according to the workstation layout, and the global trajectory instructions are decomposed into benchmark control instructions for the actuators of each workstation.

6. The method according to claim 1, characterized in that The predicted value of the production line performance index is added to the evaluation module based on multi-objective Pareto, the non-dominated sorting algorithm is combined to solve the non-dominated solution set and the target weight value of each optimization target is determined by the fuzzy hierarchical analysis algorithm, the comprehensive evaluation index of the production line is generated and added to the sample screening module based on the priority dual experience pool, the importance sampling algorithm is combined to assign a priority to each experience sample, the confidence upper bound search method is combined to determine the high-value training samples and add them to the parameter update module based on the proximal strategy optimization, the policy network and the value network in the parameter update module are jointly optimized by the trust region optimization algorithm and the KL divergence constraint, the exploration strategy is optimized by the entropy regularization method combined with the adaptive temperature parameter, the updated network parameters are obtained and the production line optimization decision plan is generated, including: The predicted value of the production line performance index is added to the evaluation module based on multi-objective Pareto, the domination relationship matrix is ​​calculated for the solution set in the evaluation module, the solution with a non-dominated count of zero is identified by combining the non-dominated sorting algorithm to construct a non-dominated solution set, the non-dominated solution set is removed from the solution set as the first layer, and the domination relationship of the remaining solutions is repeatedly calculated until all solutions are stratified to obtain a hierarchical performance index set; Based on the hierarchical performance indicator set, a hierarchical structure model is established for the optimization objectives in the non-dominated solution set, a fuzzy judgment matrix is ​​constructed through a fuzzy hierarchical analysis algorithm, a consistency check is performed on the fuzzy judgment matrix and judgments that do not meet the consistency are corrected, and a target weight value of each optimization objective is determined in combination with a fuzzy comprehensive evaluation; Combining the non-dominated solution set and the target weight value to generate a comprehensive evaluation index for the production line, and adding the comprehensive evaluation index for the production line to a sample screening module based on a priority dual experience pool, wherein the sample screening module based on a priority dual experience pool includes a long-term memory pool for storing historical high-quality samples and a short-term memory pool for dynamically updating interactive samples; For the samples in the sample screening module based on the priority dual experience pool, a priority is assigned to each experience sample according to an importance sampling algorithm, wherein the priority is jointly determined by the time recency, return rarity and comprehensive evaluation index of the production line of the sample, and a sampling probability distribution is constructed based on the priority; Based on the sampling probability distribution, samples are extracted from the priority-based dual experience pool according to the sampling probability distribution through a confidence upper bound search method, the expected return value and prediction uncertainty of the samples are calculated based on the value network, the expected return value and prediction uncertainty are weighted to construct a confidence upper bound evaluation index, and high-value training samples are determined according to the confidence upper bound evaluation index; The high-value training samples are added to the parameter update module based on proximal policy optimization, the KL divergence between the new and old distributions of the policy network is calculated by the trust region optimization algorithm and the deviation constraint is set, the policy network and the value network in the parameter update module are jointly optimized under the KL divergence constraint, and the knowledge transfer is completed through the shared feature extraction layer; The exploration strategy is optimized by combining the entropy regularization method with adaptive temperature parameters, an entropy regularization term is constructed based on the entropy value of the strategy distribution, the entropy regularization term is added to the parameter optimization objective, the temperature parameters are adaptively adjusted according to the degree of improvement of the optimization objective, a production line optimization decision plan is generated based on the updated network parameters, the production line optimization decision plan is deployed to the production line control system, new interaction data is collected to update the priority-based dual experience pool, and the optimization is repeated until convergence.

7. The method according to claim 6, characterized in that The high-value training samples are added to the parameter update module based on proximal policy optimization, the KL divergence between the new and old distributions of the policy network is calculated by the trust region optimization algorithm and the deviation constraint is set, the policy network and the value network in the parameter update module are jointly optimized under the KL divergence constraint, and the knowledge transfer is completed through the shared feature extraction layer, including: Obtain high-value training samples and perform one-hot encoding conversion on discrete data in the high-value training samples, perform maximum and minimum value scaling on continuous data, obtain a standardized state vector and add it to a pre-set parameter update module based on proximal strategy optimization; Based on the standardized state vector, a shared feature extraction layer including a three-layer fully connected network is constructed, wherein the first layer of the shared feature extraction layer receives the standardized state vector, converts the standardized state vector into a first intermediate feature through a ReLU activation function, the first intermediate feature is passed through a second layer of the fully connected network to obtain a second intermediate feature, and the second intermediate feature is passed through a third layer of the fully connected network to obtain a low-dimensional feature vector; Based on the low-dimensional feature vector, the output layer of the policy network and the output layer of the value network are constructed. The output layer of the policy network maps the low-dimensional feature vector to a new probability distribution of the control action through a Softmax function. The output layer of the value network maps the low-dimensional feature vector to a state value through a linear transformation. The action probability distribution corresponding to the current policy network parameters is recorded as the old probability distribution. A training batch is extracted from the standardized state vector and input into the shared feature extraction layer. A new action probability distribution is output through the policy network. The KL divergence between the new action probability distribution and the old probability distribution is calculated based on a trust region optimization algorithm. The advantage value is calculated based on the difference between the state value output by the value network and the actual return, the parameters of the policy network are updated based on the advantage value, the parameters of the value network are updated in combination with the time series difference error, and the parameters of the policy network and the value network are updated alternately under the condition that the KL divergence satisfies the preset threshold constraint; The second intermediate feature of the policy network is input into the prediction layer to obtain the predicted state value, and the prediction error between the predicted state value and the actual state value is back-propagated to the shared feature extraction layer; the second intermediate feature of the value network is input into the reconstruction layer to obtain the reconstructed state vector, and the reconstruction error between the reconstructed state vector and the standardized state vector is back-propagated to the shared feature extraction layer, and the knowledge transfer between the policy network and the value network is completed through the back-propagation of the prediction error and the reconstruction error.

8. An ignition tool robot production line control system based on deep reinforcement learning, used to implement the method described in any one of claims 1 to 7, characterized in that: include: The first unit is used to collect real-time status data on the production line workstation and add it to the multi-scale residual neural network, extract features through the spatial pyramid pooling module to obtain multi-scale features, enhance the multi-scale features through the channel attention module and the spatial attention module, and perform time series modeling on the enhanced features in combination with the cross-modal encoder to obtain a high-dimensional feature vector, perform anomaly detection on the high-dimensional feature vector through an anomaly detection algorithm based on soft clustering, remove abnormal data in combination with the Mahalanobis distance and perform denoising on the retained data through wavelet transform, obtain a standardized feature vector and add it to a pre-set dual deep Q network, calculate the state value and action advantage in combination with the action optimization module and the value evaluation module, and perform importance weighting in combination with the attention mechanism to obtain the production line action value function; The second unit is used to add the production line action value function to the task decomposition module based on hierarchical reinforcement learning, decompose the production line control task into multiple subtasks through a hierarchical option architecture, and for each subtask, train the corresponding policy network through a hierarchical policy value network algorithm, generate a subtask action sequence and add it to the trajectory optimization algorithm based on soft constraints, generate the workstation control instructions corresponding to each workstation in combination with a predictive control method based on time series difference, adjust the workstation control instructions in real time through an adaptive impedance control algorithm, and identify the dynamic characteristic parameters of the production line online in combination with the recursive least squares method, and add the identification results to the performance prediction model based on Gaussian process regression to obtain the predicted value of the production line performance index; The third unit is used to add the predicted value of the production line performance index to the evaluation module based on multi-objective Pareto, solve the non-dominated solution set in combination with the non-dominated sorting algorithm and determine the target weight value of each optimization target through the fuzzy hierarchical analysis algorithm, generate a comprehensive evaluation index of the production line and add it to the sample screening module based on the priority dual experience pool, assign a priority to each experience sample in combination with the importance sampling algorithm, determine the high-value training samples in combination with the confidence upper bound search method and add them to the parameter update module based on proximal strategy optimization, jointly optimize the policy network and the value network in the parameter update module through the trust region optimization algorithm and the KL divergence constraint, optimize the exploration strategy through the entropy regularization method combined with the adaptive temperature parameter, obtain the updated network parameters and generate the production line optimization decision plan.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images