Method for process parameter regulation of thin-walled parts based on multi-agent reinforcement learning

Abstract

The application discloses a kind of process parameter regulation and control methods of thin-walled part based on multi-agent reinforcement learning, comprising: obtaining at least one target machining feature of thin-walled part, and determining the target classification cluster to which target machining feature belongs according to the classification cluster divided in advance;According to the time domain characteristic data of the cutting force signal in the X-axis, Y-axis and Z-axis direction in the process parameter of target machining feature at the current time and machining process, the current observation of target machining feature is constructed;The current observation is input into the process parameter regulation and control model corresponding to target classification cluster, and the process parameter of target machining feature next time is obtained.The application improves the process parameter regulation and control efficiency of target machining feature and the machining efficiency of thin-walled part by constructing classification cluster in advance, trains the process parameter regulation and control model corresponding to each classification cluster using multi-agent reinforcement learning algorithm, and realizes the machining feature level fine-grained regulation of multi-agent.

Description

Technical Field

[0001] This invention belongs to the field of milling technology, specifically relating to a method for controlling process parameters of thin-walled parts based on multi-agent reinforcement learning. Background Technology

[0002] As a primary manufacturing system in the manufacturing industry, CNC machining systems are widely used in the processing of key components for high-end equipment in aerospace, electronics, and information technology due to their precision machining, high efficiency in material removal, and suitability for machining complex thin-walled parts with multiple machining features. In the production process of CNC milling manufacturing systems, the setting of CNC milling process parameters directly determines the controllability of material removal quality and efficiency. However, the complexity of part shapes and the characteristics of multiple machining features make it difficult for manufacturing systems to simultaneously guarantee optimal performance when facing the synergistic optimization goals of multiple machining quality and efficiency indicators. Furthermore, current process parameter control still relies on traditional simulation modeling, manual experience, or trial cutting, resulting in high uncertainty, poor accuracy, and low efficiency in optimizing CNC process parameters for complex parts with multiple machining features.

[0003] With the development of data-driven technologies, some machine learning or deep learning methods have been used to recommend process parameters for CNC machining, improving the accuracy of parameter recommendations and achieving certain practical results. However, current process parameter recommendation models often recommend universal process parameters for all machining features of the workpiece, or lack joint consideration of objectives when constructing optimization goals. This does not meet the actual production needs of manufacturing systems, resulting in low synergistic improvement of multiple workpiece quality indicators and machining efficiency, and unstable results of synergistic control of process parameters.

[0004] Furthermore, for workpieces with complex machining features in manufacturing systems, these features exhibit both differences and correlations, necessitating the recommendation of differentiated process parameters. Existing data-driven process parameter control methods lack effective simulation of complex machining environments (such as the influence of cutting forces) and effective correlation between multiple machining features and optimization control objectives. For the milling process parameter control problem of complex parts containing multiple machining features, the multi-source data such as process parameters, cutting forces, and machining indicators contain deep correlation information at the machining feature level, and the multi-dimensional machining indicators exhibit coupling characteristics, making it difficult for existing methods to output effective machining feature-level control strategies. Summary of the Invention

[0005] To address the aforementioned problems in the existing technology, this invention provides a method for controlling process parameters of thin-walled parts based on multi-agent reinforcement learning. The technical problem to be solved by this invention is achieved through the following technical solution:

[0006] This invention provides a method for controlling process parameters of thin-walled parts based on multi-agent reinforcement learning, comprising:

[0007] Obtain at least one target machining feature of a thin-walled part, and determine the target classification cluster to which the target machining feature belongs based on a pre-divided classification cluster;

[0008] Based on the process parameters of the target machining feature at the current moment and the time-domain characteristic data of the cutting force signals in the X, Y and Z axes during the machining process, the current observation of the target machining feature is constructed.

[0009] The current observation is input into the process parameter control model corresponding to the target classification cluster to obtain the process parameters of the target processing feature at the next time step.

[0010] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0011] This invention provides a method for controlling process parameters of thin-walled parts based on multi-agent reinforcement learning. The method pre-classifies multiple processing features of thin-walled parts into clusters. By extracting the depth features of each processing feature and using the maximum mutual information coefficient between the depth features to quantify the similarity between two corresponding processing features, processing features with similar sensitivity to process parameters are classified into the same cluster. When controlling the process parameters of the target processing features of thin-walled parts, it is only necessary to obtain the process parameter control model corresponding to its target cluster and control the process parameters based on the current observation, which effectively improves the processing efficiency of thin-walled parts.

[0012] Furthermore, this invention utilizes a multi-agent reinforcement learning algorithm to train the process parameter control model corresponding to each classification cluster, models each classification cluster as an agent, captures the dynamic correlation between processing parameters and cutting force within the agent through a local attention encoding layer, achieves cross-agent state collaboration through a global attention interaction layer, and constructs a highly robust multi-agent Q-value evaluation system by combining a dual evaluation network group and a soft update mechanism, thereby realizing fine-grained control of processing features at the multi-agent level.

[0013] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0014] Figure 1 This is a flowchart of a method for controlling process parameters of thin-walled parts based on multi-agent reinforcement learning, provided by an embodiment of the present invention.

[0015] Figure 2 This is a schematic diagram of the classification of clusters provided in an embodiment of the present invention;

[0016] Figure 3 This is a distribution density curve of processing index data in each classification cluster provided in the embodiments of the present invention;

[0017] Figure 4 This is a schematic diagram illustrating the construction process of the multi-task estimation model at the processing feature level corresponding to each classification cluster provided in the embodiments of the present invention;

[0018] Figure 5 This is a schematic diagram of the training process of the process parameter control model provided in an embodiment of the present invention;

[0019] Figure 6 This is a schematic diagram of the network structure of the first evaluation network, the second evaluation network, the first target network, and the second target network provided in an embodiment of the present invention;

[0020] Figure 7 It is a curve showing the combined reward value and combined return value of different process parameter control methods;

[0021] Figure 8 This is a schematic diagram illustrating the combined return rate improvement of different process parameter control methods;

[0022] Figure 9 This is a schematic diagram of the process parameter control trajectory of a classification cluster under the process parameter control method for thin-walled parts based on multi-agent reinforcement learning provided by the present invention.

[0023] Figure 10 This is a schematic diagram of the process parameter control trajectory for the classification clusters under the MATD3 method;

[0024] Figure 11 This is a comparison chart showing the optimization of processing indicators of the present invention compared to the MATD3 method. Detailed Implementation

[0025] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0026] Figure 1 This is a flowchart of a method for controlling process parameters of thin-walled parts based on multi-agent reinforcement learning, provided by an embodiment of the present invention. Figure 1 As shown, this embodiment of the invention provides a method for controlling process parameters of thin-walled parts based on multi-agent reinforcement learning, including:

[0027] S1. Obtain at least one target machining feature of the thin-walled part, and determine the target classification cluster to which the target machining feature belongs based on the pre-divided classification clusters.

[0028] It should be noted that machining features refer to the geometric structures on a part that have specific shapes, functions, and technological attributes during machining, such as chamfers and round holes.

[0029] This embodiment utilizes a stacked autoencoder and the maximum mutual information coefficient to construct classification clusters. Using multivariate processing index data corresponding to each processing feature as feature representation, the stacked autoencoder extracts depth features from the processing index data of each processing feature, resulting in a set of depth features for the processing features of complex thin-walled parts. Then, the maximum mutual information coefficient is used to calculate the data distribution similarity between each depth feature pairwise, and the processing features corresponding to depth features with high similarity are grouped into a classification cluster. This completes the construction of multi-agent deep reinforcement learning classification clusters, laying the foundation for parallel estimation of multivariate processing indices and precise control of process parameters.

[0030] For example, taxonomic clusters can be divided according to the following steps:

[0031] S101. Obtain the machining index data for each machining feature of the thin-walled part. The machining index data includes machining efficiency index data and machining quality index data. The machining efficiency index data of the machining feature is the milling completion time of the machining feature, and the machining quality index data of the machining feature is the error between the actual size and the design size of the machining feature.

[0032] The batch machining experiment of complex thin-walled parts was completed on a Mikron HSM600ULP high-speed machining center. Taking the machining of 40 workpieces as an example, blanks of the same specifications were milled with the same dimensional settings, and the tool specifications used in the machining process were kept consistent. Five identical aluminum alloy flat-end mills, each with a diameter of 1 mm, were used for batch machining. During the machining of each workpiece, the current process parameters, namely spindle speed, feed rate, and depth of cut, were recorded in real time. The combination of process parameters for different workpieces was set through orthogonal experiments. Simultaneously, a 9527 Ad Kistler force gauge was used to acquire the cutting force signals of the X, Y, and Z axes at a frequency of 400 Hz during the machining process.

[0033] After each workpiece is machined, the machine tool system records the milling completion time of each machining feature as machining efficiency index data. Then, the coordinate measuring machine records the actual measurement value and the dimensional error value calculated from the design dimensions as machining quality index data. The machining index data contained in each machining feature constitutes the information representation of that shape feature after machining, serving as an important support for subsequent depth feature extraction and feature fusion construction. In addition, the sum of the machining times of all recorded machining features is used as the workpiece-level machining efficiency index data. The workpiece-level machining quality index of thin-walled parts includes the machining quality index data of each machining feature. For example, the average machining quality of four circular holes is used as the machining quality index data of the circular holes, and the total machining time of the four circular holes is used as the machining efficiency index data of the circular holes.

[0034] S102. Input the processing index data of each processing feature into the stacked autoencoder to obtain the depth features of each processing feature, forming a depth feature set:

[0035] ;

[0036] In the formula, Indicates the first thin-walled part The depth features of each processing feature This indicates the number of encoder-decoder combinations in a stacked autoencoder. They represent the first Data on processing quality indicators and processing efficiency indicators for each processing feature.

[0037] S103. Calculate the maximum mutual information coefficient between any two deep features in the deep feature set:

[0038] ;

[0039] In the formula, express and The maximum mutual information coefficient, , express and mutual information value, This indicates that the partitioning constraint is satisfied. A set of grids, Complexity control function related to sample size (default) ).

[0040] S104, when the maximum mutual information coefficient When it exceeds the preset threshold, the first The processing feature and the first Each processing feature is grouped into the same classification cluster.

[0041] Optionally, the preset threshold is 0.9, when the maximum mutual information coefficient When it is greater than 0.9, it indicates that the first The processing feature and the first The processing features have a high degree of similarity in data distribution and can be classified into a single cluster.

[0042] S105. By traversing all depth features in the depth feature set, all machining features of thin-walled parts are divided into different classification clusters.

[0043] Figure 2This is a schematic diagram of the classification clusters provided in an embodiment of the present invention. Taking six processing features—crack, circular hole, rectangle 1, rectangle 2, rectangle 3, and rectangle 4—as examples, the maximum mutual information coefficient of the depth features of any two processing features is as follows: Figure 2 As shown, the maximum mutual information coefficient between the depth features of rectangle 2 and rectangle 3 is 0.954; the maximum mutual information coefficient between the depth features of the circular hole and rectangle 4 is 0.953; the maximum mutual information coefficient between the depth features of the crack and rectangle 4 is 0.916; and the maximum mutual information coefficient between the depth features of the crack and the circular hole is 0.953, all of which are greater than the preset threshold. Therefore, rectangle 2 and rectangle 3 are classified into one cluster, and the three processing features of rectangle 4, circular hole, and crack are classified into another cluster.

[0044] Figure 3 This is a distribution density curve of processing index data in each classification cluster provided in the embodiments of the present invention. For example... Figure 3 As shown, the horizontal axis represents the data distribution range, and the vertical axis represents the data distribution density. It can be seen that after dividing into classification clusters, the data distribution density of processing features contained in each classification cluster is similar, indicating that the clustering method used in this embodiment is effective.

[0045] S2. Based on the process parameters of the target machining feature at the current moment and the time-domain characteristic data of the cutting force signals in the X-axis, Y-axis and Z-axis directions during the machining process, construct the current observation of the target machining feature.

[0046] S3. Input the current observation into the process parameter control model corresponding to the target classification cluster to obtain the process parameters of the target processing features at the next moment.

[0047] Specifically, the process parameter control models corresponding to different classification clusters are trained according to the following steps:

[0048] S301. Acquire cutting force signals in the X, Y and Z axes of each machining feature of the thin-walled part at a preset frequency during the machining process and extract multiple time-domain features, including: mean, maximum, standard deviation, root mean square value, peak-to-peak value, skewness index, kurtosis index, peak index and impulse index.

[0049] S302. Using the principal component analysis algorithm, the 9-dimensional time-domain feature data corresponding to the X-axis, Y-axis and Z-axis directions are dimensionality reduced. Then, the Hermite interpolation algorithm is used to enhance the 9-dimensional time-domain feature data, process parameters and machining features to obtain the cutting force feature set.

[0050] After dimensionality reduction and data augmentation, the number of samples in the cutting force feature set is expanded, which solves the problem of data scarcity in small-batch manufacturing. This helps to reduce computing costs, improve model training efficiency, and make the model transferable and applicable.

[0051] S303. Pre-train the workpiece-level multi-task estimation model using the process parameters and cutting force feature set of the machining features.

[0052] Figure 4 This is a schematic diagram illustrating the construction process of the multi-task estimation model at the processing feature level corresponding to each classification cluster provided in this embodiment of the invention. For example... Figure 4 As shown, in step S303, the process parameters and feature data of the machining features are input into the constructed workpiece-level multi-task estimation model to obtain the predicted data of machining efficiency index and machining quality index of thin-walled parts. The workpiece-level multi-task estimation model includes a first expert layer, a first gating network and a first estimation tower layer. The process parameters include spindle speed, feed rate and depth of cut. The feature data are: time-domain feature data corresponding to the X-axis, Y-axis and Z-axis directions extracted from the cutting force feature set using a sliding window. Optionally, the process parameters are also extracted by the sliding window.

[0053] In the above-mentioned workpiece-level multi-task estimation model, the upstream first expert layer is composed of a residual network based on the short-circuit connection assumption combined with a convolutional neural network, and the downstream first estimation tower layer is constructed by BiLSTM.

[0054] Furthermore, based on the predicted data of processing efficiency and processing quality indicators for thin-walled parts, as well as the data of processing efficiency and processing quality indicators for thin-walled parts, the root mean square error (RMSE) is calculated to measure the performance of the workpiece-level multi-task estimation model.

[0055] At the current iteration number When the first preset number of iterations is reached, the best-performing generation of workpiece-level multi-task estimation model is used as the pre-trained workpiece-level multi-task estimation model.

[0056] S304. Construct a multi-task estimation model at the processing feature level for each category cluster. After each processing feature level multi-task estimation model inherits some parameters from the workpiece level multi-task estimation model, fine-tune each processing feature level multi-task estimation model to obtain the processing index estimation model corresponding to each category cluster.

[0057] Specifically, in step S304, a multi-task estimation model at the processing feature level is constructed for each classification cluster. The multi-task estimation model at the processing feature level includes: a second expert layer, a second gating network, and a second estimation tower layer. The structures of the second expert layer, the second gating network, and the second estimation tower layer are the same as those of the first expert layer, the first gating network, and the first estimation tower layer, respectively, so they will not be described again here.

[0058] The second expert layer in each processing feature-level multi-task estimation model inherits the parameters of the first expert layer in the pre-trained workpiece-level multi-task estimation model. The process parameters and feature data of the processing features are input into the processing feature-level multi-task estimation model corresponding to each classification cluster to obtain the predicted processing efficiency index and the predicted processing quality index for each classification cluster. Further, based on the predicted processing efficiency index and the predicted processing quality index for each classification cluster, and considering both the data, the loss value of the first preset loss function is calculated. Based on the loss value, the parameters of the second gating network and the second estimation tower layer are adjusted through backpropagation to obtain the processing index estimation model corresponding to each classification cluster.

[0059] For example, during pre-training, a deep residual network formed by connecting multiple residual blocks of the first expert layer is represented as:

[0060] ;

[0061] in, Indicates the current batch. and Indicates that for the first In the multi-task estimation model at the processing feature level constructed from the classification clusters, the Inputs and outputs of each residual block. This represents the output data after residual convolution processing.

[0062] During the fine-tuning phase, the training target is the second gating network. With the second estimated tower layer The corresponding updated parameters are the parameters of the second gating network. Parameters of the second estimated tower layer ,in, Representing taxonomic clusters The next One processing indicator, express The corresponding estimation task; the output data of the second gating network and the second expert layer. Weighted summation constitutes the underlying shared features. This serves as the input data for the second estimation layer; the downstream second estimation layer receives the shared features from the bottom layer and performs parallel estimation of the processing indicators, as shown below:

[0063] ;

[0064] in, Indicates the first The output of the multi-task estimation model for the processing feature level corresponding to the i-th cluster is the first... The predicted data for each processing indicator, namely the predicted data for processing efficiency and the predicted data for processing quality of each category cluster, do not need to be adjusted or updated since the parameters of the second expert layer are inherited from the parameters of the pre-trained first expert layer.

[0065] During fine-tuning, dynamic weight balancing (DWA) is used to assign adaptive weights to each learning task. To adjust the balance between the various processing indicators and estimation tasks, the weighted sum of the losses from each task is the result of the first task. The total loss of the multi-task estimation model for processing features of each cluster. :

[0066] ;

[0067] in, Indicates the first The first category The actual observed value of the processing indicator, i.e. the first... The first category The first processing efficiency index data and the first The processing quality index data is used. The Adam optimizer and the mini-batch gradient MMoE-BTE model are then used to update and optimize the [number]th [processing quality index data]. The parameters of the multi-task estimation model corresponding to each classification cluster are as follows:

[0068] ;

[0069] in, Indicates the first The learning rate of the multi-task estimation model for processing features of each classification cluster.

[0070] Thus, the machining feature-level multi-task estimation model generated based on fine-tuned temporal learning is completed. This embodiment addresses the issue of the large amount of data required and relatively slow training speed of the workpiece-level multi-task estimation model. When estimating machining indices for each classification cluster, the predicted machining index data for each cluster is used as output, and the enhanced cutting force feature set data is used to construct the machining feature-level multi-task estimation model, thereby improving the model's training efficiency. Furthermore, to ensure estimation accuracy, the structure and parameters of the first expert layer obtained during the pre-training phase of the workpiece-level multi-task estimation model are retained, enabling the fine-tuned machining feature-level multi-task estimation model to possess its deep feature extraction performance, further improving the training efficiency of the machining feature-level multi-task estimation model and reducing computational costs.

[0071] Figure 5 This is a schematic diagram of the training process of the process parameter control model provided in this embodiment of the invention. The following is in conjunction with... Figure 5 The training process of the process parameter control model shown in steps S305 to S315 is explained.

[0072] S305. Model each classification cluster as an agent and construct the multi-agent environment state; the multi-agent environment state includes the state of the agent corresponding to each classification cluster in... Observation of time: ;in, , Indicates the first Processing features in each cluster Process parameters at any given time, This indicates the multi-task time series prediction model of cutting force. The predicted first Each cluster in The time-domain characteristics of the cutting force signal at time t. , Indicates the number of taxonomic clusters.

[0073] S306. Use the processing index estimation model corresponding to each taxonomic cluster as a virtual environment and input the observation... , obtained the Each cluster in The processing quality indicators and processing efficiency indicators at any given time.

[0074] S307, according to the... Each cluster in The processing quality and efficiency indicators at each moment are used to calculate the actions performed by all agents. Combined reward value .

[0075] In reinforcement learning, agents tend to choose actions with greater rewards and long-term benefits, and a smaller objective function value leads to better optimization of processed features. The objective function is expressed as:

[0076] ;

[0077] Therefore, when setting the reward for a single agent, a positive constant is subtracted from the optimization objective function value for each classification cluster, and then the reward values ​​of each agent are summed to obtain the joint reward. For example, the joint reward value is calculated according to the following formula. :

[0078] ;

[0079] ;

[0080] ;

[0081] In the formula, Indicates the first The number of processing features contained in each feature cluster. Indicates the first The objective function for optimizing process parameters of a feature-based classification cluster. Indicates the first The first feature classification cluster One processing quality indicator, Indicates the first The first feature classification cluster Ideal values ​​for each processing quality indicator Indicates the first The spindle speed of each feature classification cluster Indicates the first The feed rate for each feature classification cluster Indicates the first The cutting depth of each feature classification cluster Indicates the first The number of processing quality indicators for each feature classification cluster. Indicates the first The first feature classification cluster One processing efficiency indicator Indicates the first The first feature classification cluster The ideal value of a processing efficiency index Indicates the first The number of processing efficiency indicators for each feature classification cluster.

[0082] This embodiment will contain The problem of process parameter control for complex thin-walled parts with multiple classification clusters is solved using multi-agent Markov tuples. This study demonstrates that in the multi-agent deep reinforcement learning model, the problem of classifying and controlling process parameters is transformed into a multi-agent Markov tuple. The state space, action space, and joint reward function are designed in detail, which solves the problems of dynamism, uncertainty, multi-objective collaborative optimization, policy learnability, and evaluation difficulty in the original process parameter control problem.

[0083] S308, will observe Enter the first Each cluster corresponds to an actor network, and the actor evaluation network within the actor network is based on observations. , Timing-based process parameter control strategies as well as Added noise Generate the first Each cluster in Moment of action and update Observation of time Among them, the first Initial observations of each taxonomic cluster From length of The window is randomly captured from the state of the multi-agent environment.

[0084] Each actor network corresponding to a classification cluster includes an actor evaluation network and an actor target network. Under the Centralized Training and Decentralized Execution (CTDE) paradigm, each actor network corresponds to a classification cluster, and its process parameters are adjusted. Specifically, each actor evaluation network observes... The input is the process parameter control strategy for this classification cluster, and the output is the process parameter control strategy for this classification cluster. The actions generated below Then, actions are performed according to the strategy. In order to obtain observations at the next moment. .

[0085] S309, Observation Enter the first The actor target network in the actor network corresponding to each classification cluster is based on observation. Process parameter control strategies as well as Added noise , generate the first Each cluster in Moment of action .

[0086] in, This represents truncated Gaussian noise added to prevent overfitting, with the cutoff range being [value missing]. .

[0087] Since the actor network uses a deterministic strategy, a small amount of noise is needed to expand the optimization range and form the final sequence of actions to be executed. .

[0088] S310, put all intelligent agents in Joint observation at different times and joint actions Input the overall critic network, and use the parameters of the overall critic network respectively. , The first evaluation network and the second evaluation network calculate the Q-value sequence. , And utilize the parameters in the overall critic network as , The first target network and the second target network calculate the target Q-value sequence. , .

[0089] Figure 6 This is a schematic diagram of the network structure of the first evaluation network, the second evaluation network, the first target network, and the second target network provided in an embodiment of the present invention. Figure 6 As shown, the first evaluation network, second evaluation network, first target network, and second target network in the overall critic network have the same structure, all including:

[0090] The input layer is used to receive input from all agents. Joint observation at different times and joint actions ;

[0091] The local attention encoding layer is used to encode the observation-action pairs of each agent through a multi-head attention mechanism, resulting in local encoding. ;

[0092] A global attention interaction layer is used to adjust local encodings. The global feature vector representing the dependencies between agents is calculated through a multi-head attention mechanism;

[0093] The splicing layer is used to splice together local codes. With global feature vectors;

[0094] Output a linear layer used to calculate the Q-value sequence based on the splicing results of the splicing layers. , or target Q-value sequence , .

[0095] As can be seen, the role of the local attention encoding layer is to manage the observation-action pairs of each agent. Encoding is performed based on the observations of each agent. Except for process parameters In addition, it also includes time-domain characteristic data of cutting forces in three directions. Therefore, the local attention encoding layer weights and combines different cutting force characteristics with process parameters, and then inputs them together with the mapped actions into the bus layer for encoding, thus obtaining the local encoding. For example, a local attention mechanism is used to apply the following formula: Encode:

[0096] ;

[0097] In the formula, , express Local encoding, It is a bus layer. This indicates a multi-head attention mechanism. Indicates the first Sequence of process parameters observed by an agent; Indicates the first The first observation of the agent The cutting force characteristics in the direction, i.e. x, y, z The characteristics of cutting force in a certain direction along the shaft; Indicates the action Create a linear layer that maps the output of the multi-head attention mechanism so that it can be input into the total output linear layer along with the output of the multi-head attention mechanism.

[0098] Global attention interaction layer with local encoding As input, the interaction between agents is measured. To calculate more accurate and reliable attention weights, a multi-head attention mechanism is used to generate independent attention values, i.e., global feature vectors, for the same input, thus converting local encodings. After being concatenated with the global feature vector, it is input into the total output linear layer to obtain the final Q value, as shown in the following formula:

[0099]

[0100] In the formula, Indicates a linear layer with total output. This represents the multi-head attention mechanism in the global attention interaction layer. These represent the local encodings of observation-action pairs for two different agents.

[0101] It should be understood that, in order to fully learn the correlation information between multiple agents and within a single agent, this embodiment designs a multi-agent dual-delay deep deterministic policy gradient deep reinforcement learning algorithm based on a global-local attention mechanism. The multi-agent structure and function are designed according to a centralized training and decentralized execution paradigm. During training, the overall commentator network learns Q-values ​​based on global action and observation information to comprehensively evaluate the process parameter control strategies of each agent. In the execution phase, i.e., the policy exploration phase, each actor network calculates and executes the process parameter control strategy based solely on its own local observation information. The designed method improves both the accumulated total reward value and the obtained total optimization objective value, effectively and accurately improving the processing quality of different classification clusters' processing features while ensuring processing efficiency.

[0102] S311, Based on the combined reward value Target Q-value sequence , and Q-value sequence , Calculate the first time-series difference error Second timing difference error .

[0103] Specifically, firstly based on the target Q-value sequence , and joint reward value ,calculate Multi-agent temporal difference objective at any given time:

[0104] ;

[0105] In the formula, This represents the discount factor.

[0106] By selecting the target Q-value sequences output by the first target network and the second target network , The smaller one is included in the calculation to prevent the Q value from being overestimated.

[0107] Next, calculate separately Multi-agent temporal difference objective With Q-value sequence , The resulting multi-agent temporal difference error:

[0108] ;

[0109] ;

[0110] In the formula, , , , Representing the Q-value sequences respectively , The first in Values.

[0111] S312, Based on the first timing difference error Second timing difference error Calculate the loss value of the second preset loss function, and update the parameters of the first evaluation network and the second evaluation network based on the loss value of the second preset loss function.

[0112] The second preset loss function is:

[0113] ;

[0114] In the formula, This indicates the sampled number obtained according to the priority experience playback mechanism. The sampling weights of each Markov tuple This represents the batch size from which Markov tuples are drawn from the experience replay buffer.

[0115] Please continue reading Figure 5 It should be noted that this embodiment requires a data capacity of Experience replay buffer in mini batch size Extracting Markov tuples The network is trained using this method. However, the impact of each data point in the experience replay buffer on the algorithm is actually different, a problem that becomes more pronounced in multi-agent deep reinforcement learning algorithms with larger datasets and more complex simulation environments. While randomly drawing experience data from the pool with equal weights to train the reinforcement learning network can effectively cover a wider data area, it can easily lead to convergence difficulties when facing complex business scenarios. Therefore, this embodiment uses a priority experience replay mechanism to assign weights to the Markov tuples in the experience replay buffer and prioritizes their extraction based on these weights.

[0116] S313. Update the parameters of the actor evaluation network through policy gradient.

[0117] The goal of each actor network is to iteratively optimize the process parameter control strategy to maximize the objective function. ,in The expected value of the round reward is represented by parameters updated using weighted mini-batch extraction of experience tuples from the priority experience replay buffer. This is combined with the Q-function of the evaluation network in the overall critic network to train and improve the gradient of the process parameter control strategy for the corresponding classification cluster using the following formula:

[0118] ;

[0119] In the formula, Describe the objective function relative parameters gradient, and Represents two Q-value sequences , Relative actions gradient, express The first Markov tuple The gradient generated by a Markov tuple Representation strategy relative parameters The gradient.

[0120] The process parameter control method designed in this embodiment improves the synergy and robustness of the multi-agent process parameter control strategy. For the first target network, the second target network, and the actor target network, it can avoid the instability of the overall commentator network and the divergence of the actor network during the training process.

[0121] S314. Based on the updated parameters of the first evaluation network and the updated parameters of the second evaluation network, softly update the parameters of the first target network and the second target network, and further update the parameters of the actor target network based on the updated parameters of the actor evaluation network.

[0122] In this embodiment, the parameters of the first target network are softly updated based on the updated parameters of the first evaluation network:

[0123] ;

[0124] In the formula, This indicates the updated parameters of the first evaluation network. This represents the soft update coefficient.

[0125] Furthermore, based on the updated parameters of the second evaluation network, the parameters of the second target network are softly updated:

[0126] ;

[0127] In the formula, This represents the updated parameters of the second evaluation network;

[0128] Update the parameters of the actor target network based on the updated parameters of the actor evaluation network:

[0129] ;

[0130] In the formula, This indicates the parameters updated by the actor evaluation network.

[0131] Furthermore, with new observations As the strategic calculation object of each actor network, return to the first step and proceed to the next moment. Multi-agent Markov decision process.

[0132] S315, at the current iteration number When the second preset number of iterations is reached, the actor target network in each actor network is used as the process parameter control model for the processing features in each classification cluster.

[0133] The method for controlling process parameters of thin-walled parts based on multi-agent reinforcement learning provided by this invention is further illustrated through simulation experiments.

[0134] Experimental conditions: This example was run on a platform equipped with an i5-12500H CPU, an NVIDIA RTX3060, and 16GB of RAM, using PyCharm as the compiler;

[0135] Experimental content: Based on the same CNC milling virtual environment for classifying complex thin-walled parts, the effectiveness and advantages of the process parameter control of the present invention are verified. First, the reward value and return value of each algorithm are compared. Then, the process parameter control experiment is carried out on the trained model using the process parameter sequence in the actual processing data. Finally, a comparative experiment on the optimization of processing indicators is conducted.

[0136] Experimental Results and Analysis:

[0137] (1) Visualization of reward value and cumulative return value

[0138] Figure 7 This is a graph showing the combined reward value and combined return value of different process parameter control methods. Figure 8 This is a schematic diagram illustrating the combined return rate improvement of different process parameter control methods. Specifically, as shown... Figure 7 As shown, using the MATD3 method as a benchmark, MAAC, MADDPG, MAPPO, QMIX, VDN, and the multi-agent reinforcement learning-based thin-walled part process parameter control method (GLA-MATD3) provided in this invention are compared with it. Each curve represents the average value of 10 sets of experimental results, and each curve also corresponds to a shaded area, representing the upper and lower bounds of the results obtained from 10 sets of experimental runs of the corresponding method. Through analysis of... Figure 7-8 It can be seen that the process parameter control model provided by this invention, after convergence, obtains higher reward and return values ​​than the comparative algorithm, and the return value improvement rate can reach 4%-14%, indicating that the controlled process parameters have higher reliability.

[0139] (2) Experiment on the trajectory control of process parameters for complex thin-walled parts classification clusters

[0140] Figure 9 This is a schematic diagram of the process parameter control trajectory for a classification cluster under the multi-agent reinforcement learning-based method for controlling process parameters of thin-walled parts provided by the present invention. Figure 10 This is a schematic diagram of the process parameter control trajectory for the classification cluster under the MATD3 method. Table 1 also shows the control results of the present invention and six other existing methods for the same process parameter sequence:

[0141] Table 1

[0142]

[0143] from Figures 9-10 As shown in Table 1, for the three classification clusters, the objective values ​​of the optimization function of this invention reached 0.256, 0.193, and 0.378, respectively, with a total objective function value of 0.827. Moreover, the optimization results improved by 6%-14%, 2%-26%, and 3%-17% compared with the comparative algorithms, respectively, and the total objective function value improved by 4%-14%. The number of iterations was the fewest, demonstrating the effectiveness and efficiency of process parameter control.

[0144] (3) Comparison Experiment of Processing Index Optimization Results

[0145] Figure 11 This is a comparison chart of the optimized processing parameters of this invention compared to the MATD3 method. To verify the stability of this invention in practical applications, the length of rectangle 1 in cluster 1 and processing time, the width of rectangle 3 in cluster 2 and processing time, and the diameter of the circular hole in cluster 3 and processing time were selected. Figure 11 The orange dashed line represents the ideal value of 0 for the processing quality index, which is... Figure 11 It can be seen that the multi-dimensional processing index obtained by the present invention after adjusting the process parameters of each category cluster is closer to the ideal value than the comparison algorithm, and the fluctuation of multiple experimental results is smaller. The present embodiment has more stable and better control performance for the process parameters of the category cluster of complex thin-walled parts.

[0146] As can be seen from the above embodiments, the beneficial effects of the present invention are as follows:

[0147] This invention provides a method for controlling process parameters of thin-walled parts based on multi-agent reinforcement learning. The method pre-classifies multiple processing features of thin-walled parts into clusters. By extracting the depth features of each processing feature and using the maximum mutual information coefficient between the depth features to quantify the similarity between two corresponding processing features, processing features with similar sensitivity to process parameters are classified into the same cluster. When controlling the process parameters of the target processing features of thin-walled parts, it is only necessary to obtain the process parameter control model corresponding to its target cluster and control the process parameters based on the current observation, which effectively improves the processing efficiency of thin-walled parts.

[0148] Furthermore, this invention utilizes a multi-agent reinforcement learning algorithm to train the process parameter control model corresponding to each classification cluster, models each classification cluster as an agent, captures the dynamic correlation between processing parameters and cutting force within the agent through a local attention encoding layer, achieves cross-agent state collaboration through a global attention interaction layer, and constructs a highly robust multi-agent Q-value evaluation system by combining a dual evaluation network group and a soft update mechanism, thereby realizing fine-grained control of processing features at the multi-agent level.

[0149] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0150] The use of terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples" indicates that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0151] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A method for controlling process parameters of thin-walled parts based on multi-agent reinforcement learning, characterized in that, include: Obtain at least one target machining feature of a thin-walled part, and determine the target classification cluster to which the target machining feature belongs based on a pre-divided classification cluster; Based on the process parameters of the target machining feature at the current moment and the time-domain characteristic data of the cutting force signals in the X, Y and Z axes during the machining process, the current observation of the target machining feature is constructed. The current observation is input into the process parameter control model corresponding to the target classification cluster to obtain the process parameters of the target processing feature at the next moment; Train the process parameter control models corresponding to different taxonomic clusters according to the following steps: The cutting force signals of each machining feature of a thin-walled part in the X-axis, Y-axis and Z-axis directions during the machining process are collected at a preset frequency, and multiple time-domain features are extracted. The multiple time-domain features include: mean, maximum value, standard deviation, root mean square value, peak-to-peak value, skewness index, kurtosis index, peak index and impulse index. Using principal component analysis, the 9-dimensional time-domain feature data corresponding to the X-axis, Y-axis and Z-axis directions are dimensionality-reduced. Then, Hermite interpolation algorithm is used to augment the 9-dimensional time-domain feature data, process parameters and machining features to obtain the cutting force feature set. The workpiece-level multi-task estimation model is pre-trained using the process parameters of the machining features and the cutting force feature set; Construct a multi-task estimation model at the processing feature level for each category cluster. After each processing feature level multi-task estimation model inherits some parameters from the workpiece level multi-task estimation model, fine-tune each processing feature level multi-task estimation model to obtain the processing index estimation model for each category cluster. Each classification cluster is modeled as an agent, and a multi-agent environment state is constructed; the multi-agent environment state includes the state of the agent corresponding to each classification cluster in... Observation of time: ;in, , Indicates the first Processing features in each cluster Process parameters at any given time, This indicates the multi-task time series prediction model of cutting force. The predicted first Each cluster in Time-domain characteristics of the cutting force signal at time step. , Indicates the number of taxonomic clusters; The processing index estimation model corresponding to each taxonomic cluster is used as a virtual environment, and the observation is input. , obtained the Each cluster in Processing quality indicators and processing efficiency indicators at any given time; According to the first Each cluster in The processing quality and efficiency indicators at each moment are used to calculate the actions performed by all agents. Combined reward value ; Observation Enter the first Each classification cluster corresponds to an actor network, and the actor evaluation network within that actor network is based on observations. , Timing-based process parameter control strategies as well as Added noise Generate the first Each cluster in Moment of action and update Observation of time Among them, the first Initial observations of each taxonomic cluster From length of The window is randomly captured from the multi-agent environment state; Observation Enter the first The actor target network in the actor network corresponding to each classification cluster is based on observation. Process parameter control strategies as well as Added noise , generate the first Each cluster in Moment of action ; All intelligent agents in Joint observation at different times and joint actions Input the overall critic network, and use the parameters of the overall critic network respectively. , The first evaluation network and the second evaluation network calculate the Q-value sequence. , And using the parameters in the overall critic network as , The first target network and the second target network calculate the target Q-value sequence. , ; Based on the aforementioned joint reward value Target Q-value sequence , and Q-value sequence , Calculate the first time-series difference error Second timing difference error ; Based on the first timing difference error Second timing difference error Calculate the loss value of the second preset loss function, and update the parameters of the first evaluation network and the second evaluation network based on the loss value of the second preset loss function; The parameters of the actor evaluation network are updated using policy gradients; Based on the updated parameters of the first evaluation network and the updated parameters of the second evaluation network, the parameters of the first target network and the second target network are softly updated, and the parameters of the actor target network are further updated based on the updated parameters of the actor evaluation network. At the current iteration number When the second preset number of iterations is reached, the actor target network in each actor network is used as the process parameter control model for the processing features in each classification cluster.

2. The method for controlling process parameters of thin-walled parts based on multi-agent reinforcement learning according to claim 1, characterized in that, The taxonomic clusters are divided according to the following steps: Obtain the processing index data for each processing feature of the thin-walled part. The processing index data includes processing efficiency index data and processing quality index data. The processing efficiency index data of the processing feature is the milling completion time of the processing feature, and the processing quality index data of the processing feature is the error between the actual size and the design size of the processing feature. The processing index data of each processing feature is input into the stacked autoencoder to obtain the deep features of each processing feature, forming a deep feature set: ； In the formula, Indicates the first thin-walled part The depth features of each processing feature This indicates the number of encoder-decoder combinations in the stacked autoencoder. They represent the first Processing quality index data and processing efficiency index data for each processing feature; Calculate the maximum mutual information coefficient between any two deep features in the set of deep features; when When it exceeds the preset threshold, the first The processing feature and the first Each processing feature is grouped into the same classification cluster; among them... express and The maximum mutual information coefficient, ; By traversing all depth features in the depth feature set, all machining features of thin-walled parts are classified into different clusters.

3. The method for controlling process parameters of thin-walled parts based on multi-agent reinforcement learning according to claim 2, characterized in that, The steps of pre-training the workpiece-level multi-task estimation model using process parameters and the cutting force feature set include: The process parameters and feature data of the machining features are input into the constructed workpiece-level multi-task estimation model to obtain the predicted data of machining efficiency index and machining quality index of thin-walled parts. The workpiece-level multi-task estimation model includes a first expert layer, a first gating network and a first estimation tower layer. The process parameters include spindle speed, feed rate and depth of cut. The feature data are: time-domain feature data corresponding to the X-axis, Y-axis and Z-axis directions extracted from the cutting force feature set using a sliding window. Based on the predicted data of the processing efficiency index and the predicted data of the processing quality index of the thin-walled parts, as well as the data of the processing efficiency index and the processing quality index of the thin-walled parts, the root mean square error is calculated to measure the performance of the workpiece-level multi-task estimation model. At the current iteration number When the first preset number of iterations is reached, the best-performing generation of workpiece-level multi-task estimation model is used as the pre-trained workpiece-level multi-task estimation model.

4. The method for controlling process parameters of thin-walled parts based on multi-agent reinforcement learning according to claim 3, characterized in that, The steps of constructing a multi-task estimation model at the processing feature level for each classification cluster, allowing each processing feature-level multi-task estimation model to inherit some parameters from the workpiece-level multi-task estimation model, and then fine-tuning each processing feature-level multi-task estimation model to obtain the processing index estimation model for each classification cluster include: Construct a multi-task estimation model at the processing feature level for each classification cluster. The multi-task estimation model at the processing feature level includes: a second expert layer, a second gating network, and a second estimation tower layer. The second expert layer in each processing feature-level multi-task estimation model inherits the parameters of the first expert layer in the pre-trained workpiece-level multi-task estimation model, and the process parameters and feature data of the processing features are input into the processing feature-level multi-task estimation model corresponding to each classification cluster to obtain the processing efficiency index prediction data and the processing quality index prediction data of each classification cluster. Based on the predicted data of processing efficiency index of each category cluster, the predicted data of processing quality index of each category cluster, and the data of processing efficiency index and processing quality index of each category cluster, calculate the loss value of the first preset loss function. Based on the loss value, the parameters of the second gating network and the second estimation tower are adjusted through backpropagation to obtain the processing index estimation model corresponding to each category.

5. The method for controlling process parameters of thin-walled parts based on multi-agent reinforcement learning according to claim 2, characterized in that, The combined reward value is calculated using the following formula. : ； ； ； In the formula, Indicates the first The number of processing features contained in each feature cluster. Indicates the first The objective function for optimizing process parameters of a feature-based classification cluster. Indicates the first The first feature classification cluster One processing quality indicator, Indicates the first The first feature classification cluster Ideal values ​​for each processing quality indicator Indicates the first The spindle speed of each feature classification cluster Indicates the first The feed rate for each feature classification cluster Indicates the first The cutting depth of each feature classification cluster Indicates the first The number of processing quality indicators for each feature classification cluster. Indicates the first The first feature classification cluster One processing efficiency indicator Indicates the first The first feature classification cluster The ideal value of a processing efficiency index Indicates the first The number of processing efficiency indicators for each feature classification cluster.

6. The method for controlling process parameters of thin-walled parts based on multi-agent reinforcement learning according to claim 5, characterized in that, Based on the aforementioned joint reward value Target Q-value sequence , and Q-value sequence , Calculate the first time-series difference error Second timing difference error The steps include: Based on the target Q value sequence , and the aforementioned joint reward value ,calculate Multi-agent temporal difference objective at any given time: ； In the formula, Indicates the discount factor; Calculate separately Multi-agent temporal difference objective With Q-value sequence , The resulting multi-agent temporal difference error: ； ； In the formula, , , , Representing the Q-value sequences respectively , The first in Values.

7. The method for controlling process parameters of thin-walled parts based on multi-agent reinforcement learning according to claim 6, characterized in that, The second preset loss function is: ； In the formula, This indicates the sampled number obtained according to the priority experience playback mechanism. The sampling weights of each Markov tuple This represents the batch size from which Markov tuples are drawn from the experience replay buffer.

8. The method for controlling process parameters of thin-walled parts based on multi-agent reinforcement learning according to claim 7, characterized in that, Based on the updated parameters of the first evaluation network, the parameters of the first target network are softly updated: ； In the formula, This represents the updated parameters of the first evaluation network. Indicates soft update coefficient Based on the updated parameters of the second evaluation network, the parameters of the second target network are softly updated: ； In the formula, This represents the updated parameters of the second evaluation network; The parameters of the actor target network are updated based on the updated parameters of the actor evaluation network: ； In the formula, This indicates the updated parameters of the actor evaluation network.

9. The method for controlling process parameters of thin-walled parts based on multi-agent reinforcement learning according to claim 8, characterized in that, The first evaluation network, the second evaluation network, the first target network, and the second target network have the same structure, each including: The input layer is used to receive input from all agents. Joint observation at different times and joint actions ; The local attention encoding layer is used to encode the observation-action pairs of each agent through a multi-head attention mechanism, resulting in local encoding. ; A global attention interaction layer is used to adjust local encodings. The global feature vector representing the dependencies between agents is calculated through a multi-head attention mechanism; The splicing layer is used to splice the local codes. With the global feature vector; Output a linear layer for calculating the Q-value sequence based on the splicing result of the splicing layer. , or target Q-value sequence , .

Images