A hand-made mold curved surface machining parameter scheduling method and system based on reinforcement learning

Abstract

The application relates to a hand tool mold curved surface processing parameter scheduling method and system based on reinforcement learning, and particularly relates to the field of artificial intelligence. Through meta-training and inverse reinforcement learning, historical data and a small amount of expert demonstration are used to quickly establish an intelligent agent with general prior knowledge and a clear goal for a new processing task. The uncertainty perception strategy constructed by the application can dynamically explore under the constraint of a safety threshold, ensuring that the data collection process is efficient and reliable. The rapid adaptive update based on the collected data can quickly fine-tune the general strategy into a high-performance special-purpose strategy for the current task. The online fine-tuning and continuous monitoring mechanism further ensures the robust evolution of the strategy and the safety warning of abnormal states in long-term operation, thereby realizing high-quality, efficient and self-adaptive processing of different materials and curved surfaces.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and more specifically, to a method and system for scheduling machining parameters of mold surfaces for figurines based on reinforcement learning. Background Technology

[0002] As the high-end consumer market continues to demand more refined and personalized figurine products, the mold manufacturing industry is facing increasingly severe technological challenges. The geometric shapes of figurine molds are becoming increasingly complex, incorporating a large number of free-form surfaces, micro-textures, and thin-walled features, which places extremely high demands on processing accuracy and surface quality. At the same time, in order to meet diverse designs, new mold materials such as high-performance composite materials, special metal alloys, and photocurable resins are constantly being introduced, and their cutting characteristics often lack mature process database support. In addition, small-batch, rapid-response production models for limited editions and customized orders have become an industry trend, which requires the manufacturing system to have a strong ability to quickly adapt to new mold models and even new materials. Against this backdrop, the precise scheduling of CNC machining parameters has become the key to ensuring mold quality, cost, and delivery cycle.

[0003] However, existing technologies struggle to effectively support rapid optimization and adaptive scheduling of machining parameters for unknown materials or novel surface designs. Current mainstream methods primarily rely on physical process simulation to construct digital twin models or on parameter libraries based on limited expert experience. The former requires establishing high-fidelity mechanical and thermal simulation models, which are costly and often contain significant deviations in the constitutive relation descriptions of new materials, leading to a gap between simulation results and physical reality that is difficult to bridge, and the optimized parameter strategies may fail in actual machine tools. The latter, limited by the scope and subjectivity of expert knowledge, often fails to provide effective guidance when facing entirely new machining tasks, or provides conservative parameters that are difficult to exploit the process potential. In recent years, although some research has... Attempts have been made to introduce reinforcement learning techniques into machining parameter optimization. The agent learns the optimal strategy autonomously through trial and error with the environment. However, this approach reveals fundamental flaws when dealing with cold-start scenarios. Deep reinforcement learning algorithms typically require massive amounts of interactive data samples for training. If the exploration starts directly on a real machine tool from random strategies, a large number of defective or even scrap products will inevitably be generated in the initial learning stage, accompanied by huge tool wear and wasted machine time, making the economic costs and risks unbearable. Therefore, there is an urgent need to develop an intelligent parameter scheduling method that can make full use of limited prior knowledge and historical data to achieve safe and efficient initial exploration, and can quickly converge to a high-performance machining strategy after a small number of trial cuts, in order to break through the key bottleneck of intelligent manufacturing of small-batch customized molds. Summary of the Invention

[0004] This invention addresses the technical problems existing in the prior art by providing a method and system for scheduling surface machining parameters of figurine molds based on reinforcement learning, thereby solving the problems mentioned in the background art.

[0005] The technical solution of this invention to solve the above-mentioned technical problems is as follows: a method for scheduling surface machining parameters of figurine molds based on reinforcement learning, specifically including the following steps:

[0006] Step S1: Obtain historical processing task datasets and expert demonstration data for new processing tasks; perform meta-training on a reinforcement learning policy network for outputting processing parameters based on the historical processing task datasets to obtain meta-initialized policy parameters; based on the expert demonstration data, use an inverse reinforcement learning algorithm to infer the reward function suitable for the new processing task.

[0007] Step S2: Using the initialization policy parameters and reward function as initial inputs, construct an uncertainty-aware policy network that integrates a Bayesian neural network; while exploring the actual processing environment, calculate in real time the uncertainty estimate of the action suggested by the uncertainty-aware policy network for the current processing state; based on the uncertainty estimate, dynamically adjust the randomness of the action exploration, and when the uncertainty estimate exceeds a preset safety threshold, activate a conservative policy guided by the reward function to replace the uncertainty-aware policy network in outputting the final action, thereby collecting experience data for new processing tasks;

[0008] Step S3: Using the collected experience data of the new processing task and the reward function, perform a fast adaptive update based on gradient descent on the meta-initialized policy parameters to obtain the fine-tuned processing policy parameters;

[0009] Step S4: Load the finely tuned processing strategy parameters into the uncertainty-aware strategy network to generate actual processing instructions; continuously collect real-time processing status data and processing result feedback during the processing process and store them in a pre-built experience playback buffer; periodically sample data from the experience playback buffer, use the sampled data to finely adjust and update the finely tuned processing strategy parameters online, and continuously monitor the uncertainty generated by the processing strategy parameters during decision-making, and trigger a safety alarm when continuous high uncertainty is detected.

[0010] In a preferred embodiment, the specific process of performing meta-training on a reinforcement learning policy network for outputting processing parameters based on a historical processing task dataset to obtain meta-initialized policy parameters is as follows:

[0011] Construct a parameterized reinforcement learning policy network that receives processing states as input and outputs processing parameters and actions. The weights and biases of this reinforcement learning policy network are collectively referred to as policy network parameters.

[0012] Extract a historical processing task instance from the historical processing task dataset, and divide the processing data contained in the historical processing task instance into a support set and an evaluation set;

[0013] The loss metric of the current policy network parameters on the extracted historical task is calculated based on the processing data of the support set. Based on this loss metric, the policy network parameters are updated in one step using the gradient descent principle to generate a temporary local policy network parameter that is adapted to the extracted historical task.

[0014] Subsequently, using the processing data in the evaluation set, the generalization loss of this local policy network parameters on the extracted historical tasks is calculated; the entire process of randomly selecting task instances, splitting the data, calculating the support set loss, updating and generating local parameters, and calculating the evaluation set generalization loss is repeated on multiple different tasks included in the historical processing task dataset.

[0015] By iterating continuously, the initial policy network parameters are adjusted to minimize the sum of generalization losses accumulated over all historical tasks; when the iteration process converges, the final policy network parameters are defined as the meta-initialized policy parameters.

[0016] In a preferred embodiment, the specific process of inferring the reward function suitable for the new processing task based on expert demonstration data using an inverse reinforcement learning algorithm is as follows:

[0017] The expert demonstration data is modeled as a trajectory sequence composed of states and processing parameters arranged in chronological order; a parameterized reward function model is initialized, and the internal weights of the reward function model are collectively referred to as reward function parameters;

[0018] The optimization objective of the inverse reinforcement learning algorithm is to adjust the reward function parameters so that the expert demonstration trajectory in the trajectory sequence has the maximum likelihood probability in the probability distribution guided by the current reward function, while introducing the maximum entropy constraint to avoid overfitting to the expert behavior.

[0019] During the iteration process, based on the current reward function parameters, a temporary policy model for sampling is trained through a policy optimization algorithm, and a series of non-expert trajectories are generated through sampling.

[0020] The cumulative reward value of the expert demonstration trajectory under the current reward function is calculated, and the cumulative reward value of the sampled non-expert trajectory under the current reward function is also calculated. The gradient estimation logic of the reward function parameters is as follows: calculate the difference in cumulative rewards between the expert demonstration trajectory and the non-expert trajectory under the current reward function; calculate the log-likelihood ratio of the expert demonstration trajectory and the non-expert trajectory under the probability distribution induced by the current reward function; multiply the cumulative reward difference by the log-likelihood ratio, and calculate the expectation over all possible trajectories to obtain the gradient estimate; use this gradient estimate to update the reward function parameters according to the predetermined learning rate.

[0021] Repeat the iterative process, which includes sampling, calculating cumulative reward, calculating gradient estimate, and updating parameters, until the reward function parameters make the cumulative reward of the expert demonstration trajectory significantly higher than that of the non-expert trajectory, and the trajectory distribution entropy induced by the reward function remains at a high level. The reward function is determined by the obtained reward function parameters.

[0022] In a preferred embodiment, the specific process of constructing an uncertainty-aware policy network integrating a Bayesian neural network in step S2 is as follows:

[0023] The meta-initialization policy parameters are loaded as the core weights of the uncertainty-aware policy network; in the output layer of the uncertainty-aware policy network, a parameter structure is added to generate a vector representing the discreteness of the action distribution.

[0024] The uncertainty-aware policy network takes the current processing state acquired in real time as input, and after transformation by multiple fully connected layers and nonlinear activation functions contained within the uncertainty-aware policy network, it outputs the action mean vector and the action log variance vector in parallel. The action log variance vector is then transformed by a nonlinear transformation function that ensures that the output value is always positive to obtain the action variance vector.

[0025] The action mean vector and the action variance vector together define a multidimensional Gaussian probability distribution. The mean of this multidimensional Gaussian probability distribution is determined by the action mean vector, and its covariance matrix is ​​determined by a diagonal matrix formed by using the action variance vector as its diagonal elements.

[0026] The uncertainty estimate of the action suggested by the uncertainty-aware policy network for the current processing state is obtained by calculating the sum of all diagonal elements of the covariance matrix.

[0027] In a preferred embodiment, the process of dynamically adjusting the action exploration and activating the conservative strategy is as follows:

[0028] A safety threshold is pre-set; at each decision moment when interacting with the processing environment, the uncertainty estimate for the current processing state is obtained by the uncertainty-aware policy network.

[0029] The current uncertainty estimate is compared with the safety threshold. If the current uncertainty estimate is less than the safety threshold, the final processing parameters are sampled from the multidimensional Gaussian probability distribution defined by the uncertainty-aware strategy network.

[0030] If the current uncertainty estimate is greater than the safety threshold, the uncertainty-aware policy network is ignored, and instead an optimization problem is solved: find a processing parameter action such that the output value of the reward function obtained in step S1 under the current processing state and this processing parameter action is as large as possible, while minimizing the squared Euclidean distance between this processing parameter action and a preset nominal safety action.

[0031] Nominal safety action is a set of predefined, known conservative machining parameter values ​​that can guarantee the basic safety of the machining process.

[0032] The obtained result is used as the final processing parameter action, i.e. the final action, and applied to the actual processing environment; the new processing state generated by the processing environment after receiving the processing parameter action is obtained; and an immediate reward value is obtained.

[0033] The quadruple data generated during this decision-making and interaction process, including the processing state at the moment of decision, the final processing parameter action, the immediate reward value obtained, and the new processing state generated, are stored as a piece of experience data in the experience data set pre-built for new processing tasks.

[0034] This process is repeated cyclically, continuously collecting experience data for new processing tasks.

[0035] In a preferred embodiment, in step S3, the collected experience data from the new processing task and the reward function are used to perform a fast adaptive update of the meta-initialization policy parameters based on gradient descent. The specific process is as follows:

[0036] Based on the reward function and the empirical data collected for the new processing task, calculate the advantage function estimate for each processing state and processing parameter action pair in the empirical data collected for the new processing task.

[0037] The process of calculating the advantage function estimate is as follows: through temporal difference learning, a target value is recursively calculated. This target value is equal to the sum of the immediate reward value plus a discount factor and the estimated reward of the new processing state. This target value is used as the reward estimate of the current processing state.

[0038] Then, for each state-action pair, the estimated reward of the current processing state calculated above is compared and combined with the sum of the immediate reward value obtained from the empirical data collected for the new processing task plus the discount factor and the estimated reward of the subsequent new processing state to form a basic advantage estimate.

[0039] Finally, the fundamental advantage estimate at the current moment is mixed with the fundamental advantage estimates at multiple consecutive future moments, and the fundamental advantage estimates at more distant moments are given exponentially decaying weights. A final advantage function estimate is obtained by weighted summation.

[0040] In a preferred embodiment, the gradient descent-based fast adaptive update specifically involves the following process:

[0041] First, define the policy probability ratio, which is the ratio of the probability density of the same processing parameter action recorded in the empirical data collected for a new processing task under the given processing state, between the policy defined by the policy parameters to be optimized and the policy defined by the fixed meta-initialized policy parameters; the initial value of the policy parameters to be optimized is set to the meta-initialized policy parameters.

[0042] Next, a pruning objective function is constructed. The calculation logic of the pruning objective function is as follows: multiply the policy probability ratio by the corresponding advantage function estimate to obtain an unpruned update objective term.

[0043] Meanwhile, the policy probability ratio is restricted to a closed interval centered on the value 1 and with a preset positive small pruning parameter as the radius, to obtain a pruned probability ratio. The pruned probability ratio is then multiplied by the same advantage function estimate to obtain the pruned updated target term.

[0044] Finally, the smaller value between the unpruned update target item and the pruned update target item is taken as the effective contribution of this state action to the pruning objective function;

[0045] The effective contributions of all state-action pairs in the empirical data collected for the new processing task are summed to form the overall adaptive update target;

[0046] With the goal of maximizing this overall adaptive update, the stochastic gradient ascent algorithm is used to perform a finite number of iterative updates on the processing strategy parameters. The updated parameters are the finely tuned processing strategy parameters.

[0047] In a preferred embodiment, the online fine-tuning and updating of the refined processing strategy parameters using sampled data is specifically performed using an elastic weight consolidation algorithm, and the specific update process is as follows:

[0048] Before performing online fine-tuning, the importance weights required for the elastic weight consolidation algorithm are first calculated. The calculation process is as follows: based on the empirical data collected in step S3 for rapid adaptation to new processing tasks, the importance weight of each adjustable parameter in the fine-tuned processing strategy parameters is calculated. This importance weight is used to measure the criticality of the parameter in maintaining the performance of the learned processing strategy. Its value is estimated by calculating the diagonal elements of the Fisher information matrix of the parameter on the empirical data.

[0049] The finely tuned processing strategy parameters are loaded into the uncertainty-aware strategy network, replacing the original processing strategy parameters of the uncertainty-aware strategy network, so that the uncertainty-aware strategy network can generate actual processing instructions based on the real-time processing status.

[0050] During the processing, real-time processing status and result feedback are continuously collected, and online experience data is stored in a pre-built experience playback buffer.

[0051] During online fine-tuning, a small batch of the latest processing experience data is sampled from the experience playback buffer;

[0052] Based on the latest processing experience data obtained from sampling, a strategy optimization loss is calculated, and the initial value of the current processing strategy parameter is the fine-tuned processing strategy parameter.

[0053] Meanwhile, an elastic weight consolidation regularization term is constructed, which penalizes the deviation between the current processing strategy parameters and the fine-tuned processing strategy parameters, and the penalty strength for each parameter is proportional to its pre-calculated importance weight.

[0054] The total loss is obtained by adding the policy optimization loss to the scaled regularization term. Gradient descent is then performed to update the parameters of the current processing policy by minimizing the total loss.

[0055] In a preferred embodiment, the uncertainty generated during decision-making by the strategy of continuously monitoring processing strategy parameters, and triggering a safety alarm when persistent high uncertainty is detected, specifically involves the following process:

[0056] By utilizing the uncertainty-aware policy network loaded with the current processing policy parameters, the random deactivation mechanism inside the uncertainty-aware policy network is enabled during the inference process. Multiple forward propagation calculations are performed in the Monte Carlo random deactivation manner to obtain several slightly different action probability distributions.

[0057] Calculate the Jensen-Shannon divergence between each pair of action probability distributions obtained from multiple forward propagations, and take the average of the divergence values ​​for all pairwise combinations. Define this average as the decision inconsistency measure at the current decision moment.

[0058] Maintain a sliding window of fixed time length and calculate the exponentially weighted moving average of the integrated decision inconsistency measure for the most recent series of decision moments within the sliding window;

[0059] This exponentially weighted moving average is compared to a dynamic safety threshold, which is pre-set based on the distribution characteristics of integrated decision inconsistency measures statistically analyzed during historical safe operation phases.

[0060] If the number of decision moments in which the index-weighted moving average exceeds the dynamic safety threshold consecutively reaches a preset alarm trigger count, it is determined that a state of continuous high uncertainty has been entered, and the following safety response is immediately triggered: an alarm message containing the current status and uncertainty value is sent to the operation interface.

[0061] This application also provides a parameter scheduling system for surface machining of figurine molds based on reinforcement learning, specifically including: a meta-training and reward inference module, a safety exploration and data collection module, a rapid adaptation module, and an online operation and monitoring module connected in sequence, wherein;

[0062] Meta-training and reward inference module: acquires historical processing task datasets and expert demonstration data for new processing tasks; performs meta-training on the reinforcement learning policy network based on the historical processing task datasets and outputs meta-initialized policy parameters; and outputs the reward function for the new processing task using an inverse reinforcement learning algorithm based on the expert demonstration data.

[0063] Safety Exploration and Data Collection Module: Receives initialization policy parameters and reward function; constructs an uncertainty-aware policy network integrating Bayesian neural network; calculates the uncertainty estimate of the action suggested by the uncertainty-aware policy network; adjusts the exploration randomness accordingly; and when the preset safety threshold is exceeded, activates a conservative policy guided by the reward function to output the final action, thereby collecting and outputting experience data for new processing tasks.

[0064] The rapid adaptation module receives experience data and a reward function for new processing tasks; it then uses this experience data and reward function to perform a rapid adaptive update based on gradient descent on the meta-initialized policy parameters, and outputs the fine-tuned processing policy parameters.

[0065] Online operation and monitoring module: Receives fine-tuned processing strategy parameters and loads them into the uncertainty-aware strategy network to generate actual processing instructions; continuously collects real-time processing status and result feedback during processing and stores them in a pre-built experience playback buffer; periodically samples data from the experience playback buffer to fine-tune and update the fine-tuned processing strategy parameters online; continuously monitors the uncertainty generated during strategy decision-making and triggers a safety alarm when high uncertainty is detected.

[0066] The beneficial effects of this invention are as follows: through meta-training and inverse reinforcement learning, and utilizing historical data and a small number of expert demonstrations, an intelligent agent with general prior knowledge and clear objectives can be quickly established for a completely new processing task; the uncertainty perception strategy it constructs can dynamically explore under safety threshold constraints, ensuring that the data collection process is efficient and reliable; based on the rapid adaptive updates performed on the collected data, the general strategy can be quickly fine-tuned into a high-performance dedicated strategy for the current task; the online fine-tuning and continuous monitoring mechanism further ensures the robust evolution of the strategy and the safety warning of abnormal states during long-term operation, thereby achieving high-quality, efficient, and adaptive processing of different materials and surfaces. Attached Figure Description

[0067] Figure 1 This is a flowchart of the method of the present invention;

[0068] Figure 2 This is a block diagram of the system structure of the present invention. Detailed Implementation

[0069] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0070] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0071] In the description of this application, the term "for example" is used to mean "used as an example, illustration, or description." Any embodiment described as "for example" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use the invention. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that the invention can be made without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid obscuring the description of the invention with unnecessary detail. Therefore, the invention is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.

[0072] Example 1

[0073] This embodiment provides, for example Figure 1 The method for scheduling surface machining parameters of figurine molds based on reinforcement learning is shown, and specifically includes the following steps:

[0074] Step S1: Obtain historical processing task datasets and a small amount of expert demonstration data for new processing tasks; perform meta-training on a reinforcement learning policy network for outputting processing parameters based on the historical processing task datasets to obtain meta-initialized policy parameters; and infer a reward function suitable for the new processing task using an inverse reinforcement learning algorithm based on the expert demonstration data.

[0075] Step S2: Using the initialization policy parameters and reward function as initial inputs, construct an uncertainty-aware policy network that integrates a Bayesian neural network; while exploring the actual processing environment, calculate in real time the uncertainty estimate of the action suggested by the uncertainty-aware policy network for the current processing state; based on the uncertainty estimate, dynamically adjust the randomness of the action exploration, and when the uncertainty estimate exceeds a preset safety threshold, activate a conservative policy guided by the reward function to replace the uncertainty-aware policy network in outputting the final action, thereby collecting experience data for new processing tasks;

[0076] Step S3: Using the collected experience data of the new processing task and the reward function, perform a fast adaptive update based on gradient descent on the meta-initialized policy parameters to obtain the fine-tuned processing policy parameters;

[0077] Step S4: Load the finely tuned processing strategy parameters into the uncertainty-aware strategy network to generate actual processing instructions; continuously collect real-time processing status data and processing result feedback during processing and store them in a pre-built experience playback buffer; periodically sample data from the experience playback buffer and use the sampled data to finely adjust and update the finely tuned processing strategy parameters online, while continuously monitoring the uncertainty generated by the processing strategy parameters during decision-making, and triggering a safety alarm or strategy rollback when continuous high uncertainty is detected.

[0078] In this embodiment, it is specifically necessary to explain the process in step S1, which involves meta-training a reinforcement learning policy network for outputting processing parameters based on a historical processing task dataset to obtain the meta-initialized policy parameters.

[0079] A parameterized reinforcement learning policy network is constructed. This network receives the machining state as input and outputs machining parameter actions. The weights and biases of this network are collectively referred to as policy network parameters. The specific structure of the reinforcement learning policy network can be a multilayer perceptron, with the number of input layer nodes being the same as the feature dimension of the machining state, and the number of output layer nodes being the same as the number of adjustable machining parameters. It contains at least one hidden layer with a nonlinear activation function. The machining state features can include the current spindle load, the current position coordinates of the tool on the surface, and the local surface curvature. The machining parameter actions can include the spindle speed adjustment and the feed rate adjustment. The policy network parameters are the set of all weight matrices and bias vectors in this multilayer perceptron.

[0080] A historical machining task instance is randomly selected from the historical machining task dataset according to a probability distribution. The machining process data contained in the historical machining task instance is divided into a support set and an evaluation set. The historical machining task dataset contains machining records of workpieces with various materials and various surface geometries. Each historical machining task instance corresponds to a complete machining process record of a specific material and surface combination. The ratio of the support set to the evaluation set is usually set so that the support set accounts for 10% to 30% of the total data of the task instance, and the evaluation set accounts for the remaining part. The partitioning process is random to ensure that the data distribution of the two subsets is consistent.

[0081] The loss metric of the current policy network parameters on the extracted historical task is calculated based on the processing data in the support set. Based on this loss metric, the policy network parameters are updated in one step using the gradient descent principle to generate a temporary local policy network parameter adapted to the extracted historical task. The calculation process of the loss metric is as follows: each processing state in the support set is input into the current reinforcement learning policy network to obtain the proposed processing parameter action output by the reinforcement learning policy network; the mean square error between the proposed action and the action executed in the actual historical record is calculated; the average mean square error of all support set data points is calculated, and this average value is the loss metric.

[0082] The specific implementation process of gradient descent is as follows: calculate the partial derivative of the loss metric with respect to each element of the policy network parameters to form the gradient vector; subtract the product of a preset internal learning rate and the gradient vector from the current value of the policy network parameters to obtain the updated local policy network parameters; the internal learning rate is a small positive number, such as 0.001.

[0083] Subsequently, using the processing data in the evaluation set, the generalization loss of this local policy network parameters on the extracted historical tasks is calculated. The entire process of randomly selecting task instances, splitting the data, calculating the support set loss, updating and generating local parameters, and calculating the evaluation set generalization loss is repeated across multiple different tasks included in the historical processing task dataset. The calculation of the generalization loss is similar to the calculation of the support set loss, but uses data from the evaluation set. Its purpose is to evaluate the performance of the local policy network parameters on new data that has not been used for updates, thereby measuring the effectiveness of rapid adaptation. Repeating the entire process means performing multiple iterations. In each iteration, a new historical processing task instance is randomly selected from the historical processing task dataset, and all steps from splitting to calculating the generalization loss are fully executed.

[0084] By iteratively adjusting the initial policy network parameters, the total generalization loss accumulated over all historical tasks is minimized. Once the iteration converges, the final policy network parameters are defined as the meta-initialized policy parameters. The adjustment of the initial policy network parameters is implemented using a meta-optimizer, such as the Adam optimizer. Its objective function is the sum of the generalization losses of all sampled historical task instances. In each iteration, the gradient of this sum with respect to the initial policy network parameters is calculated, and the initial policy network parameters are updated. The convergence criterion can be that the decrease in total loss over multiple consecutive iterations is less than a preset convergence threshold, such as 1e-5, or that a preset maximum number of iterations is reached, such as 10,000. This meta-initialized policy parameter enables the reinforcement learning policy network to quickly generate an effective policy for a new task based on limited supporting data and a few similar gradient updates.

[0085] Based on expert demonstration data, the specific process of inferring the reward function suitable for the new processing task using the inverse reinforcement learning algorithm is as follows:

[0086] The expert demonstration data is modeled as a trajectory sequence composed of states and processing parameter actions arranged in chronological order; a parameterized reward function model is initialized, and the internal weights of this reward function model are collectively referred to as reward function parameters; the reward function model can be a linear function or a shallow neural network; its input is a concatenated vector of processing states and processing parameter actions, and its output is a scalar reward value; the reward function parameters are the weights of this reward function model; initialization is usually done randomly, for example, sampling from a normal distribution with a mean of 0 and a small variance;

[0087] The optimization objective of the inverse reinforcement learning algorithm is to adjust the reward function parameters so that the expert demonstration trajectory in the trajectory sequence has the maximum likelihood probability in the probability distribution guided by the current reward function. At the same time, a maximum entropy constraint is introduced to avoid overfitting to the expert behavior. The maximum entropy constraint means that among all possible trajectories with the same cumulative reward as the expert demonstration trajectory, no trajectory is presumed to be more likely, and its distribution is the most uniform. This is achieved by introducing the logarithmic term of the partition function into the log-likelihood function.

[0088] During the iterative optimization process, a temporary policy model for sampling is first trained using a policy optimization algorithm based on the current reward function parameters. This temporary policy model is then used to sample the environment and generate a series of non-expert trajectories. The policy optimization algorithm can be a proximal policy optimization algorithm. When training the temporary policy model, the goal is to maximize the expected cumulative discounted return under the current reward function. The number of non-expert trajectories generated by sampling is a preset value, such as 100, to ensure the stability of gradient estimation.

[0089] The cumulative reward value of the expert demonstration trajectory under the current reward function is calculated, and the cumulative reward value of the sampled non-expert trajectories under the current reward function is also calculated. The calculation process of the cumulative reward value is as follows: for each state-action pair in a trajectory, the immediate reward is calculated using the current reward function model; all immediate rewards for the entire trajectory are summed, or summed with a discount factor, where the discount factor is usually set to a value slightly less than 1, such as 0.99. The gradient estimate of the reward function parameters is calculated as follows: the difference between the cumulative rewards of the expert demonstration trajectory and the non-expert trajectory under the current reward function is calculated; the log-likelihood ratio of the expert demonstration trajectory and the non-expert trajectory under the probability distribution induced by the current reward function is calculated; the cumulative reward difference is multiplied by the log-likelihood ratio, and the expectation is calculated for all possible trajectories to obtain the gradient estimate. The specific calculation process of the gradient estimate is as follows: First, calculate the average cumulative reward for all expert demonstration trajectories and the average cumulative reward for all non-expert trajectories; the difference between the two is the average cumulative reward difference. Second, calculate the likelihood probability of a single trajectory, which is proportional to the exponential function value of the trajectory's cumulative reward. The log-likelihood ratio between expert and non-expert trajectories is calculated as the average log-likelihood of expert trajectories minus the average log-likelihood of non-expert trajectories. Then, multiply the average cumulative reward difference by this log-likelihood ratio. Finally, this product is an unbiased estimate of the gradient of the reward function parameters. Perform the above calculations on all sampled trajectories and take the average to approximate the "expectation over all possible trajectories." Using this gradient estimate, update the reward function parameters according to a predetermined learning rate. The learning rate is a preset positive number, such as 0.01. The update process uses gradient ascent, adding the current reward function parameters to the product of the learning rate and the gradient estimate.

[0090] Repeat the iterative process, including sampling, calculating cumulative reward, calculating gradient estimate, and updating parameters, until the reward function parameters make the cumulative reward of the expert demonstration trajectory significantly higher than that of the non-expert trajectory, and the trajectory distribution entropy induced by the reward function remains at a high level. At this point, the optimization stops. The criterion for "significantly higher" can be set as the average cumulative reward of the expert trajectory being greater than 1.5 times the average cumulative reward of the non-expert trajectory. "The entropy remains at a high level" can be judged by calculating the trajectory distribution entropy and setting it to be no less than a preset entropy threshold, for example, the entropy value is no less than 80% of log(number of trajectories). The optimization stops when these two conditions are met, or when the maximum number of iterations (e.g., 200 times) is reached.

[0091] The reward function determined by the final reward function parameters is defined as a reward function adapted to the new processing task. This reward function can implicitly extract a comprehensive evaluation index of processing quality, efficiency and safety from expert demonstration data.

[0092] In this embodiment, it is specifically necessary to explain the process of constructing an uncertainty-aware policy network that integrates a Bayesian neural network in step S2 as follows:

[0093] The meta-initialization policy parameters are loaded as the core weights of the uncertainty-aware policy network. In the output layer of the uncertainty-aware policy network, a parameter structure is added to generate a vector representing the discreteness of the action distribution. The main structure of the uncertainty-aware policy network is a multilayer perceptron with three hidden layers, each with 128, 64, and 32 neurons respectively. The ReLU function is used for the hidden layer activation. The output layer is split into two parallel fully connected layers, one outputting the action mean vector and the other outputting the action log-variance vector. The meta-initialization policy parameters are loaded into the weight matrices and bias vectors of the network input layer and all hidden layers. The sub-network weights corresponding to the action mean vector in the output layer are also initialized by the corresponding part of the meta-initialization policy parameters. The sub-network weights corresponding to the action log-variance vector are independently initialized using random sampling from a normal distribution with a mean of 0 and a standard deviation of 0.01. No bias term is set in the output layer.

[0094] The uncertainty-aware policy network takes the real-time acquired current machining state as input. After transformation by multiple fully connected layers and nonlinear activation functions within the network, it outputs two vectors in parallel: the first vector is the action mean vector, which describes the center position of the suggested machining parameters under a given machining state; the dimension of the action mean vector is equal to the number of adjustable machining parameters, for example, 2-dimensional, corresponding to the spindle speed adjustment and feed rate adjustment, respectively; this action mean vector is directly used as the linear transformation result of the output layer without the need for additional activation functions;

[0095] The second vector is the action logarithmic variance vector, used to describe the uncertainty of each dimension of the suggested processing parameters under a given processing state. The action logarithmic variance vector is transformed using a nonlinear transformation function that ensures the output value is always positive to obtain the action variance vector. The nonlinear transformation function that ensures the output value is always positive is the softplus function, defined as: softplus(x) = log(1 + exp(x)). The calculation process of the action variance vector is as follows: for each element of the action logarithmic variance vector, the softplus function is applied one by one to obtain the action variance vector of the same dimension. This process ensures that the variance value is positive, conforming to the definition of a probability distribution.

[0096] The action mean vector and the action variance vector together define a multidimensional Gaussian probability distribution. The mean of this multidimensional Gaussian probability distribution is determined by the action mean vector, and its covariance matrix is ​​determined by a diagonal matrix formed by using the action variance vector as its diagonal elements. This multidimensional Gaussian probability distribution is an independent distribution for each dimension. The probability density function is calculated as follows: calculate the probability density of each processing parameter dimension under a univariate Gaussian distribution. The mean of this univariate Gaussian distribution is the element of the corresponding dimension of the action mean vector, and the variance is the element of the corresponding dimension of the action variance vector. Then, multiply the probability densities of all dimensions to obtain the joint probability density.

[0097] The uncertainty estimate of the action suggested by the uncertainty-aware policy network for the current processing state is obtained by calculating the sum of all diagonal elements of the covariance matrix. It is a quantitative indicator reflecting the overall grasp of the proposed action by the uncertainty-aware policy network. The calculation process of the uncertainty estimate is as follows: extract all elements of the action variance vector; sum these elements; the resulting scalar sum is the uncertainty estimate; the larger this value is, the higher the overall uncertainty of the uncertainty-aware policy network's suggestions for all adjustable processing parameters in the current state.

[0098] Based on the uncertainty estimate, the randomness of action exploration is dynamically adjusted. When the uncertainty estimate exceeds a preset safety threshold, a conservative strategy guided by a reward function is activated to replace the uncertainty-aware strategy network in outputting the final action. The specific process is as follows:

[0099] A pre-defined value greater than zero is used as a safety threshold. The safety threshold is set based on the statistical analysis of the uncertainty estimates of the meta-initialization policy network on the historical task verification set. Specifically, all states in the historical task verification set are input into the uncertainty-aware policy network to calculate a series of uncertainty estimates. The 90th percentile of these uncertainty estimates is taken as the safety threshold. For example, if the calculated 90th percentile is 2.5, the safety threshold is set to 2.5. At each decision moment when interacting with the processing environment, the uncertainty estimate calculated by the uncertainty-aware policy network for the current processing state is obtained.

[0100] The current uncertainty estimate is numerically compared with a safety threshold. If the current uncertainty estimate is less than or equal to the safety threshold, the final processing parameter action is sampled from the multidimensional Gaussian probability distribution defined by the uncertainty-aware strategy network. The variance of the probability distribution used for sampling is obtained by multiplying the original action variance vector by a scaling factor positively correlated with the current uncertainty estimate. The scaling factor is calculated by dividing the current uncertainty estimate by the safety threshold to obtain a ratio. This ratio is then used as the scaling factor. Therefore, in a low-uncertainty state, the lower the uncertainty estimate, the smaller the scaling factor, the smaller the variance of the sampling distribution is compressed, and the weaker the randomness of the exploration. The closer the uncertainty estimate is to the safety threshold, the closer the scaling factor is to 1, the amplified variance is, and the stronger the randomness of the exploration.

[0101] If the current uncertainty estimate is greater than the safety threshold, the uncertainty-aware policy network is ignored, and instead an optimization problem is solved: finding a processing parameter action that maximizes the output value of the reward function obtained in step S1 under the current processing state and this processing parameter action, while minimizing the squared Euclidean distance between this processing parameter action and a preset nominal safety action; the mathematical form of this optimization problem is a regularized maximization problem; the solution process uses the L-BFGS-B numerical optimization algorithm; the calculation process is as follows: first, the objective function is defined as the output value of the reward function minus a penalty term, where the penalty term is the regularization coefficient multiplied by the processing parameter. The square of the Euclidean distance between the numerical action and the nominal safe action is calculated. Then, the mean vector of the actions output by the uncertainty-aware strategy network is used as the initial point for optimization iteration. Next, within the feasible domain of the machining parameter actions (e.g., the spindle speed adjustment is in the range of [-500, 500] RPM, and the feed rate adjustment is in the range of [-100, 100] mm / min), the L-BFGS-B algorithm is run for iterative optimization, using the objective function value and gradient information to find the action that maximizes the objective function. Iteration stops when the change in the objective function value is less than the preset tolerance (e.g., 1e-6) or the maximum number of iterations (e.g., 100 times) is reached.

[0102] The nominal safety action is a set of predefined, known conservative machining parameter values ​​that can ensure the basic safety of the machining process. The nominal safety action is obtained by consulting the process manual based on the type of material being machined and the tool specifications. For example, for a certain aluminum alloy material, its nominal safety action can be set to a spindle speed of 2000 RPM and a feed rate of 50 mm / min.

[0103] The obtained results are used as the final machining parameters, i.e., the final action, which is applied to the actual machining environment. Specifically, through the CNC system's application programming interface, the specific values ​​of the final machining parameters, i.e., the spindle speed adjustment and feed rate adjustment, are converted into G-code or direct digital control instructions that can be executed by the machine tool and sent to the machine tool controller for execution. The duration of the action, i.e., the control cycle, is usually set to 10 milliseconds to 100 milliseconds, consistent with the interpolation cycle of the CNC system. The new machining state generated by the machining environment after receiving the machining parameters is acquired, and the new machining state is collected through the sensor network installed on the machine tool. The data acquisition and processing process of the sensors is as follows: real-time spindle load current is acquired through a spindle power sensor; real-time position coordinates of the tool center are read through a grating ruler or encoder; vibration signals are acquired through an accelerometer mounted on the tool holder; and local point cloud data of the machined surface is intermittently acquired through an online measurement probe to calculate real-time curvature. All sensor data are synchronously sampled at a frequency of not less than 1kHz. After low-pass filtering to remove high-frequency noise, a fixed-dimensional feature vector is extracted. This fixed-dimensional feature vector is the new machining state. Spindle load current, position coordinates, vibration frequency amplitude, and local curvature are jointly encoded as elements of this state vector.

[0104] The instant reward value obtained by executing the processing parameter action is calculated based on the reward function obtained in step S1. The calculation process of the instant reward value is as follows: the processing state at the decision time and the final executed processing parameter action are concatenated into a combined feature vector, which is input into the reward function model that has been trained and fixed in step S1. The reward function model is a three-layer fully connected neural network, which performs forward propagation calculation and finally outputs a scalar value, which is the instant reward value. The processing quality feedback fusion logic involved in the reward calculation is as follows: during a partial control cycle, when the surface roughness measurement value of the online measurement probe is available, the measurement value is converted into an auxiliary reward component, such as -exp(roughness value / reference value), through a negative exponential scaling function. This auxiliary reward component is then weighted and added to the basic reward value output by the reward function model with a preset mixed weight, such as 0.3, to form the final instant reward value.

[0105] The four-tuple data generated during this decision-making and interaction process, including the processing state at the moment of decision, the final executed processing parameter action, the obtained immediate reward value, and the generated new processing state, are stored as a single piece of experience data in an experience data set pre-built for new processing tasks. The specific data structure and management strategy of the experience data set are as follows: a fixed-capacity first-in-first-out queue is allocated in memory as the experience data set, with a preset capacity of 10,000 experience data entries; each piece of experience data is encoded as a tuple, and the data type of its four elements is a 32-bit floating-point array; the storage process is as follows: the processing state, processing parameter action, immediate reward value, and new processing state are serialized and appended to the tail of the queue; when the queue is full, a new storage operation will automatically overwrite the oldest data in the queue; in addition, a data integrity check is performed before storage: it checks whether the immediate reward value is a finite real number and whether the old and new processing state vectors contain non-numeric values; if the check fails, the data is discarded and not stored.

[0106] This process is repeated cyclically, continuously collecting experience data for new processing tasks. The logic and termination conditions of the loop control are as follows: the decision-making, execution, collection, calculation, and storage process is executed in a fixed control cycle. The termination conditions of the loop are controlled by two indicators: first, the amount of data stored in the experience data set reaches the preset collection target, such as 5,000 records; second, the processing process itself ends, such as the tool completing the traversal of the entire mold surface. When either condition is met, the exploration and data collection phase terminates.

[0107] In this embodiment, it is particularly important to explain step S3, where the collected experience data from the new processing task and the reward function are used to perform a fast adaptive update of the meta-initialization policy parameters based on gradient descent. The specific process is as follows:

[0108] From the experience data collected for the new processing task, all stored processing states, processing parameter actions, instant reward values, and new processing states are read. The experience data is stored in the form of quadruples, each containing a processing state vector, a processing parameter action vector, a scalar instant reward value, and a new processing state vector. This experience data was collected during the safety exploration process in step S2 and represents state transition samples under the new processing task. The reading process sequentially loads all quadruples from memory or storage media to form a batch dataset for subsequent batch computation. The size of this batch dataset is equal to the total number of experience data entries collected in step S2, typically between several hundred and several thousand.

[0109] Based on the reward function and the empirical data collected for the new processing task, the advantage function estimate is calculated for each processing state and processing parameter action pair in the empirical data collected for the new processing task. The advantage function estimate is used to quantify the superiority of the reward obtained by performing the specified processing parameter action in a specified processing state compared with the reward obtained by performing the average action in that processing state. The calculation of the advantage function estimate is a key step in policy evaluation. A value greater than zero indicates that the action is better than the average action, and a value less than zero indicates that it is worse than the average action. The calculation process depends on the value estimate of each processing state, that is, the expected cumulative reward that can be obtained from that state.

[0110] The process of calculating the advantage function estimate includes: First, using the immediate reward value and new processing state stored in the empirical data collected for new processing tasks, a target value is recursively calculated through temporal difference learning. This target value is equal to the sum of the immediate reward value plus a discount factor and the estimated reward of the new processing state. This target value serves as the reward estimate for the current processing state. The specific calculation process of the temporal difference target value is as follows: Initialize a state value function estimation network. The structure of this network is a three-layer fully connected neural network. The input is a processing state vector, and the output is a scalar value estimate. First, input all new processing states from the empirical data into this state value function estimation network to obtain the value estimate for each new processing state. Then, for each piece of empirical data, the target value of its temporal difference is equal to the immediate reward value of that data record, plus a discount factor between zero and one (usually set to 0.99), and then multiplied by the value estimate of the new processing state.

[0111] The discount factor reflects the degree to which the importance of future rewards diminishes relative to immediate rewards; this target value serves as the reward estimation target for the current processing state and is used to train the state value function estimation network; the state value function estimation network is trained using mean squared error loss, which minimizes the squared difference between the network's value prediction for the current processing state and the temporal difference target value, and updates the network parameters through gradient descent, iterating until convergence or reaching a preset number of rounds, such as one hundred rounds.

[0112] Then, for each state-action pair, the calculated reward estimate for the current processing state is compared and combined with the sum of the immediate reward value obtained from the empirical data collected for the new processing task, plus a discount factor, and the reward estimate for the subsequent new processing state, to form a basic advantage estimate. The calculation process of the basic advantage estimate is as follows: for a piece of empirical data, the basic advantage estimate equals the immediate reward value of that empirical data, plus a discount factor multiplied by the value estimate of the next state, and then subtracts the value estimate of the current state; where the value estimates of the current state and the next state are both given by the trained state value function estimation network; this basic advantage estimate reflects the advantage brought by a single-step transition.

[0113] Finally, the fundamental advantage estimate at the current time is mixed with the fundamental advantage estimates at multiple consecutive future times, and the fundamental advantage estimates at more distant times are given exponentially decaying weights. A final advantage function estimate is obtained by weighted summation. This method helps to obtain an advantage function estimate with a lower estimation variance. The specific mixing process of the generalized advantage estimate is as follows: a mixing decay parameter between zero and one is set, usually 0.95. For each time step in a trajectory, its generalized advantage estimate is calculated as follows: from the current time step to the end of the trajectory, the fundamental advantage estimate of each future time step is multiplied by the mixing decay parameter raised to the power of (the difference between the future time step and the current time step), and then all these weighted fundamental advantage estimates are summed.

[0114] This means that the advantage estimate at the current moment incorporates advantage information from multiple future steps, but the weight of information from more distant moments decays exponentially. This hybrid approach effectively balances the bias and variance of the estimate, resulting in a more stable and reliable estimate of the underlying advantage function. During calculation, empirical data needs to be processed in chronological order. If the empirical data is not stored in the form of a complete trajectory, trajectory reconstruction is required based on state continuity information.

[0115] The fast adaptive update based on gradient descent works as follows:

[0116] First, define the policy probability ratio. This ratio is the ratio of the probability density of actions with the same processing parameters recorded in the empirical data collected for a new processing task, under the given processing state, to the policy defined by the policy parameters to be optimized and the policy defined by the policy defined by the fixed meta-initialized policy parameters. The initial values ​​of the policy parameters to be optimized are set as the meta-initialized policy parameters. The calculation process of the policy probability ratio is as follows: the processing state in the empirical data is input into two policy networks: one is a policy network composed of the current policy parameters to be optimized, and the other is a policy network composed of the fixed meta-initialized policy parameters. The two policy networks have the same structure, namely the main part of the uncertainty-aware policy network (outputting the action mean). For a given processing state and recorded actions, both policy networks will output an action mean vector and an action variance vector (for the meta-initialized policy network, its variance part can be ignored or a fixed small variance can be used). Assuming that the action distribution is a diagonal Gaussian distribution, the probability density of the recorded actions under this distribution is calculated.

[0117] The probability density is calculated as follows: For each action dimension, the probability density of the action value in that dimension is calculated under a Gaussian distribution (mean is the mean of the corresponding dimension output by the policy network, and variance is the variance of the corresponding dimension output by the policy network), and then the probability densities of all dimensions are multiplied together; the policy probability ratio is the quotient of the probability density calculated from the policy parameters to be optimized and the probability density calculated from the meta-initialized policy parameters; at the start of the update, the policy parameters to be optimized are equal to the meta-initialized policy parameters, so the initial policy probability ratio is one;

[0118] Next, a pruning objective function is constructed. The calculation logic of this pruning objective function is as follows: multiply the policy probability ratio by the corresponding advantage function estimate to obtain an unpruned update objective term. The unpruned update objective term reflects the policy update direction brought about by directly scaling the advantage function estimate through the policy probability ratio. If the policy probability ratio is greater than one, it indicates that the new policy is more inclined to choose this action. If the advantage function estimate is positive at this time, this product encourages increasing the probability of this action; if the advantage function estimate is negative, it encourages decreasing the probability of this action.

[0119] Simultaneously, the policy probability ratio is restricted to a closed interval centered at the value 1 and with a preset positive small pruning parameter as its radius, resulting in a pruned probability ratio. This pruned probability ratio is then multiplied by the same dominance function estimate to obtain the pruned update target term. The pruning operation is implemented through a pruning function. This function restricts the input value of the policy probability ratio to the range of (1 minus the pruning parameter) to (1 plus the pruning parameter). For example, if the pruning parameter is set to 0.2, the policy probability ratio is restricted to between 0.8 and 1.2. The pruned probability ratio is then multiplied by the dominance function estimate to obtain the pruned update target term. The pruning parameter is a crucial hyperparameter, typically set between 0.1 and 0.3, such as 0.2. Its function is to define a trust region, preventing the policy probability ratio from deviating too far from 1 in a single update, thereby avoiding drastic policy changes and ensuring learning stability.

[0120] Finally, the smaller of the unpruned update target term and the pruned update target term is taken as the effective contribution of the state action to the pruning objective function. Taking the smaller value ensures that when the policy probability ratio is within the pruning interval, the unpruned target term is used for updating; when the policy probability ratio exceeds the pruning interval, the pruned target term is used. Since the pruned probability ratio is restricted to the boundary value, its product with the advantage function will be smaller than that of the unpruned target term (when the advantage is positive and the probability ratio is greater than the upper bound, or the advantage is negative and the probability ratio is less than the lower bound), thus suppressing excessive updates. This minimization operation is the core of the near-end policy optimization pruning method, which constructs a conservative lower bound for the objective function.

[0121] The effective contributions of all state-action pairs in the empirical data collected for the new processing task are summed to form the overall adaptive update target. The overall adaptive update target is the sum of the effective contributions of all empirical data samples. This is a scalar value that represents the estimated lower bound of the improvement of policy performance relative to the old policy (meta-initialization policy) under the current policy parameters to be optimized. Maximizing this sum means seeking the parameter update direction that is most likely to improve policy performance within the trust region.

[0122] With the goal of maximizing the overall adaptive update, the stochastic gradient ascent optimization algorithm is used. Initializing the policy parameters with meta-initialized values, it performs one or a finite number of iterative updates, each using a small learning rate, such as 0.0001. The specific execution process of the stochastic gradient ascent optimization algorithm is as follows: an adaptive moment estimator optimizer is employed; first, the gradient of the overall adaptive update objective with respect to the policy parameters is calculated; gradient calculation is achieved through automatic differentiation; then, the optimizer estimates the update step size for each parameter based on the first and second moments of the historical gradients; the learning rate is set to a small fixed value, such as one part per hundred thousand.

[0123] Starting with the meta-initialized policy parameters, gradient ascent updates are performed, i.e., the policy parameters move one step along the gradient direction. This process is repeated, but the number of iterations is strictly limited, for example, three to five updates. This design of a small number of iterations is to adapt quickly, avoid overfitting to the limited current empirical data, and at the same time take advantage of the good starting point provided by the meta-initialized policy parameters.

[0124] Once the iteration is complete, the final updated policy parameters are defined as the fine-tuned processing policy parameters. The fine-tuned processing policy parameters are the policy network weights obtained after rapid adaptive updates. They inherit the cross-task general knowledge of the meta-initialization policy and are specifically fine-tuned based on the specific experience data of the current new task, so that they can output better processing parameter actions in the processing environment of the current new task. These parameters will be used to update the core decision-making part of the uncertainty-aware policy network.

[0125] In this embodiment, it is specifically required that the finely tuned processing strategy parameters be updated online using sampled data. This is specifically performed using the elastic weight consolidation algorithm, and the update process is as follows:

[0126] Before online fine-tuning, the importance weights required for the elastic weight consolidation algorithm are first calculated. The calculation process is as follows: Based on the empirical data collected for the new processing task in step S3 for rapid adaptation, the importance weight of each adjustable parameter in the fine-tuned processing strategy parameters is calculated. This importance weight is used to measure the criticality of the parameter in maintaining the performance of the learned processing strategy. Its value is estimated by calculating the diagonal elements of the Fisher information matrix of the parameter on the empirical data. The specific calculation steps of the importance weight are as follows: Load the batch of empirical data used in the rapid adaptation stage of step S3 from the storage. This batch of empirical data contains multiple quadruple records of processing states, processing parameter actions, immediate rewards, and new processing states; input the processing state in each piece of empirical data into the strategy network with parameters of the fine-tuned processing strategy parameters to obtain the action probability distribution of the strategy network corresponding to the processing state.

[0127] Then, the gradient of the policy network output with respect to each adjustable parameter is calculated, that is, the partial derivative of the logarithmic action probability with respect to each parameter is calculated; finally, for all empirical data, the average of the squares of the partial derivatives of each parameter is calculated, and this average is the importance weight estimate of the parameter; the larger the importance weight, the greater the influence of the parameter on the policy output in the rapid adaptation phase, so its value needs to be more strictly protected from being changed drastically in online fine-tuning.

[0128] The finely tuned processing strategy parameters are loaded into the uncertainty-aware strategy network, replacing the original processing strategy parameters of the uncertainty-aware strategy network. This enables the uncertainty-aware strategy network to generate actual processing instructions based on the real-time processing status. The specific loading process is as follows: the finely tuned processing strategy parameters, i.e., a numerical vector containing all weights and biases, are completely overwritten in the corresponding decision part (i.e., the action mean generation network) of the uncertainty-aware strategy network through memory copying or model parameter assignment operations. After loading, the decision-making behavior of the uncertainty-aware strategy network is completely determined by the finely tuned processing strategy parameters.

[0129] During processing, the data is processed according to instructions generated by an uncertainty-aware policy network loaded with finely tuned policy parameters. Real-time processing status and result feedback are continuously collected, forming online experience data which is stored in a pre-built experience replay buffer. The experience replay buffer is a fixed-size first-in-first-out queue, typically set to hold 10,000 experience data points. When new online experience data is generated, it is stored at the tail of the queue in the form of a quadruple. If the queue is full, the old data at the head of the queue is automatically deleted. The collection frequency of online experience data is consistent with the control cycle, for example, once every ten milliseconds. Before storage, the data is serialized and converted into a unified binary or tensor format.

[0130] During online fine-tuning, a small batch of the latest processing experience data is sampled from the experience replay buffer. Small batch sampling refers to randomly and uniformly extracting a certain number of experience data from the experience replay buffer, for example, extracting 256 experience data each time. The sampling process is random to ensure the independent and identically distributed nature of the data, which is beneficial to the stability of gradient updates. The temporal relationship of the data is ignored during sampling, and the buffer is regarded as an unordered set.

[0131] Based on the latest processing experience data obtained from sampling, a policy optimization loss is calculated. This policy optimization loss is used to evaluate and improve the performance of the processing policy parameters of the currently loaded uncertainty-aware policy network on the latest processing experience data. The initial values ​​of the current processing policy parameters are the fine-tuned processing policy parameters. The calculation of the policy optimization loss adopts the near-end policy optimization pruning objective function. The specific calculation process is as follows: First, the action probability distribution of each processing state in the sampled data is calculated using the current processing policy parameters, and the probability ratio with the old policy (i.e., the fine-tuned policy) is calculated. Then, the advantage function estimate of each state-action pair is calculated. Next, the probability ratio is multiplied by the advantage function estimate, and the pruning function is applied to limit the range of change of the probability ratio, finally obtaining the pruned target value. The average of the target values ​​of all sampled data is taken and the negative value is the policy optimization loss. The smaller this loss is, the better the performance of the policy on the latest data.

[0132] Simultaneously, an elastic weight consolidation regularization term is constructed. This term penalizes the deviation between the current processing strategy parameters and the fine-tuned processing strategy parameters, and the penalty strength for each parameter is proportional to its pre-calculated importance weight. The specific calculation process of the elastic weight consolidation regularization term is as follows: the square of the difference between the current value and the fine-tuned value of each adjustable parameter is multiplied by the importance weight of that parameter. Then, the weighted squared differences of all parameters are summed, and then multiplied by half of a preset regularization coefficient. The regularization coefficient is used to control the strength of the regularization term and is usually set to a large number, such as one thousand, to effectively protect important parameters.

[0133] The total loss for online fine-tuning is obtained by adding the policy optimization loss to the regularization term scaled by a preset strength. The total loss is the weighted sum of the policy optimization loss and the elastic weight consolidation regularization term. The preset strength is the regularization coefficient, which is used to balance the learning of new knowledge and the protection of old knowledge. The calculation of the total loss is a scalar addition operation, and the result is a scalar value. In the optimization process, the goal is to find the processing policy parameters that minimize the total loss.

[0134] By minimizing the total loss, gradient descent updates are performed on the current processing strategy parameters. This optimizes the strategy using the latest processing experience data while protecting the knowledge learned during the fine-tuning phase that is crucial to the current processing task, preventing catastrophic forgetting due to online learning. The gradient descent update employs an adaptive moment estimation algorithm. The specific steps are: calculating the gradient of the total loss with respect to the current processing strategy parameters; then using the adaptive moment estimation algorithm to calculate the learning step size for each parameter based on the first and second moments of the historical gradients; finally, updating each parameter according to the calculated step size. Each online fine-tuning typically performs one or a few gradient updates, such as a single update, to avoid overfitting to a small amount of the latest data. The learning rate is set to a very small value, such as one part per hundred thousand.

[0135] The uncertainty generated during the decision-making process of the continuous monitoring of processing strategy parameters is addressed, and a safety alert or strategy rollback is triggered when persistently high uncertainty is detected. The specific process is as follows:

[0136] By utilizing an uncertainty-aware policy network loaded with the current processing strategy parameters, a random deactivation mechanism within the network is enabled during inference. This Monte Carlo random deactivation method performs multiple forward propagation calculations on the same real-time acquired processing state at the same processing decision moment. Monte Carlo random deactivation means that during network inference, the randomly deactivated layer remains active, and a portion of neurons in the network are randomly discarded according to a preset probability during each forward propagation. Typically, the random deactivation rate is set between 0.1 and 0.5, for example, 0.2. Multiple independent forward propagation calculations are performed on the same processing state, for example, twenty forward propagations. Each propagation yields slightly different outputs due to the different random neuron discards. The number of forward propagations is a hyperparameter that needs to be balanced between accuracy and computational cost.

[0137] During each forward propagation, due to the different random deactivated nodes, the uncertainty-aware policy network outputs a slightly different action probability distribution of processing parameters. The output action probability distribution is a multidimensional Gaussian distribution defined by the mean vector and variance vector. With each forward propagation, the network outputs different means and variances, thus obtaining different Gaussian distributions. These distributions represent the network's predictions under different random subnetwork structures. The variance part reflects the network's confidence in its own predictions, while the differences between multiple predictions reflect structural uncertainty.

[0138] The Jensen-Shannon divergence between each pair of action probability distributions obtained from multiple forward propagations is calculated, and the average of the divergence values ​​for all pairwise combinations is taken. This average value is defined as the decision inconsistency measure at the current decision moment. This decision inconsistency measure quantifies the degree of disagreement within the uncertainty-aware policy network regarding the current optimal action, i.e., cognitive uncertainty. The calculation process of Jensen-Shannon divergence is as follows: For any two probability distributions P and Q, first calculate their intermediate distribution M, where M is equal to the arithmetic mean of P and Q; then calculate the Kourbach-Leibler divergence between P and M, and the Kourbach-Leibler divergence between Q and M; the Jensen-Shannon divergence is equal to the average of these two divergences; for multiple distributions, calculate the Jensen-Shannon divergence for all pairwise combinations, and then take the arithmetic mean; the larger the decision inconsistency measure value, the greater the prediction difference between different random subnetworks within the network, i.e., the higher the cognitive uncertainty of the model regarding the current state;

[0139] Maintain a sliding window of fixed time length and calculate the exponentially weighted moving average of the integrated decision inconsistency measure for the most recent series of decision moments within the sliding window. The length of the sliding window is typically set to one hundred decision moments. The exponentially weighted moving average is calculated as follows: the new average equals the smoothing coefficient multiplied by the current decision inconsistency measure, plus one minus the smoothing coefficient multiplied by the old average. The smoothing coefficient is generally between 0.1 and 0.3, for example, 0.2. The resulting moving average can smooth out instantaneous fluctuations and reflect the medium-term trend of uncertainty. The old average is initialized to zero.

[0140] This exponentially weighted moving average is compared to a dynamic safety threshold, which is pre-set based on the distribution characteristics of integrated decision inconsistency measures statistically collected during historical safe operation phases, for example, set as the historical mean plus three standard deviations. The process for setting the dynamic safety threshold is as follows: During the initial safe operation phase of the system, for example, the first thousand decision moments processed using the fine-tuned strategy, the decision inconsistency measures at each moment are collected; the arithmetic mean and standard deviation of these measures are calculated; the dynamic safety threshold is set as the arithmetic mean plus three standard deviations; for example, if the arithmetic mean is 0.5 and the standard deviation is 0.1, then the dynamic safety threshold is 0.8; this dynamic safety threshold can be recalculated and updated periodically based on actual operating conditions.

[0141] If the number of decision moments in which the exponentially weighted moving average exceeds the dynamic safety threshold consecutively reaches a preset alarm trigger count, a state of sustained high uncertainty is determined, and the following safety response is immediately triggered: an alarm message containing the current state and uncertainty value is sent to the operation interface, and the processing strategy parameters currently loaded in the uncertainty-aware strategy network used to generate actual processing instructions are rolled back to the finely tuned processing strategy parameters saved as a safety snapshot; the alarm trigger count is typically set to five to twenty times, for example, ten times; consecutively exceeding the threshold means that the exponentially weighted moving average is higher than the dynamic safety threshold in consecutive decision moments; the safety response includes: a warning window popping up on the operation interface, displaying the current processing state and the current uncertainty value, and automatically replacing the strategy parameters with the previously saved finely tuned processing strategy parameters; the rollback operation is achieved by loading a copy of the finely tuned parameters from non-volatile memory or a secure storage area and overwriting the parameters in current memory; after the rollback, the system can continue to run, but strategy performance may degrade, requiring operator intervention for inspection.

[0142] Example 2

[0143] This embodiment provides, for example Figure 2The system shown is a parameter scheduling system for surface machining of figurine molds based on reinforcement learning. Specifically, it includes: a meta-training and reward inference module, a safety exploration and data collection module, a rapid adaptation module, and an online operation and monitoring module connected sequentially.

[0144] Meta-training and reward inference module: acquires historical processing task datasets and expert demonstration data for new processing tasks; performs meta-training on the reinforcement learning policy network based on the historical processing task datasets and outputs meta-initialized policy parameters; and outputs the reward function for the new processing task using an inverse reinforcement learning algorithm based on the expert demonstration data.

[0145] Safety Exploration and Data Collection Module: Receives initialization policy parameters and reward function; constructs an uncertainty-aware policy network integrating Bayesian neural network; calculates the uncertainty estimate of the action suggested by the uncertainty-aware policy network; adjusts the exploration randomness accordingly; and when the preset safety threshold is exceeded, activates a conservative policy guided by the reward function to output the final action, thereby collecting and outputting experience data for new processing tasks.

[0146] The rapid adaptation module receives experience data and a reward function for new processing tasks; it then uses this experience data and reward function to perform a rapid adaptive update based on gradient descent on the meta-initialized policy parameters, and outputs the fine-tuned processing policy parameters.

[0147] Online operation and monitoring module: Receives fine-tuned processing strategy parameters and loads them into the uncertainty-aware strategy network to generate actual processing instructions; continuously collects real-time processing status and result feedback during processing and stores them in a pre-built experience playback buffer; periodically samples data from the experience playback buffer to fine-tune and update the fine-tuned processing strategy parameters online; continuously monitors the uncertainty generated during strategy decision-making and triggers a safety alarm when high uncertainty is detected.

[0148] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0149] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0150] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0151] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0152] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0153] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0154] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for scheduling machining parameters of a figurine mold surface based on reinforcement learning, characterized in that, Specifically, the following steps are included: Step S1: Obtain historical processing task datasets and expert demonstration data for new processing tasks; perform meta-training on a reinforcement learning policy network for outputting processing parameters based on the historical processing task datasets to obtain meta-initialized policy parameters; based on the expert demonstration data, use an inverse reinforcement learning algorithm to infer the reward function suitable for the new processing task. Step S2: Using the initialization policy parameters and reward function as initial inputs, construct an uncertainty-aware policy network that integrates a Bayesian neural network; while exploring the actual processing environment, calculate in real time the uncertainty estimate of the action suggested by the uncertainty-aware policy network for the current processing state; based on the uncertainty estimate, dynamically adjust the randomness of the action exploration, and when the uncertainty estimate exceeds a preset safety threshold, activate a conservative policy guided by the reward function to replace the uncertainty-aware policy network in outputting the final action, thereby collecting experience data for new processing tasks; Step S3: Using the collected experience data of the new processing task and the reward function, perform a fast adaptive update based on gradient descent on the meta-initialized policy parameters to obtain the fine-tuned processing policy parameters; Step S4: Load the finely tuned processing strategy parameters into the uncertainty-aware strategy network to generate actual processing instructions; continuously collect real-time processing status data and processing result feedback during the processing process and store them in a pre-built experience playback buffer; periodically sample data from the experience playback buffer, use the sampled data to finely adjust and update the finely tuned processing strategy parameters online, and continuously monitor the uncertainty generated by the processing strategy parameters during decision-making, and trigger a safety alarm when continuous high uncertainty is detected.

2. The method for scheduling surface machining parameters of figurine molds based on reinforcement learning according to claim 1, characterized in that: In step S1, the specific process of meta-training a reinforcement learning policy network for outputting processing parameters based on the historical processing task dataset to obtain the meta-initialized policy parameters is as follows: Construct a parameterized reinforcement learning policy network that receives processing states as input and outputs processing parameters and actions. The weights and biases of this reinforcement learning policy network are collectively referred to as policy network parameters. Extract a historical processing task instance from the historical processing task dataset, and divide the processing data contained in the historical processing task instance into a support set and an evaluation set; The loss metric of the current policy network parameters on the extracted historical task is calculated based on the processing data of the support set. Based on this loss metric, the policy network parameters are updated in one step using the gradient descent principle to generate a temporary local policy network parameter that is adapted to the extracted historical task. Subsequently, using the processing data in the evaluation set, the generalization loss of this local policy network parameters on the extracted historical tasks is calculated; the entire process of randomly selecting task instances, splitting the data, calculating the support set loss, updating and generating local parameters, and calculating the evaluation set generalization loss is repeated on multiple different tasks included in the historical processing task dataset. By iterating continuously, the initial policy network parameters are adjusted to minimize the sum of generalization losses accumulated over all historical tasks; when the iteration process converges, the final policy network parameters are defined as the meta-initialized policy parameters.

3. The method for scheduling machining parameters of a figurine mold surface based on reinforcement learning according to claim 2, characterized in that: The specific process of inferring the reward function suitable for the new processing task based on expert demonstration data and using an inverse reinforcement learning algorithm is as follows: The expert demonstration data is modeled as a trajectory sequence composed of states and processing parameters arranged in chronological order; a parameterized reward function model is initialized, and the internal weights of the reward function model are collectively referred to as reward function parameters; The optimization objective of the inverse reinforcement learning algorithm is to adjust the reward function parameters so that the expert demonstration trajectory in the trajectory sequence has the maximum likelihood probability in the probability distribution guided by the current reward function, while introducing the maximum entropy constraint to avoid overfitting to the expert behavior. During the iteration process, based on the current reward function parameters, a temporary policy model for sampling is trained through a policy optimization algorithm, and a series of non-expert trajectories are generated through sampling. Calculate the cumulative reward value of the expert demonstration trajectory under the current reward function, and at the same time calculate the cumulative reward value of the sampled non-expert trajectory under the current reward function; The logic for calculating the gradient estimate of the reward function parameters is as follows: calculate the cumulative reward difference between the expert demonstration trajectory and the non-expert trajectory under the current reward function; calculate the log-likelihood ratio between the expert demonstration trajectory and the non-expert trajectory under the probability distribution induced by the current reward function; multiply the cumulative reward difference by the log-likelihood ratio and calculate the expectation over all possible trajectories to obtain the gradient estimate; use this gradient estimate to update the reward function parameters according to the predetermined learning rate. Repeat the iterative process, which includes sampling, calculating cumulative reward, calculating gradient estimate, and updating parameters, until the reward function parameters make the cumulative reward of the expert demonstration trajectory significantly higher than that of the non-expert trajectory, and the trajectory distribution entropy induced by the reward function remains at a high level. The reward function is determined by the obtained reward function parameters.

4. The method for scheduling machining parameters of a figurine mold surface based on reinforcement learning according to claim 3, characterized in that: In step S2, the specific process of constructing an uncertainty-aware policy network that integrates a Bayesian neural network is as follows: The meta-initialization policy parameters are loaded as the core weights of the uncertainty-aware policy network; in the output layer of the uncertainty-aware policy network, a parameter structure is added to generate a vector representing the discreteness of the action distribution. The uncertainty-aware policy network takes the current processing state acquired in real time as input, and after transformation by multiple fully connected layers and nonlinear activation functions contained within the uncertainty-aware policy network, it outputs the action mean vector and the action log variance vector in parallel. The action log variance vector is then transformed by a nonlinear transformation function that ensures that the output value is always positive to obtain the action variance vector. The action mean vector and the action variance vector together define a multidimensional Gaussian probability distribution. The mean of this multidimensional Gaussian probability distribution is determined by the action mean vector, and its covariance matrix is ​​determined by a diagonal matrix formed by using the action variance vector as its diagonal elements. The uncertainty estimate of the action suggested by the uncertainty-aware policy network for the current processing state is obtained by calculating the sum of all diagonal elements of the covariance matrix.

5. The method for scheduling surface machining parameters of figurine molds based on reinforcement learning according to claim 4, characterized in that: The process of dynamically adjusting actions and activating conservative strategies is as follows: A safety threshold is pre-set; at each decision moment when interacting with the processing environment, the uncertainty estimate for the current processing state is obtained by the uncertainty-aware policy network. The current uncertainty estimate is compared with the safety threshold. If the current uncertainty estimate is less than the safety threshold, the final processing parameters are sampled from the multidimensional Gaussian probability distribution defined by the uncertainty-aware strategy network. If the current uncertainty estimate is greater than the safety threshold, the uncertainty-aware policy network is ignored, and instead an optimization problem is solved: find a processing parameter action such that the output value of the reward function obtained in step S1 under the current processing state and this processing parameter action is as large as possible, while minimizing the squared Euclidean distance between this processing parameter action and a preset nominal safety action. Nominal safety action is a set of predefined, known conservative machining parameter values ​​that can guarantee the basic safety of the machining process. The obtained result is used as the final processing parameter action, i.e. the final action, and applied to the actual processing environment; the new processing state generated by the processing environment after receiving the processing parameter action is obtained; and an immediate reward value is obtained. The quadruple data generated during this decision-making and interaction process, including the processing state at the moment of decision, the final processing parameter action, the immediate reward value obtained, and the new processing state generated, are stored as a piece of experience data in the experience data set pre-built for new processing tasks. This process is repeated cyclically, continuously collecting experience data for new processing tasks.

6. The method for scheduling surface machining parameters of figurine molds based on reinforcement learning according to claim 5, characterized in that: In step S3, the collected empirical data from the new processing task and the reward function are used to perform a fast adaptive update of the meta-initialization policy parameters based on gradient descent. The specific process is as follows: Based on the reward function and the empirical data collected for the new processing task, calculate the advantage function estimate for each processing state and processing parameter action pair in the empirical data collected for the new processing task. The process of calculating the advantage function estimate is as follows: through temporal difference learning, a target value is recursively calculated. This target value is equal to the sum of the immediate reward value plus a discount factor and the estimated reward of the new processing state. This target value is used as the reward estimate of the current processing state. Then, for each state-action pair, the estimated reward of the current processing state calculated above is compared and combined with the sum of the immediate reward value obtained from the empirical data collected for the new processing task plus the discount factor and the estimated reward of the subsequent new processing state to form a basic advantage estimate. Finally, the fundamental advantage estimate at the current moment is mixed with the fundamental advantage estimates at multiple consecutive future moments, and the fundamental advantage estimates at more distant moments are given exponentially decaying weights. A final advantage function estimate is obtained by weighted summation.

7. The method for scheduling surface machining parameters of figurine molds based on reinforcement learning according to claim 6, characterized in that: The specific process of the fast adaptive update based on gradient descent is as follows: First, define the policy probability ratio, which is the ratio of the probability density of the same processing parameter action recorded in the empirical data collected for a new processing task under the given processing state, between the policy defined by the policy parameters to be optimized and the policy defined by the fixed meta-initialized policy parameters; the initial value of the policy parameters to be optimized is set to the meta-initialized policy parameters. Next, a pruning objective function is constructed. The calculation logic of the pruning objective function is as follows: multiply the policy probability ratio by the corresponding advantage function estimate to obtain an unpruned update objective term. Meanwhile, the policy probability ratio is restricted to a closed interval centered on the value 1 and with a preset positive small pruning parameter as the radius, to obtain a pruned probability ratio. The pruned probability ratio is then multiplied by the same advantage function estimate to obtain the pruned updated target term. Finally, the smaller value between the unpruned update target item and the pruned update target item is taken as the effective contribution of this state action to the pruning objective function; The effective contributions of all state-action pairs in the empirical data collected for the new processing task are summed to form the overall adaptive update target; With the goal of maximizing this overall adaptive update, the stochastic gradient ascent algorithm is used to perform a finite number of iterative updates on the processing strategy parameters. The updated parameters are the fine-tuned processing strategy parameters.

8. The method for scheduling surface machining parameters of figurine molds based on reinforcement learning according to claim 7, characterized in that: The online fine-tuning and updating of the refined processing strategy parameters using sampled data is specifically performed using an elastic weight consolidation algorithm. The specific update process is as follows: Before performing online fine-tuning, the importance weights required for the elastic weight consolidation algorithm are first calculated. The calculation process is as follows: based on the empirical data collected in step S3 for rapid adaptation to new processing tasks, the importance weight of each adjustable parameter in the fine-tuned processing strategy parameters is calculated. This importance weight is used to measure the criticality of the parameter in maintaining the performance of the learned processing strategy. Its value is estimated by calculating the diagonal elements of the Fisher information matrix of the parameter on the empirical data. The finely tuned processing strategy parameters are loaded into the uncertainty-aware strategy network, replacing the original processing strategy parameters of the uncertainty-aware strategy network, so that the uncertainty-aware strategy network can generate actual processing instructions based on the real-time processing status. During the processing, real-time processing status and result feedback are continuously collected, and online experience data is stored in a pre-built experience playback buffer. During online fine-tuning, a small batch of the latest processing experience data is sampled from the experience playback buffer; Based on the latest processing experience data obtained from sampling, a strategy optimization loss is calculated, and the initial value of the current processing strategy parameter is the fine-tuned processing strategy parameter. Meanwhile, an elastic weight consolidation regularization term is constructed, which penalizes the deviation between the current processing strategy parameters and the fine-tuned processing strategy parameters, and the penalty strength for each parameter is proportional to its pre-calculated importance weight. The total loss is obtained by adding the policy optimization loss to the scaled regularization term. Gradient descent is then performed to update the parameters of the current processing policy by minimizing the total loss.

9. The method for scheduling surface machining parameters of figurine molds based on reinforcement learning according to claim 8, characterized in that: The strategy of continuously monitoring the processing strategy parameters generates uncertainty during decision-making, and triggers a safety alarm when persistent high uncertainty is detected. The specific process is as follows: By utilizing the uncertainty-aware policy network loaded with the current processing policy parameters, the random deactivation mechanism inside the uncertainty-aware policy network is enabled during the inference process. Multiple forward propagation calculations are performed in the Monte Carlo random deactivation manner to obtain several slightly different action probability distributions. Calculate the Jensen-Shannon divergence between each pair of action probability distributions obtained from multiple forward propagations, and take the average of the divergence values ​​for all pairwise combinations. Define this average as the decision inconsistency measure at the current decision moment. Maintain a sliding window of fixed time length and calculate the exponentially weighted moving average of the integrated decision inconsistency measure for the most recent series of decision moments within the sliding window; This exponentially weighted moving average is compared to a dynamic safety threshold, which is pre-set based on the distribution characteristics of integrated decision inconsistency measures statistically analyzed during historical safe operation phases. If the number of decision moments in which the index-weighted moving average exceeds the dynamic safety threshold consecutively reaches a preset alarm trigger count, it is determined that a state of continuous high uncertainty has been entered, and the following safety response is immediately triggered: an alarm message containing the current status and uncertainty value is sent to the operation interface.

10. A parameter scheduling system for surface machining of figurine molds based on reinforcement learning, applied to the parameter scheduling method for surface machining of figurine molds based on reinforcement learning as described in any one of claims 1-9, characterized in that: Specifically, it includes: The module is connected in sequence to the following modules: meta-training and reward inference module, security exploration and data collection module, rapid adaptation module, and online operation and monitoring module. Meta-training and reward inference module: acquires historical processing task datasets and expert demonstration data for new processing tasks; performs meta-training on the reinforcement learning policy network based on the historical processing task datasets and outputs meta-initialized policy parameters; and outputs the reward function for the new processing task using an inverse reinforcement learning algorithm based on the expert demonstration data. Safety Exploration and Data Collection Module: Receives initialization policy parameters and reward function; constructs an uncertainty-aware policy network integrating Bayesian neural network; calculates the uncertainty estimate of the action suggested by the uncertainty-aware policy network; adjusts the exploration randomness accordingly; and when the preset safety threshold is exceeded, activates a conservative policy guided by the reward function to output the final action, thereby collecting and outputting experience data for new processing tasks. The rapid adaptation module receives experience data and a reward function for new processing tasks; it then uses this experience data and reward function to perform a rapid adaptive update based on gradient descent on the meta-initialized policy parameters, and outputs the fine-tuned processing policy parameters. Online operation and monitoring module: Receives fine-tuned processing strategy parameters and loads them into the uncertainty-aware strategy network to generate actual processing instructions; continuously collects real-time processing status and result feedback during processing and stores them in a pre-built experience playback buffer; periodically samples data from the experience playback buffer and performs online fine-tuning and updates to the fine-tuned processing strategy parameters; Continuously monitor the uncertainty generated during policy decisions and trigger security alerts when persistent high uncertainty is detected.

Images