Abrasive water jet machining parameter real-time reverse optimization method based on reinforcement learning

Abstract

This invention discloses a real-time inverse optimization method for abrasive waterjet machining parameters based on reinforcement learning, relating to the field of abrasive waterjet machining technology. It primarily addresses the technical problem of poor real-time performance in traditional inverse optimization methods. This invention proposes a real-time inverse optimization method for abrasive waterjet machining parameters based on reinforcement learning, modeling the inverse optimization problem as a Markov decision process (MDP). The target surface roughness and preference weights are used as state inputs, and the process parameter combination (P, d, h, v) is used as action outputs. A carefully designed composite reward function guides policy learning. After training, the policy network can output the optimal parameter combination that satisfies the constraints and conforms to the preferences within a single forward propagation (approximately 3.5 ms), achieving true real-time inverse optimization. The time complexity of each query is O(1).

Description

Technical Field

[0001] This invention relates to the field of abrasive waterjet machining technology, specifically a real-time inverse optimization method for abrasive waterjet machining parameters based on reinforcement learning. Background Technology

[0002] Abrasive waterjet machining is a special machining technology that uses high-pressure water mixed with abrasive particles to form a high-speed jet for material removal. This technology has advantages such as no heat-affected zone, the ability to process various materials, and environmental friendliness. It is widely used in aerospace, automobile manufacturing, ammunition dismantling and other fields. In the AWJ machining quality evaluation system, surface roughness Ra is one of the most critical process responses, which directly affects the fit accuracy, fatigue life and service reliability of parts.

[0003] In engineering practice, the optimization of AWJ machining parameters involves a reverse optimization problem. Existing technologies for addressing this problem mainly fall into the following categories:

[0004] The first category is iterative optimization methods based on metaheuristic algorithms, represented by genetic algorithms (GA) and particle swarm optimization (PSO). These methods model inverse optimization as a problem of minimizing the objective function. The disadvantage is that each query requires iterative solution, resulting in high computational overhead and poor real-time performance. Experimental results show that GA takes about 3240 ms to solve a single inverse optimization problem, and PSO takes about 2983 ms, which cannot meet the millisecond-level response requirements of online control.

[0005] The second category: fast response methods based on lookup tables or empirical formulas; pre-calculating and storing a large number of target surface roughness values. - The process parameter combination (P, d, h, v) has a corresponding relationship, but there are two fundamental defects: First, discretized storage leads to a loss of accuracy and makes it difficult to handle continuous target values; second, it cannot flexibly handle multi-target preferences and the lookup results are fixed.

[0006] The third category: end-to-end regression methods based on deep learning; training neural networks to directly learn the surface roughness of the target. The method of inverse mapping to the process parameter combination (P, d, h, v) has a fast inference speed, but it suffers from difficulties in constraint handling: the neural network output is a continuous real number, making it difficult to guarantee strict satisfaction of the upper and lower bound constraints of the parameters. In addition, the one-to-many characteristic of the inverse mapping (the same target surface roughness) The possibility of multiple feasible combinations of process parameters (P, d, h, v) also increases the learning difficulty.

[0007] To address the triple challenges of poor real-time performance, lack of preference awareness, and difficulty in constraint handling, this invention proposes a real-time inverse optimization method for abrasive waterjet machining parameters based on reinforcement learning. This method models the inverse optimization problem as a Markov decision process (MDP), with the target surface roughness as the primary factor. Using preference weights as state inputs and process parameter combinations (P, d, h, v) as action outputs, the policy learning is guided by a carefully designed composite reward function. After training, the policy network can output the optimal parameter combination that satisfies the constraints and conforms to the preferences within a single forward propagation (about 3.5 ms), achieving true real-time inverse optimization. The time complexity of each query is O(1). Summary of the Invention

[0008] The purpose of this invention is to solve the problems mentioned above by proposing a real-time inverse optimization method for abrasive waterjet machining parameters based on reinforcement learning.

[0009] The objective of this invention can be achieved through the following technical solution: a real-time inverse optimization method for abrasive waterjet machining parameters based on reinforcement learning, the method comprising the following steps:

[0010] Step S1: Construct a proxy model as a simulation environment;

[0011] Step S2: Define the MDP state space and actions;

[0012] Step S3: Design a composite reward function;

[0013] Step S4: Construct and train the policy network;

[0014] Step S5: Deploy the policy network for real-time inference.

[0015] Further, step S1 is as follows:

[0016] A machine learning proxy model is used to replace the real processing environment, providing simulation interaction capabilities for reinforcement learning; the proxy model takes the combination of process parameters (P, d, h, v) as input and the predicted surface roughness Ra' as output; the proxy model is specifically constructed by using Gaussian regression, neural network, and support vector regression.

[0017] Furthermore, step S2 is as follows:

[0018] The inverse optimization problem is modeled as a one-step Markov decision process (MDP), which is then transformed into standard mathematical form; the MDP consists of quadruples.<S, A, r,γ> Define S as the state space, A as the action space, r as the reward function, and γ as the discount factor. This invention adopts single-step decision-making, and the discount factor γ is set to 1.

[0019] Furthermore, the state space S (input layer)

[0020] The state space S is defined as a vector containing the target quality index and preference weights:

[0021] ;

[0022] in, The normalized target quality indicators; w1, w2, ..., w k The user-defined preference weights correspond to different optimization objectives such as cost and efficiency, and their values ​​range from [0,1].

[0023] Action Space A (Output Layer)

[0024] The action space A is defined as a normalized parameter vector, used to represent the continuous output of the policy network;

[0025] ;

[0026] The action components output by the network (i.e., any element in A) is mapped to the actual process parameter x;

[0027] Specifically, this involves linearly mapping the values ​​in the interval [-1, 1] to the actual range of values ​​for the process parameter [x]. min ,x max ]; Discretize the mapped continuous values ​​using the Q operation.

[0028] The formula for calculating the physical parameter x is as follows:

[0029] ;

[0030] in, The range of values ​​is shifted and scaled from [−1,1] to [0,1]. This is a quantization function used to output the desired surface roughness. The optimal combination of process parameters (P, d, h, v).

[0031] Furthermore, step S3 is as follows:

[0032] Design a composite reward function consisting of four components to guide the policy network in learning an optimal policy that meets quality requirements, aligns with user preferences, and adheres to parameter constraints; the total reward function is defined as: ;

[0033] Quality penalty item r quality The quality penalty term is used to penalize positive deviations of the predicted surface roughness Ra' from the target surface roughness. In cases where relative error is used, it ensures that different target surface roughnesses are addressed. The consistency of the penalty scale is calculated using the following formula:

[0034] ;

[0035] Among them, ypred The surface roughness Ra' is predicted by the surrogate model; For target surface roughness λ1 is the quality weighting coefficient;

[0036] Constraint penalty item r constraint The constraint penalty term is used to ensure that the predicted surface roughness Ra' does not exceed the target surface roughness. When the predicted surface roughness Ra' exceeds the target surface roughness When the margin is sufficient, a second penalty is applied, calculated using the following formula:

[0037] ;

[0038] Where λ2 is the constraint penalty coefficient; δ is the allowable tolerance ratio;

[0039] Preference guide r preference Based on the user-specified preference weights, the guiding strategy network outputs a combination of process parameters that conforms to specific preferences while meeting quality requirements. The calculation formula is as follows:

[0040] ;

[0041] Where λ3 is the preference penalty coefficient; For the j-th preference objective with respect to the normalized parameters The function;

[0042] Constraint satisfaction term r bonus When all constraints are met, a fixed reward is given to the agent, calculated using the following formula:

[0043] ;

[0044] The value of c should be determined based on the gradual parameter adjustment experiments in actual engineering.

[0045] Furthermore, step S4 is as follows:

[0046] Policy Network: A fully connected neural network is adopted. The input layer (module 401) receives a state vector with a dimension of (1+K). Here, 1 corresponds to the target quality index and k corresponds to the number of preference weights. After processing by several hidden layers, the output layer outputs an action vector with a dimension of n, where n is the number of parameters to be optimized. The hidden layer uses the ReLU activation function to enhance the nonlinear expression capability. The output layer uses the tanh activation function to restrict the output to the range of [-1,1], ensuring that the parameter boundary constraints are met from the network level.

[0047] Value Network: The input layer receives the concatenation of the state vector and the action vector, with a dimension of (1+k+n). After processing through several hidden layers, the output layer outputs a scalar Q-value, which is used to evaluate the expected cumulative reward of the state-action pair and guide the optimization of the policy network. The hidden layers of the value network also use ReLU-type activation functions, while the output layer uses linear activation.

[0048] Furthermore, step S5 is as follows:

[0049] Input reception: Receive target quality indicators and preference weights {w j The target quality indicators are input by the upper control system or user interface; the preference weights can be preset in multiple modes, which can be selected by the user or automatically switched according to the working conditions.

[0050] Status coding and normalization: The target quality index is normalized, and its calculation formula is as follows: ;

[0051] Among them, y min and y max Define the training data range boundary for the quality metric; concatenate the normalized target value with the preference weights to construct the state vector. ;

[0052] Policy grid forward propagation: Input the state vector into the trained policy network, perform a single forward propagation, and obtain the normalized action vector. ;

[0053] Inverse normalization and quantization: The normalization process is mapped to actual process parameters, and the calculation formula is as follows: ;

[0054] Among them, the quantization function Map continuous values ​​to discrete values ​​allowed by the engineering process;

[0055] Output the optimal combination of process parameters: This involves quantifying the combination of process parameters. Output to the lower-level control system or display interface.

[0056] Compared with the prior art, the beneficial effects of the present invention are:

[0057] 1. For a given target surface roughness in abrasive waterjet (AWJ) machining. This paper proposes a real-time solution method based on deep reinforcement learning for the inverse optimization problem of finding the optimal combination of process parameters (P, d, h, v). The method models the inverse optimization problem as a Markov decision process (MDP) with the target surface roughness as the objective. The state space is formed by the preference weights, and the action space is formed by the combination of process parameters (P, d, h, v). The policy network is trained by reinforcement learning algorithm.

[0058] In this embodiment, compared with the traditional GA method requiring 3240 ms and the PSO method requiring 2983 ms for iterative solution time, the present invention only requires a single forward propagation of 3.50 ms to output the optimization result, improving the inference speed by about 900 times, and effectively solving the core defect of poor real-time performance of traditional methods.

[0059] 2. To achieve the flexibility of a single model supporting multiple optimization preferences, this invention innovatively introduces cost preference weights wcost and efficiency preference weights wspeed into the MDP state space. This enables the policy network to output differentiated combinations of process parameters based on preference information in the state. Compared with traditional methods that require modification of the fitness function and rerun of optimization, this invention can achieve instant switching by adjusting preference weights during the inference stage without retraining the model, significantly improving the flexibility of engineering applications.

[0060] 3. To ensure the feasibility of the output parameters, a dual constraint processing mechanism based on network structure constraints and reward function penalties is proposed. At the network structure level, the Actor network output layer adopts the tanh activation function to limit the output to the range of [-1,1]. After inverse normalization and quantization, the parameter boundary constraints will be automatically satisfied. At the reward function level, a quadratic constraint penalty term is designed to penalize predictions that exceed the tolerance margin. The dual mechanism ensures a constraint satisfaction rate of 98%.

[0061] 4. By designing a four-component composite reward function, an effective balance of multi-objective optimization is achieved. Experimental results show that while ensuring millisecond-level response speed and preference perception capability, the prediction error of this invention is 1.36%, which fully meets the accuracy requirements of industrial applications. Attached Figure Description

[0062] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings.

[0063] Figure 1 This is a flowchart of the overall process of the present invention;

[0064] Figure 2 This is a structural diagram of the composite reward function in this invention;

[0065] Figure 3 This is a diagram of the MDP interaction framework in this invention;

[0066] Figure 4 This is a diagram of the strategy network structure (Actor-Critic architecture) in this invention.

[0067] Figure 5 This is a flowchart for real-time inference. Detailed Implementation

[0068] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0069] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0070] Example 1: As Figure 1 As shown, the real-time inverse optimization method for abrasive waterjet machining parameters based on reinforcement learning provided by this invention has the following core idea: The inverse parameter optimization problem is modeled as a Markov decision process (MDP), and a policy network is trained using a deep reinforcement learning algorithm, enabling it to optimize the target surface roughness based on the parameters of the target surface. Based on user preferences, the system directly outputs the optimal combination of process parameters (P, d, h, v).

[0071] Figure 1 The complete technical process is shown, including step S1 agent model construction (module 101), step S2 MDP state and action space definition (module 102), step S3 composite reward function design (module 103), step S4 policy network construction and training (module 104), and step S5 real-time inference deployment (module 105).

[0072] The overall methodology framework comprises five core steps:

[0073] Step S1: Construct a proxy model as a simulation environment;

[0074] Step S2: Define the MDP state space and actions;

[0075] Step S3: Design a composite reward function;

[0076] Step S4: Construct and train the policy network;

[0077] Step S5: Deploy the policy network for real-time inference.

[0078] Steps S1 to S4 belong to the offline training stage, and step S5 belongs to the online inference stage.

[0079] The data flow of this method can be summarized as (e.g.) Figure 3 As shown): The machine learning agent model provides a simulation environment → MDP state encoding of goals and preferences → policy network output parameters of actions → composite reward function to evaluate action quality → policy network parameter update → deployment and inference after convergence.

[0080] Figure 3 In the middle, the state coding module (304) will determine the target surface roughness. The preference weight w is encoded as a state s and input into the policy network (301); the policy network outputs action a to the agent model (302) to obtain the predicted value ypred; the reward parameter (303) calculates the reward r based on the predicted value and feeds it back to the policy network for parameter update. This framework transforms the traditional iterative optimization for each query into training and inference in one go, fundamentally solving the real-time problem.

[0081] Step S1: Proxy Model Construction

[0082] Since the actual abrasive waterjet processing process is costly and time-consuming to conduct experiments, it cannot support the large amount of experimental data required for reinforcement learning. Therefore, a machine learning surrogate model (explanation: a surrogate model is a method of simulating a real physical process using a mathematical model, which can quickly predict the processing results without conducting actual experiments) is first used to replace the real processing environment, providing simulation interaction capabilities for reinforcement learning. The surrogate model takes the combination of process parameters (P, d, h, v) as input and the predicted surface roughness Ra' as output.

[0083] The surrogate model can be constructed using methods such as Gaussian Regression (GPR), Neural Network (ANN), and Support Vector Regression (SVM).

[0084] Step S2: Define the state space and action space (MDP modeling)

[0085] like Figure 1 Module 102 and Figure 3 As shown, Figure 3 Markov Decision Process (MDP) interaction framework diagram, Figure 3 The interaction relationships between the policy network (module 301), the agent model / environment (module 302), the reward function (module 303), and the state encoding (module 304) are shown, including the data flow of state s, action a, predicted value ypred, and reward r, as well as the target roughness value of the external input. And preference weight w;

[0086] This invention models the inverse optimization problem as a one-step Markov decision process (MDP). To enable computers to understand and solve the optimization problem, it needs to be transformed into a standard mathematical form. An MDP consists of quadruples.<S, A,r,γ> Define S as the state space, A as the action space, r as the reward function, and γ as the discount factor. Since this invention uses single-step decision-making, the discount factor γ is set to 1.

[0087] State space S (input layer)

[0088] The state space S is defined as a vector containing the target quality index and preference weights:

[0089] ;

[0090] Among them, y ∗ The normalized target quality indicators; w1, w2, ..., w k The user-defined preference weights correspond to different optimization objectives such as cost and efficiency, and their values ​​range from [0,1].

[0091] Incorporating preference weights into the state space is a key design feature of this invention. This allows the same policy network to generate differentiated combinations of process parameters for the same target based on different preference inputs. In this embodiment, this method can perceive different cutting preferences, enabling the same trained policy to achieve the same target surface roughness according to different priorities. Generate different optimal combinations of process parameters (P, d, h, v).

[0092] (2) Action Space A (Output Layer)

[0093] The action space A is defined as a normalized parameter vector, used to represent the continuous output of the policy network.

[0094]

[0095] Since the policy network outputs normalized values, it cannot be directly output as parameters. Therefore, it is necessary to map the action components ai (i.e., any element in A) output by the network to the actual process parameters x.

[0096] The process involves two steps:

[0097] ① Linear normalization: This linearly maps the values ​​in the interval [-1, 1] to the actual range of values ​​for the process parameter [x...]. min ,x max ].

[0098] ② Quantization processing: In order to meet the minimum scale requirement in engineering implementation (for example, the minimum pressure regulation accuracy of a pump in engineering is only 0.5 MPa), it is necessary to discretize the mapped continuous value by Q.

[0099] Therefore, the formula for calculating the physical parameter x is as follows:

[0100] ;

[0101] in, The range of values ​​is shifted and scaled from [−1,1] to [0,1]. This is a quantization function used to output the desired surface roughness. The optimal combination of process parameters (P, d, h, v).

[0102] Step S3: Design the composite reward function r

[0103] like Figure 1 Module 103 and Figure 2 As shown, Figure 2 After demonstrating the calculation of four sub-items—quality penalty rquality (module 201), constraint penalty rconstraint (module 202), preference guidance rpreference (module 203), and constraint satisfaction rbonus (module 204)—the total reward r is output after summing and summing.

[0104] This invention designs a composite reward function consisting of four components to guide the policy network in learning an optimal policy that meets quality requirements, aligns with user preferences, and adheres to parameter constraints. The total reward function is defined as follows: ;

[0105] (1) Quality penalty item r quality

[0106] The quality penalty term is used to penalize positive deviations of the predicted surface roughness Ra' from the target surface roughness. In cases where relative error is used, it ensures that different target surface roughnesses are addressed. The consistency of the penalty scale is calculated using the following formula:

[0107] ;

[0108] Among them, y pred The surface roughness Ra' is predicted by the surrogate model; For target surface roughness λ1 is the quality weighting coefficient.

[0109] (2) Constraint term rconstraint

[0110] The constraint penalty term is used to ensure that the predicted surface roughness Ra' does not exceed the target surface roughness. Too much. When the predicted surface roughness Ra' exceeds the target surface roughness. When the margin is set (the margin should be adjusted according to the actual engineering needs and set within the allowable tolerance range of the actual engineering), a secondary penalty is applied to ensure that a larger penalty is imposed when the predicted value exceeds the limit too much. The calculation formula is:

[0111] ;

[0112] Where λ2 is the constraint penalty coefficient; δ is the allowable tolerance ratio (in machining, the tolerance for machining roughness is generally allowed not to exceed its upper limit).

[0113] (3) Preference guiding term r preference

[0114] The preference-guided term is used to prioritize parameters during the cutting process. Based on the user-specified preference weights, this guiding strategy network tends to output process parameter combinations that conform to specific preferences, while meeting quality requirements. The calculation formula is as follows:

[0115] ;

[0116] Where λ3 is the preference penalty coefficient; For the j-th preference objective with respect to the normalized parameters The function.

[0117] (4) Constraint satisfaction term r bonus

[0118] The constraint satisfaction reward term provides additional positive reinforcement to feasible solutions. A fixed reward is given to the agent when all constraints are satisfied, calculated as follows:

[0119] ;

[0120] The value of c should be determined based on the gradual parameter adjustment experiments in actual engineering.

[0121] Step S4: Policy Network Construction and Training;

[0122] like Figure 4 As shown, this invention uses an Actor-Critic architecture to construct a policy network. The left side of the figure shows the structure of the Actor network (policy network), including an input layer (module 401), a hidden layer (module 402), and an output layer (module 403); the right side shows the structure of the Critic network (value network), including an input layer (module 404), a hidden layer (module 405), and an output layer (module 406).

[0123] Policy Network (Actor) Figure 4 Left side): A fully connected neural network is used. The input layer (module 401) receives a state vector with a dimension of (1+K), where 1 corresponds to the target quality index and k corresponds to the number of preference weights. After being processed by several hidden layers (module 402), the output layer (module 403) outputs an action vector with a dimension of n, where n is the number of parameters to be optimized. The hidden layers use the ReLU activation function to enhance the nonlinear expression capability, and the output layer uses the tanh activation function to limit the output to the range of [-1,1], ensuring that the parameter boundary constraints are met from the network level.

[0124] Value Network (Critic) Figure 4 Right side): The input layer (module 404) receives the concatenation of the state vector and the action vector, with a dimension of (1+k+n). After being processed by several hidden layers (module 405), the output layer (module 406) outputs a scalar Q-value (representing the expected reward of taking the action in the current state), which is used to evaluate the expected cumulative reward of the state-action pair and guide the optimization of the policy network. The hidden layers of the value network also use ReLU-type activation functions, and the output layer uses linear activation.

[0125] Step S5: Real-time Inference Deployment

[0126] like Figure 5 As shown, after training is complete, only the policy network (Actor network) is deployed for inference, while the value network is only used during the training phase. Figure 5 This demonstrates the complete reasoning process from inputting target quality indicators (module 501) and preference weights (module 502), through state encoding and normalization (module 503), policy network forward propagation (module 504), denormalization and quantization (module 505), to finally outputting the optimal combination of process parameters (module 506).

[0127] The reasoning process is as follows:

[0128] (1) Input reception: Receiving target quality indicators (In this embodiment, the target surface roughness) () Figure 5 Module 501) and preference weights {w j}( Figure 5 (Module 502) The target quality indicators are input by the upper control system or user interface. The preference weight can be preset in multiple modes (such as cost priority, efficiency priority, and balance mode), which can be selected by the user or automatically switched according to the working conditions.

[0129] (2) State coding and normalization Figure 5Module 503): The target quality indicators are normalized using the following formula: ;

[0130] Among them, y min and y max This represents the training data range boundary for the quality metric. The normalized target value is concatenated with the preference weights to construct the state vector. .

[0131] (3) Policy grid forward propagation ( Figure 5 Module 504): Input the state vector into the trained policy network, perform a single forward propagation, and obtain the normalized action vector. ;

[0132] (4) Inverse normalization and quantization processing ( Figure 5 Module 505): Maps normalized actions to actual process parameters; the calculation formula is: ;

[0133] Among them, the quantization function Map continuous values ​​to discrete scale values ​​allowed by the project, specifically by rounding to the nearest scale value.

[0134] (5) Output the optimal combination of process parameters ( Figure 5 (506): Combining the quantified process parameters Output to the lower-level control system or display interface.

[0135] The method involves only a few matrix multiplications and activation function operations in a single inference step, with a time complexity of O(n log n). (Regardless of the problem size being optimized), it can achieve millisecond-level response, meeting real-time control requirements.

[0136] Example 2: The following uses the reverse optimization of surface roughness of carbon structural steel Q235 by abrasive waterjet machining as an example to illustrate the specific implementation of the present invention.

[0137] 2.1 Proxy Model

[0138] A Gaussian process regression (GPR) model is used, with the process parameter combination (P, d, h, v) as input parameters, to predict the surface roughness Ra'. The model's coefficient of determination R0 is [not specified]. 2 =0.9726, RMSE=0.0193.

[0139] 2.2 State Space and Action Space

[0140] state space .in, For target surface roughness , Cost preference weight, Efficiency preference weights.

[0141] The ranges of the four process parameters corresponding to the motion space are shown in Table 1.

[0142]

[0143] 2.3 Reward Function Parameters

[0144] The quality penalty coefficient λ1=15, the constraint penalty coefficient λ2=80, the margin δ=2%, the preference reward coefficient λ3=2, and the constraint satisfaction reward c=0.5.

[0145] The specific form of the preference reward item is as follows: ;

[0146] 2.4 Network Structure and Training Parameters

[0147]

[0148] Training parameters: This embodiment uses the TD3 (Twin Delayed DDPG) algorithm (an improved deep reinforcement learning algorithm, which is used for policy optimization in continuous action space), with a learning rate of 3×10-4, a batch size of 256, and 50,000 training rounds.

[0149] 2.5 Preference Mode Configuration

[0150]

[0151] 2.6 Performance Test Results

[0152]

[0153] This invention improves the inference speed of this embodiment by approximately 900 times, meeting real-time control requirements. Due to the continuous output characteristics and quantization error of the neural network, there are approximately 2% of boundary cases that slightly exceed the constraints, but all are within the engineering tolerance range. In addition, due to its preference-aware function, it is more suitable for engineering requirements (for a given target surface roughness). It can output three sets of process parameter combinations (P, d, h, v) that satisfy preference perception.

[0154] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to any specific implementation. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A real-time inverse optimization method for abrasive waterjet machining parameters based on reinforcement learning, characterized in that, The method includes the following steps: Step S1: Construct a proxy model as a simulation environment; Step S2: Define the MDP state space and actions; Step S3: Design a composite reward function; Step S4: Construct and train the policy network; Step S5: Deploy the policy network for real-time inference.

2. The method for real-time inverse optimization of abrasive waterjet machining parameters based on reinforcement learning according to claim 1, characterized in that, Step S1 is as follows: A machine learning proxy model is used to replace the real processing environment, providing simulation interaction capabilities for reinforcement learning; the proxy model takes the combination of process parameters (P, d, h, v) as input and the predicted surface roughness Ra' as output; The surrogate model is constructed using Gaussian regression, neural networks, and support vector regression.

3. The real-time inverse optimization method for abrasive waterjet machining parameters based on reinforcement learning according to claim 1, characterized in that, Step S2 is as follows: The inverse optimization problem is modeled as a one-step Markov decision process (MDP), which is then transformed into standard mathematical form; the MDP consists of quadruples.<S, A, r,γ> Define S as the state space, A as the action space, r as the reward function, and γ as the discount factor. This invention adopts single-step decision-making, and the discount factor γ is set to 1.

4. The real-time inverse optimization method for abrasive waterjet machining parameters based on reinforcement learning according to claim 3, characterized in that, The state space S (input layer) is defined as a vector containing the target quality metric and preference weights: ； in, The normalized target quality indicators; w1, w2, ..., w k The user-defined preference weights correspond to different optimization objectives of cost and efficiency, and their values ​​range from [0,1]. The action space A (output layer) is defined as a normalized parameter vector that represents the continuous output of the policy network. ； The action components output by the network (i.e., any element in A) is mapped to the actual process parameter x; Specifically, this involves linearly mapping the values ​​in the interval [-1, 1] to the actual range of values ​​for the process parameter [x]. min ,x max ]; Discretize the mapped continuous values ​​using the Q operation; The formula for calculating the physical parameter x is as follows: ； in, The range of values ​​is shifted and scaled from [−1,1] to [0,1]. This is a quantization function used to output the desired surface roughness. The optimal combination of process parameters (P, d, h, v).

5. The real-time inverse optimization method for abrasive waterjet machining parameters based on reinforcement learning according to claim 1, characterized in that, Step S3 is as follows: Design a composite reward function consisting of four components to guide the policy network in learning an optimal policy that meets quality requirements, aligns with user preferences, and adheres to parameter constraints. The total reward function is defined as follows: ; Quality penalty item r quality The quality penalty term is used to penalize the predicted surface roughness. Positive deviation from target surface roughness In scenarios where relative error is used, it ensures that different target surface roughnesses are addressed. The consistency of the penalty scale is calculated using the following formula: ； Among them, y pred The surface roughness Ra' is predicted by the surrogate model; For target surface roughness λ1 is the quality weighting coefficient; Constraint penalty item r constraint The constraint penalty term is used to ensure that the predicted surface roughness Ra' does not exceed the target surface roughness. When the predicted surface roughness Ra' exceeds the target surface roughness When the margin is sufficient, a second penalty is applied, calculated using the following formula: ； Where λ2 is the constraint penalty coefficient; δ is the allowable tolerance ratio; Preference guide r preference Based on the user-specified preference weights, the guiding strategy network outputs a combination of process parameters that conforms to specific preferences while meeting quality requirements. The calculation formula is as follows: ； Where λ3 is the preference penalty coefficient; For the j-th preference objective with respect to the normalized parameters The function; Constraint satisfaction term r bonus When all constraints are met, a fixed reward is given to the agent, calculated using the following formula: ； The value of c should be determined based on the gradual parameter adjustment experiments in actual engineering.

6. The real-time inverse optimization method for abrasive waterjet machining parameters based on reinforcement learning according to claim 1, characterized in that, Step S4 is as follows: Policy Network: A fully connected neural network is used. The input layer receives a state vector with dimension (1+K), where 1 corresponds to the target quality index and k corresponds to the number of preference weights. After processing by several hidden layers, the output layer outputs an action vector with dimension n, where n is the number of parameters to be optimized. The hidden layers use the ReLU activation function to enhance nonlinear expressiveness. The output layer uses the tanh activation function to restrict the output to the range [-1,1], ensuring that the parameter boundary constraints are met at the network level. Value Network: The input layer receives the concatenation of the state vector and the action vector, with a dimension of (1+k+n). After processing through several hidden layers, the output layer outputs a scalar Q-value, which is used to evaluate the expected cumulative reward of the state-action pair and guide the optimization of the policy network. The hidden layers of the value network also use ReLU-type activation functions, while the output layer uses linear activation.

7. The real-time inverse optimization method for abrasive waterjet machining parameters based on reinforcement learning according to claim 1, characterized in that, Step S5 is as follows: Input reception: Receive target quality indicators and preference weights {w j The target quality indicators are input by the upper control system or user interface; the preference weights can be preset in multiple modes, which can be selected by the user or automatically switched according to the working conditions. Status coding and normalization: The target quality index is normalized using the following formula: ; Among them, y min and y max Define the training data range boundary for the quality metric; concatenate the normalized target value with the preference weights to construct the state vector. ; Policy grid forward propagation: Input the state vector into the trained policy network, perform a single forward propagation, and obtain the normalized action vector. ; Inverse normalization and quantization: The normalization process is mapped to actual process parameters, and the calculation formula is as follows: ; Among them, the quantization function Map continuous values ​​to discrete values ​​allowed by the engineering process; Output the optimal combination of process parameters: This involves quantifying the combination of process parameters. Output to the lower-level control system or display interface.

Images