Milling processing path automatic generation method based on reinforcement learning

Abstract

The invention discloses a milling path automatic generation method based on reinforcement learning, and relates to the technical field of machine tool machining, and the milling path automatic generation method comprises the following steps: step 1, defining action output; step 2, defining state input including a cutter state, a global observation space and a local observation space; 3, defining a reward function; value feedback given to the intelligent agent by the environment according to the current state and action is determined through the reward function; 4, setting deep neural networks including a feature extractor, a strategy network and a value network; 5, constructing a process autonomous generation model and a machining process world model; step 6, strategy training; step 7, outputting a strategy, and autonomously completing a machining process flow; by means of the method, the problems that in the prior art, manual experience dependence is too heavy, local optimization limitation is remarkable, and physical simulation efficiency is insufficient are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of machine tool processing, and in particular to a method for automatically generating a milling machining path based on reinforcement learning. Background Art

[0002] Currently, the programming of the machining path of numerically controlled machine tools highly depends on CAM software (such as Vericut, UG, etc.). Such systems use the geometric inversion method to reversely generate the tool path, and calculate the material removal trajectory by analyzing the geometric features of the surface of the target workpiece. Although they can quickly generate a non-interfering path and ensure geometric integrity, it is difficult to integrate key physical constraints such as the dynamic fluctuation of cutting force, tool wear, and the acceleration / jerk threshold of the machine tool servo system. This single geometric perspective planning method leads to the lack of prediction of process stability during the machining process. During actual cutting, surface quality defects or even equipment damage are often caused due to insufficient chatter suppression or overloading. To avoid risks, engineers have to rely on experience to repeatedly adjust parameters such as feed speed and cutting depth, and gradually approach a feasible solution through multiple iterative optimization processes of "process planning - trial cutting verification - performance evaluation". This not only significantly prolongs the process preparation cycle, but also causes unnecessary losses of high-value materials and man-hour resources. The finally formed process plan often stays at the level of local sub-optimal solutions and is difficult to achieve global parameter collaborative optimization. Summary of the Invention

[0003] The purpose of the present invention is to provide a method for automatically generating a milling machining path based on reinforcement learning, so as to solve the problems of excessive dependence on manual experience, significant limitations in local optimization, and insufficient physical simulation efficiency existing in the prior art.

[0004] To achieve the above purpose, the present invention provides a method for automatically generating a milling machining path based on reinforcement learning, including the following steps:

[0005] Step 1, define the action output; determine the machining path and parameters of each interaction between the agent and the environment through the action output;

[0006] Step 2, define the state input; determine the workpiece and tool information provided by the environment to the agent through the state input. The state input includes three items of information, namely: tool state, global observation space, and local observation space;

[0007] Step 3, define the reward function; determine the value feedback given by the environment to the agent according to the current state and action through the reward function;

[0008] Step 4, set up a deep neural network; set up three deep neural networks, namely: the first deep neural network is a feature extractor, the second deep neural network is a policy network π φ , and the third deep neural network is a value network Q θ ;

[0009] Step 5: Construct a process autonomous generation model and a machining process world model; The process autonomous generation model is based on the soft actor-critic algorithm and includes: action output, state input, feature extractor, policy network, and value network; At the same time, in order to train the process autonomous generation model through reinforcement learning, it is necessary to construct a machining process world model to digitally model the workpiece and the tool, which can perform geometric-physical co-simulation on the machining process and provide a stable environment for the training of reinforcement learning;

[0010] Step 6: Policy training; The process autonomous generation model continuously tries and makes mistakes in the machining process world model, accumulates experience, and stores the experience in the experience replay buffer. Each piece of experience will be used as a training set to train the feature extractor that recognizes the state input and the value network that predicts the reward expectation; The policy network is trained by finding the action with the highest reward expectation in the value network.

[0011] Step 7: Policy output; The policy network directly interacts with the machining environment and gives a reasonable cutting strategy according to the state at each moment, and autonomously completes the machining process flow.

[0012] Preferably, the specific process of Step 1 is as follows:

[0013] Take the tool as the object controlled by reinforcement learning. During the machining process, the state of the tool is determined by the position of the tool tip point (Tool Center Point, denoted as tcp), the tool axis direction (Tool Axis Direction, denoted as tad), the spindle speed (Spindle Speed, denoted as RPM), and the feed rate (FeedRate, denoted as F);

[0014] The action A at the analysis step at time t t is defined as:

[0015]

[0016] In the formula, represents the change in the position of the tool center point, represents the change in the tool axis direction, ΔRPM represents the change in the spindle speed, and ΔF t represents the change in the feed rate; The expressions for the above four parameters are:

[0017]

[0018] ΔRPM t = RPM t - SRPM t-1 ;

[0019] ΔF t = Ft -F t-1 ;

[0020] Wherein, is the position of the tool tip at time t, is the position of the tool tip at time t-1, is the tool axis direction at time t, is the tool axis direction at time t-1, RPM t is the spindle speed at time t, RPM t-1 is the spindle speed at time t-1, F t is the feed rate at time t, F t-1 is the feed rate at time t-1.

[0021] Preferably, the specific process of step 2 is as follows:

[0022] S21. Determine the tool state; assume that the tool state is a one-dimensional floating-point vector of length 9, including the position tcp of the current tool tip, the current tool axis direction tad, and the previous action action. The lengths of the above three parameters are all 3;

[0023] S22. Determine the global observation space; the global observation space uses a three-dimensional floating-point matrix to represent the workpiece, providing overall trend information for the action decision of the algorithm; the global observation space contains two three-floating-point-dimensional matrices, both centered on the current tool tip, and sampling the current workpiece and the target workpiece respectively; the sizes of the two matrices are both 10×10×10; the value range of each voxel is [0.0, 1.0], which is used to represent the material ratio within the range of the voxel;

[0024] S23. Determine the local observation space; the local observation space uses a two-dimensional floating-point matrix to represent the workpiece adjacent to the intelligent body, providing information for the algorithm to decide the next specific action; in order to minimize the loss of information during the sampling process, the local observation space contains 18 two-dimensional floating-point matrices, sampling the three objects of the current workpiece, the target workpiece, and the tool shape from six directions: the positive x-axis direction, the negative x-axis direction, the positive y-axis direction, the negative y-axis direction, the positive z-axis direction, and the negative z-axis direction.

[0025] Preferably, the specific process of step 3 is as follows:

[0026] S31. In order to enable the process of autonomous interaction and learning to consider physical constraints such as cutting force and tool wear while ensuring the integrity of material removal, as well as the dynamic adaptation of the acceleration / jerk limit of the machine tool servo system, construct a "geometry-physics-control" trinity reward mechanism, and the reward mechanism R total The expression is as follows:

[0027] R total = C f*(R cut +R surface )-R control -R step ;

[0028] In the formula, R cut and R surface are geometric rewards and also the main rewards to ensure the integrity of material removal; R cut represents the cutting material volume reward; R surface represents the area removal reward; C f is the physical penalty factor; R control represents the control reward; R step represents the path length penalty term;

[0029] Among them, the definition expression of R cut is:

[0030] R cut =λ cut (V valid -V overcut );

[0031] In the formula, V valid is the effective removal volume cut off in this step, that is, the non-overcut volume; V overcut represents the overcut volume in this step; λ cut is the proportionality factor used to control the proportion of the cutting volume reward in the total reward;

[0032] The definition expression of R surface is:

[0033] R surface =-λ surface (A cur -A pre );

[0034] In the formula, A cur is the remaining workpiece surface area after performing the action of the current analysis step, A pre is the remaining workpiece surface area before performing the action of the current analysis step, λ surface is the proportionality factor;

[0035] The definition expression of C f is:

[0036] C f =C ap ×min(C max ,C mean );

[0037] In the formula, C ap represents the cutting depth factor, C max is the maximum cutting force factor, Cmean is the average cutting force factor; C ap , C max and C mean are both in the range of [0, 1]. Therefore, the value range of C f is also in the range of [0, 1]; The cutting depth factor C ap gradually increases with the increase of the cutting depth ap, showing an exponential relationship, C ap = e 3(ap-1) ; The maximum cutting force factor and the average cutting force factor show a piecewise function relationship with the change of the cutting force. When the cutting force f is less than the non-destructive threshold f1, the cutting force factor C always remains 1. When the cutting force f is between the non-destructive threshold f1 and the zeroing threshold f2, C decreases linearly; When the cutting force f is greater than the zeroing threshold f2, C is always 0; The expression is as follows:

[0038]

[0039] For actual machine tool processing, the acceleration and jerk that the servo system can provide are limited. Therefore, it is required that the path reaches G02 continuity, that is, the speed and acceleration change continuously. Just through geometric rewards, the algorithm cannot be guided in this regard. So, the control reward R control is introduced. The expression of the control reward R control is as follows:

[0040] R control = R v_control + R a_control ;

[0041] In the formula, R v_control is the speed control reward, and R a_control is the acceleration control reward, which control the speed within the G02 range and the acceleration within the G01 range respectively. The expressions of R v_control and R a_control are respectively:

[0042]

[0043] In the formula, λ v_control and λ a_control represent the speed control reward factor and the acceleration control reward factor respectively; represents the action of the current step, represents the action of the previous step, represents the action of the step before that;

[0044] R step encourages a shorter machining path. Its defined expression is:

[0045]

[0046] In the formula, λ step is the scaling factor, which is used to control the proportion of R step in the total reward.

[0047] Through the above "geometry-physics-control" trinity reward mechanism, the process generation model can be guided to generate the optimal process route with the highest efficiency under the premise of ensuring material removal, meeting physical constraints, and meeting control constraints.

[0048] Preferably, the specific content of step 4 is as follows:

[0049] The feature extractor analyzes the three-modal inputs of the tool state, global vision, and local vision in the state input S t and outputs a feature vector

[0050] The policy network formulates a tool action output plan according to the extracted feature vector;

[0051] The value network predicts the current action value reward according to the feature vector and the corresponding action output.

[0052] Preferably, in the process autonomous generation model constructed in step 5, the policy network interacts with the machining process world model, enabling the machining process world model to execute the tool action plan output by the policy network, and the machining process world model outputs the updated state value and reward value to the policy network.

[0053] Preferably, the specific process of step 6 is as follows:

[0054] S61. The process autonomous generation model and the machining process world model interact autonomously. Each experience generated by the interaction includes the state input S t , the action output A t , the obtained reward R t and the next state input S t+1 . All the experiences generated by the interaction are saved in the experience replay buffer;

[0055] S62. The experience replay buffer is used as the training set to train the value network and the feature extractor. On the one hand, it enables the feature extractor to fully analyze all the effective features in the three-modal input, and on the other hand, it enables the value network to learn all the value expectations under all input-output combinations;

[0056] S63. Guided by the value expectations learned by the value network, the policy network searches for the action value that meets the maximum threshold of the value expectation, and gradually obtains the optimal process path with the most global rewards through autonomous exploration.

[0057] Preferably, the specific process of step 7 is as follows:

[0058] Policy network π φ Interacts directly with the machining process world model, and extracts the state input S corresponding to each moment t in the machining process from the machining process world model t And inputs it into the feature extractor, which extracts the feature vector Transfers it to the policy network, and the policy network outputs the action A t , driving the machine tool to complete the machining task; the action A t The expression of is:

[0059]

[0060] Therefore, the present invention adopts the above-mentioned method for automatically generating a milling machining path based on reinforcement learning. By constructing a geometric-physical co-simulation environment and a reinforcement learning autonomous decision-making architecture, it breaks through the traditional one-way progressive mode: uses digital twin technology to achieve accurate prediction of multi-physical fields such as cutting force and tool wear, designs a multi-dimensional reward mechanism to guide the intelligent agent to synchronously optimize path continuity, cutting stability and equipment dynamics performance, forms a globally optimal process strategy for positive exploration, and significantly improves the one-time forming rate and quality consistency of complex part machining.

[0061] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Description of the Drawings

[0062] Figure 1 It is a framework diagram of the process autonomous generation model and training and planning process adopted by the present invention;

[0063] Figure 2 It is a schematic diagram of the action A at the analysis step at time t of the present invention t Schematic diagrams of each parameter;

[0064] Figure 3 It is a schematic diagram of the tool state in the state input of the present invention;

[0065] Figure 4 It is a schematic diagram of the global observation space in the state input of the present invention; among them, a is a schematic diagram of the global observation space for sampling the current workpiece, and b is a schematic diagram of the global observation space for sampling the target workpiece;

[0066] Figure 5 It is a schematic diagram of the local observation space in the state input of the present invention;

[0067] Figure 6 It is a schematic diagram of the cutting factor of the present invention; among them, a is a schematic diagram of the cutting depth factor changing with the cutting depth, b is a schematic diagram of the maximum cutting factor changing with the cutting force, and c is a schematic diagram of the average cutting factor changing with the cutting force. Detailed Embodiments

[0068] The following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the scope of protection of the present invention.

[0069] A method for automatically generating a milling machining path based on reinforcement learning includes the following steps:

[0070] Step 1, define the action output; determine the machining path and parameters for each step of the interaction between the agent and the environment through the action output.

[0071] Take the tool as the object of reinforcement learning control. During the machining process, the state of the tool is determined by the position of the tool tip point (Tool Center Point, denoted as tcp), the tool axis direction (Tool Axis Direction, denoted as tad), the spindle speed (Spindle Speed, denoted as RPM), and the feed rate (Feed Rate, denoted as F);

[0072] The action A at the analysis step at time t t is defined as:

[0073]

[0074] Please refer to Figure 2 , where represents the change in the position of the tool center point, represents the change in the tool axis direction, ΔS t represents the change in the spindle speed, ΔF t represents the change in the feed rate; the expressions for the above four parameters are:

[0075]

[0076] ΔRPM t = RPM t - RPM t-1 ;

[0077] ΔF t = F t - F t-1 ;

[0078] where is the position of the tool tip point at time t, is the position of the tool tip point at time t - 1, is the tool axis direction at time t, is the tool axis direction at time t - 1, RPM t is the spindle speed at time t, RPMt-1 is the spindle speed at time t-1, F t is the feed rate at time t, F t-1 is the feed rate at time t-1.

[0079] Step 2: Define the state input; determine the workpiece and tool information provided by the environment to the agent through the state input. The state input includes three pieces of information, namely: tool state, global observation space, and local observation space.

[0080] S21: Determine the tool state; assume that the tool state is a one-dimensional floating-point vector of length 9, including the position tcp of the current tool tip point, the current tool axis direction tad, and the previous action action. Please refer to Figure 3 , and the lengths of the above three parameters are all 3;

[0081] S22: Determine the global observation space; the global observation space uses a three-dimensional floating-point matrix to represent the workpiece, providing overall trend information for the action decision of the algorithm; please refer to Figure 4 , the global observation space is a two-channel three-dimensional floating-point matrix. a samples the current workpiece, and b samples the target workpiece, both with the current tool tip point as the sampling center; the sizes of the two matrices are both 10×10×10; the value range of each voxel is [0.0, 1.0], which is used to represent the material proportion within the range of the voxel;

[0082] S23: Determine the local observation space; the local observation space uses a two-dimensional floating-point matrix to represent the workpiece adjacent to the agent, providing information for the algorithm to decide the next specific action; in order to minimize the loss of information during the sampling process, the local observation space contains 18 two-dimensional floating-point matrices. Please refer to Figure 5 , and sample the three objects of the current workpiece, target workpiece, and tool shape from six directions: the positive x-axis direction, the negative x-axis direction, the positive y-axis direction, the negative y-axis direction, the positive z-axis direction, and the negative z-axis direction.

[0083] Step 3: Define the reward function; determine the value feedback given by the environment to the agent according to the current state and action through the reward function.

[0084] S31: In order to enable the process of autonomous interaction and learning to consider physical constraints such as cutting force and tool wear while ensuring the integrity of material removal, as well as the dynamic adaptation of the acceleration / jerk limits of the machine tool servo system, construct a "geometry-physics-control" three-in-one reward mechanism, and the reward mechanism R total The expression is as follows:

[0085] R total = C f *(R cut + R surface ) - Rcontrol -R step ;

[0086] In the formula, R cut and R surface are geometric rewards and also the main rewards to ensure the integrity of material removal; R cut represents the volume reward of the cutting material; R surface represents the area removal reward; C f is the physical penalty factor; R control represents the control reward; R step represents the path length penalty term;

[0087] Among them, R cut is defined by the expression:

[0088] R cut = λ cut (V valid - V overcut );

[0089] In the formula, V valid is the effective removal volume cut in this step, that is, the non-overcut volume; V overcut represents the overcut volume in this step; λ cut is the proportionality factor used to control the proportion of the cutting volume reward in the total reward;

[0090] The definition expression of R surface is:

[0091] R surface = -λ surface (A cur - A pre );

[0092] In the formula, A cur is the remaining workpiece surface area after performing the action of the current analysis step, A pre is the remaining workpiece surface area before performing the action of the current analysis step, λ surface is the proportionality factor;

[0093] The definition expression of C f is:

[0094] C f = C ap × min(C max , C mean );

[0095] In the formula, C ap represents the cutting depth factor, C max is the maximum cutting force factor, C mean is the average cutting force factor; C ap , Cmax and C mean are both in the range of [0, 1]. Therefore, C f also has a value range within [0, 1]. Please refer to Figure 6 , the cutting depth factor C ap gradually increases with the increase of the cutting depth ap, showing an exponential relationship. C ap = e 3(ap-1) ; The maximum cutting force factor and the average cutting force factor show a piecewise function relationship with the change of the cutting force. When the cutting force f is less than the non-destructive threshold f1, the cutting force factor C always remains 1. When the cutting force f is between the non-destructive threshold f1 and the zeroing threshold f2, C decreases linearly. When the cutting force f is greater than the zeroing threshold f2, C is always 0. The expression is as follows:

[0096]

[0097] For actual machine tool processing, the acceleration and jerk that the servo system can provide are limited. Therefore, it is required that the path reaches G02 continuity, that is, the speed and acceleration change continuously. Just through geometric rewards, the algorithm cannot be guided in this regard. So, the control reward R control is introduced. The control reward R control has the following expression:

[0098] R control = R v_control + R a_control ;

[0099] In the formula, R v_control is the speed control reward, and R a_control is the acceleration control reward, which respectively control the speed to meet G02 continuity and the acceleration to meet G01 continuity. The expressions of R v_control and R a_control are respectively:

[0100]

[0101]

[0102] In the formula, λ v_control and λ a_control respectively represent the speed control reward factor and the acceleration control reward factor; represents the action of the current step, represents the action of the previous step, represents the action of the step before that;

[0103] R step has the following defined expression:

[0104]

[0105] In the formula, λ step is the scaling factor used to control the proportion of R step in the total reward..

[0106] Through the above "geometry - physics - control" trinity reward mechanism, the process generation model can be guided to generate the optimal process route with the highest efficiency under the premise of ensuring material removal, meeting physical constraints, and meeting control constraints.

[0107] Step 4: Set up the deep neural network; Set up three deep neural networks, namely: The first deep neural network is the feature extractor, which analyzes the three - modality inputs of the tool state, global vision, and local vision in the state input S t and outputs a feature vector The second deep neural network is the policy network π φ , and the policy network formulates a tool action output plan according to the extracted feature vector; The third deep neural network is the value network Q θ , and the value network predicts the current action value reward according to the feature vector and the corresponding action output.

[0108] Step 5: Construct the process autonomous generation model; Please refer to Figure 1 , the process autonomous generation model is based on the soft actor - critic algorithm and includes: action output, state input, feature extractor, policy network, and value network. The policy network in the constructed process autonomous generation model interacts with the processing - process world model, enabling the processing - process world model to execute the tool action plan output by the policy network. The processing - process world model outputs the updated state value and reward value to the policy network. The pseudo - code in the soft actor - critic algorithm is as follows: The pseudo - code in the soft actor - critic algorithm is as follows:[[]]

[0109]

[0110] Among them, θ1 and θ2 represent the parameters of the action - value network Q θ ; and represent the parameters of the target action - value network , whose role is to stabilize the training of the action - value function; φ represents the parameters of the policy network π; represents the experience replay buffer, which is used to store the accumulated experience of exploration; α represents the adaptive temperature term; τ represents the update rate of the target action - value function, which is used to regulate the exploration in the training process; Each time of training, several pieces of experience are taken out from the experience replay buffer to update the action - value network Q θ , the policy network π φ , the adaptive temperature term α, and the target action - value network

[0111] Step 6, Policy Training; The process autonomous generation model continuously tries and makes mistakes in the processing process world model, accumulates experience, stores the experience in the experience replay buffer, and each piece of experience will be used as a training set to train the feature extractor for recognizing the state input and the value network for predicting the reward expectation; The policy network searches for the action with the highest reward expectation in the value network through training.

[0112] S61. The process autonomous generation model and the processing process world model interact autonomously. Each piece of experience generated by the interaction includes the state input S t , the action output A t , the obtained reward R t and the next state input S t+1 . Save all the experiences generated by the interaction into the experience replay buffer;

[0113] S62. Use the experience replay buffer as a training set to train the value network and the feature extractor. On the one hand, enable the feature extractor to fully analyze all the effective features in the three-modal input, and on the other hand, enable the value network to learn all the value expectations under all input-output combinations;

[0114] S63. Guided by the value expectations learned by the value network, the policy network searches for the action value that meets the maximum threshold of the value expectation, and gradually obtains the optimal process path with the most global rewards through autonomous exploration.

[0115] Step 7, Policy Output; The policy network directly interacts with the processing environment, gives a reasonable cutting strategy according to the state at each moment, and autonomously completes the processing technological process.

[0116] The policy network π φ directly interacts with the processing process world model, extracts the state input S corresponding to each moment t in the processing process from the processing process world model t and inputs it into the feature extractor. The feature extractor extracts the feature vector and passes it to the policy network. The policy network outputs the action A t , driving the machine tool to complete the processing task;

[0117] The expression of the action A t is:

[0118]

[0119] Therefore, the present invention adopts the above-mentioned automatic generation method for milling machining paths based on reinforcement learning. By constructing a geometric-physical co-simulation environment and a reinforcement learning autonomous decision-making architecture, it breaks through the traditional one-way progressive mode: uses digital twin technology to achieve accurate prediction of multi-physical fields such as cutting force and tool wear, designs a multi-dimensional reward mechanism to guide the intelligent agent to synchronously optimize path continuity, cutting stability and equipment dynamic performance, forms a globally optimal process strategy for positive exploration, and significantly improves the one-time forming rate and quality consistency of complex part machining.

[0120] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that they can still modify or equivalently replace the technical solutions of the present invention, and these modifications or equivalent replacements cannot make the modified technical solutions deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. An automatic generation method for milling machining paths based on reinforcement learning, characterized in that, It includes the following steps: Step 1: Define action output; determine the processing path and parameters of each interaction between the agent and the environment through the action output. Step 2: Define state input; determine the workpiece and tool information provided by the environment to the agent through the state input. The state input includes three items of information, namely: tool state, global observation space, and local observation space. Step 3: Define the reward function; determine the value feedback given by the environment to the agent according to the current state and action through the reward function. Step 4: Set up deep neural networks; Set up three deep neural networks, namely: The first deep neural network is a feature extractor, the second deep neural network is a policy network π φ , and the third deep neural network is a value network Q θ ; Step 5: Build a process autonomous generation model and a machining process world model; the process autonomous generation model is based on the soft actor-critic algorithm and includes: action output, state input, feature extractor, policy network, and value network; at the same time, in order to train the process autonomous generation model through reinforcement learning, build a machining process world model to digitally model the workpiece and tool, and conduct geometric-physical co-simulation of the machining process. Step 6: Policy training; the process autonomous generation model tries out in the machining process world model, stores the experience in the experience replay buffer, and each piece of experience will be used as a training set to train the feature extractor for identifying state input and the value network for predicting reward expectations; the policy network searches for the action with the highest reward expectation in the value network through training. Step 7: Policy output; the policy network directly interacts with the machining environment, gives the cutting strategy according to the state at each moment, and autonomously completes the machining process flow.

2. The automatic generation method of milling machining path based on reinforcement learning according to claim 1, characterized in that The specific process of Step 1 is as follows: Take the tool as the object controlled by reinforcement learning. During the machining process, the state of the tool is determined by the position of the tool tip point tcp, the tool axis direction tad, the spindle speed RPM, and the feed rate F. Action A of the analysis step at time t t is defined as: In the formula, represents the change in the position of the tool center point, represents the change in the tool axis direction, and ΔRPM t represents the change in the spindle speed, and ΔF t represents the change in the feed rate; ΔRPM t and ΔF t are respectively expressed as: ΔRPM t = RPM t - RPM t-1 ; ΔF t = F t - F t-1 ; In the formula, is the position of the tool tip at time t, is the position of the tool tip at time t - 1, is the tool axis direction at time t, is the tool axis direction at time t - 1, RPM t is the spindle speed at time t, RPM t-1 is the spindle speed at time t - 1, F t is the feed rate at time t, F t-1 is the feed rate at time t - 1.

3. The automatic generation method of a milling machining path based on reinforcement learning according to claim 2, characterized in that, The specific process of Step 2 is as follows: S21: Determine the tool state; assume that the tool state is a one-dimensional floating-point vector of length 9, including the current position of the tool tip point tcp, the current tool axis direction tad, and the previous action action. The lengths of the current tool tip position, the current tool axis direction, and the previous action are all 3. S22: Determine the global observation space; The global observation space uses a three-dimensional floating-point matrix to represent the workpiece; the global observation space contains two three-floating-point-dimensional matrices, both centered on the current tool tip point, and samples the current workpiece and the target workpiece respectively; the sizes of the two three-floating-point-dimensional matrices are both 10×10×10; the value range of each voxel is [0.0, 1.0]. S23: Determine the local observation space; the local observation space uses a two-dimensional floating-point matrix to represent the workpiece adjacent to the agent; the local observation space contains 18 two-dimensional floating-point matrices, and samples the three objects of the current workpiece, the target workpiece, and the tool shape from six directions: the positive x-axis direction, the negative x-axis direction, the positive y-axis direction, the negative y-axis direction, the positive z-axis direction, and the negative z-axis direction respectively.

4. A method for automatically generating a milling machining path based on reinforcement learning according to claim 3, characterized in that, The specific process of Step 3 is as follows: S31. Construct a "geometry-physics-control" trinity reward mechanism, the reward mechanism R total The expression is as follows: R total = C f *(R cut + R surface ) - R control - R step ; In the formula, R cut represents the cutting material volume reward; R surface represents the area removal reward; C f is the physical penalty factor; R control represents the control reward; R step represents the path length penalty term; Among them, R cut The defined expression is: R cut = λ cut (V valid - V overcut ); Where, V valid is the effective removal volume cut off; V overcut represents the overcut volume; λ cut is the proportionality factor; R surface The defining expression is: R surface = -λ surface (A cur - A pre ); Where, A cur is the remaining surface area of the workpiece after performing the actions of the current analysis step, A pre is the remaining surface area of the workpiece before performing the actions of the current analysis step, and λ surface is the scaling factor used to control the proportion of R surface in the total reward; C f The defining expression is: C f = C ap × min(C max , C mean ); where C ap represents the cutting depth factor, C max is the maximum cutting force factor, and C mean is the average cutting force factor; C ap , C max and C mean are all in the range of [0, 1], so the value range of C f is also in the range of [0, 1]; the cutting depth factor C ap gradually increases with the increase of the cutting depth ap, showing an exponential relationship, C ap = e 3(ap-1) ; the maximum cutting force factor and the average cutting force factor show a piecewise function relationship with the change of the cutting force magnitude. When the cutting force f is less than the non-destructive threshold f1, the cutting force factor C always remains 1. When the cutting force f is between the non-destructive threshold f1 and the zeroing threshold f2, C linearly decreases. When the cutting force f is greater than the zeroing threshold f2, C is always 0. The expression is as follows: Control Reward R control The expression is as follows: R control = R v_control + R a_control ; where R v_control is the speed control reward, and R a_control is the acceleration control reward, which respectively control the speed to satisfy G02 continuity and the acceleration to satisfy G01 continuity. The expressions of R v_control and R a_control are respectively: where λ v_control and λ a_control represent the speed control reward factor and the acceleration control reward factor respectively; denotes the action at the current step, denotes the action at the previous step, represents the action at the step before that; R step The defining expression is: where λ step is a scaling factor used to control the proportion of R step in the total reward.

5. The automatic generation method of a milling machining path based on reinforcement learning according to claim 4, characterized in that The specific content of Step 4 is as follows: The feature extractor analyzes the status input S t among the three-modal inputs of the tool state, global vision, and local vision, and outputs a feature vector The policy network formulates a tool action output plan according to the extracted feature vector, interacts with the machining process world model, enables the machining process world model to execute the tool action plan output by the policy network, and the machining process world model outputs the updated state value and reward value to the policy network. The value network predicts the current action-value reward based on the feature vector and the corresponding action output.

6. The automatic generation method of milling machining path based on reinforcement learning according to claim 5, characterized in that The specific process of Step 6 is as follows: S61. The process autonomous generation model and the machining process world model interact autonomously. Each piece of experience generated by the interaction includes the state input S t , the action output A t , the obtained reward R t and the next state input S t+1 . All the experiences generated by the interaction are saved in the experience replay buffer; S62. Use the experience replay buffer as the training set to train the value network and the feature extractor. On the one hand, enable the feature extractor to fully parse all valid features in the three-modal input. On the other hand, enable the value network to learn all value expectations under all input-output combinations; S63. Guided by the value expectations learned by the value network, the policy network searches for action values that meet the value expectation threshold, and through autonomous exploration, obtains the process path with the most global rewards.

7. A method for automatically generating a milling machining path based on reinforcement learning according to claim 6, characterized in that The specific process of step 7 is as follows: Policy network π φ Directly interact with the machining process world model, and extract the state input S corresponding to each moment t in the machining process from the machining process world model t And input it into the feature extractor, which extracts the feature vector Transmit it to the policy network, and the policy network outputs the action A t , driving the machine tool to complete the machining task; Action A t The expression of

Images