A strategy-free milling tool path generation method and system

Abstract

The present application belongs to the technical field of numerical control machining and artificial intelligence, and discloses a strategy-free milling tool path generation method and system, which comprises the following steps: obtaining a grid model of a part to be machined to determine a machining area, converting a tool path generation problem into a grid center point access sequence problem, and converting it into a reinforcement learning task; establishing an MDP model of the reinforcement learning task, wherein the agent in the MDP model is a tool, the environment is the state of the grid center point, the state space of the tool and the corresponding action space are defined, and the immediate reward function of the tool action is set; solving the MDP model to generate the tool path of the part to be machined. The present application effectively combines the reinforcement learning method with the tool path generation, converts the tool path generation problem into a reinforcement learning task, does not need to give a tool path generation strategy, can autonomously generate a tool path, thereby reducing the dependence on manpower and improving the machining efficiency.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of numerical control machining and artificial intelligence, and more particularly to a strategy-free milling tool path generation method and system. BACKGROUND

[0002] A numerical control machine tool is a mechanical device capable of high-precision machining of complex parts, which is widely used in important production and manufacturing fields such as aerospace, mold manufacturing, and automobile manufacturing. Tool path planning of a numerical control machine tool is an important part of the numerical control machining process, which directly affects the quality and efficiency of numerical control machining, and even affects the service life of the numerical control machine tool.

[0003] A point cloud model three-axis equal residual height numerical control machining tool path generation method is disclosed in Chinese patent specification CN 110308702 B. This method calculates the point cloud contour point set on the feed direction plane and directly calculates the residual height point on the tool contour. This method can directly calculate the three-axis equal residual height numerical control machining tool path for the point cloud model. While meeting the residual height requirement, the row distance is large and the total length of the tool path is small. A triangular mesh model equal residual height tool contact point trajectory generation method is disclosed in Chinese patent specification CN 108594764 B. This method divides the tool contact points in the tool contact point trajectory into convex tool contact points, concave tool contact points, and smooth tool contact points. Based on the anisotropic bias in the tool contact point tangent plane, the actual bias point is calculated, finally making the machining residual height between tool paths closer to the maximum allowable value, and improving the numerical control machining efficiency. A free-form surface three-axis numerical control machining equal residual height tool path generation method is disclosed in Chinese patent specification CN 116974237 A. This method does not need to equidistantly process the surface, and can directly and efficiently calculate the free-form surface equal residual height tool path.

[0004] Currently, the main tool path generation method is to generate G code to control the movement of the tool by using computer aided manufacturing software (CAM) for tool path planning, and then transmitting the G code to the numerical control machine tool for numerical control machining. In the mainstream CAM software such as NX and MasterCAM, many traditional tool path generation methods are integrated, among which the more common methods include the iso-planar method, the iso-parametric method, and the iso-cusp height method. The above three tool path generation methods need to input a large number of parameters and options, including cutting parameters, geometric constraints, etc. The input of these machining parameters directly affects the machining quality and efficiency, hinders beginners, reduces machining efficiency and quality, and even reduces the service life of the numerical control machine tool or the tool. At present, the setting of these parameters is seriously dependent on the experience of CAM engineers. The current research on tool path generation automation mainly focuses on traditional methods. The selection of strategy parameters in the tool path generation process has a high dependence on the experience of CAM engineers, which cannot effectively improve the intelligent degree, and a more advanced and adaptive tool path generation method is needed to make up for the defects of traditional methods in machining quality, machining efficiency and intelligence. SUMMARY

[0005] In view of the above defects or improvement needs of the prior art, the present application provides a strategy-free milling machining tool path generation method and system, which solves the problem of reducing the dependence of strategy parameter selection in the tool path generation process on the experience of CAM engineers. The method can autonomously generate tool paths, without the need for engineers to input strategy parameters, thereby reducing the dependence on engineer experience, improving intelligence and machining efficiency.

[0006] To achieve the above-mentioned purpose, according to one aspect of the present application, a strategy-free milling machining tool path generation method is provided, comprising:

[0007] S1, obtaining a grid model of a part to be machined and determining a machining area on the grid model, taking the center point of the grid as a tool path point, converting the tool path generation problem of the machining area into a tool path point access sequence problem, and converting it into a reinforcement learning task;

[0008] S2, a Markov decision process model (MDP model) of the reinforcement learning task is established, wherein the agent in the MDP model is the tool, the environment is the state of the grid center point in the grid model, the state space of the tool in the environment and the corresponding action space are defined, the immediate reward function of the tool action is set, and after the tool performs an action according to the action space corresponding to the state space, the environment and the state space change and an immediate reward value corresponding to the immediate reward function can be obtained;

[0009] S3, the MDP model is solved based on the immediate reward value of the tool action, and the tool path of the part to be processed is generated.

[0010] According to another aspect of the present application, a strategy-free milling tool path generation system is provided, comprising:

[0011] A grid establishment module is configured to obtain a grid model of a part to be processed and determine a to-be-processed area on the grid model, take a grid center point as a tool path point, convert a tool path generation problem of the to-be-processed area into a tool path point access sequence problem, and thus convert it into a reinforcement learning task.

[0012] A model establishment module is configured to establish a Markov decision process model (MDP model) of the reinforcement learning task, wherein the agent in the MDP model is the tool, the environment is the state of the grid center point in the grid model, the state space of the tool in the environment and the corresponding action space are defined, the immediate reward function of the tool action is set, and after the tool performs an action according to the action space corresponding to the state space, the environment and the state space change and an immediate reward value corresponding to the immediate reward function can be obtained.

[0013] A model solving module is configured to solve the MDP model based on the immediate reward value of the tool action, and generate the tool path of the part to be processed.

[0014] Overall, compared with the prior art, the strategy-free milling tool path generation method and system provided by the present application have the following advantages:

[0015] 1. The method effectively combines the reinforcement learning method with tool path generation, converts the tool path generation problem into solving a reinforcement learning task, does not need to give a tool path generation strategy, can autonomously generate a tool path, thereby reducing the dependence on manual work, improves the intelligent degree while ensuring the processing quality, and since there is no tool path generation strategy at the geometric level, the tool does not need to be limited by the given strategy when generating the path, can select a reasonable motion direction according to the current milling workpiece state and the tool motion state, so that the generated tool path has less redundant path and higher processing efficiency, and a tool path with higher processing efficiency can be generated.

[0016] 2. The MDP model deeply considers the influencing factors of machining efficiency in the machining process, and comprehensively sets the reward function, which can guide the decision of the agent to a certain extent;

[0017] 3. The tool path is calculated by using the multi-layer value simulation algorithm, so that the value of a state can be quickly and accurately calculated, thereby guiding the agent to complete the reinforcement learning task. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 It is the flow chart of the strategy-free milling machining tool path generation method provided by the application;

[0019] Figure 2 It is the overall flow chart of the application;

[0020] Figure 3 It is the grid conversion process schematic diagram in the implementation process of the application;

[0021] Figure 4 It is the corresponding relationship diagram of environment and feature matrix M in the implementation process of the application, (a) initial state of environment; (b) the tool makes a decision and moves to a new state, and the feature matrix is updated; (c) the termination state of the environment;

[0022] Figure 5 It is the state space and action space of the agent in the implementation process of the application, (a) agent state; (b) agent action space; (c) special state processing;

[0023] Figure 6 It is the relationship diagram of the time cost of the feed direction change and the feed speed in the implementation process of the application;

[0024] Figure 7 It is the relationship diagram of the time cost of the feed direction change and the angle in the implementation process of the application;

[0025] Figure 8 It is the relationship diagram of the time cost of the feed direction change and the feed speed and angle in the implementation process of the application;

[0026] Figure 9 It is the reward function schematic diagram of the feed direction change in the implementation process of the application;

[0027] Figure 10 It is the reward function schematic diagram of the lifting tool action in the implementation process of the application;

[0028] Figure 11 It is the reward function schematic diagram of the tool movement in the implementation process of the application;

[0029] Figure 12A final instant reward function schematic diagram in the implementation process of the present application;

[0030] Figure 13 A schematic diagram of selecting actions according to the value of actions in the implementation process of the present application;

[0031] Figure 14 A schematic diagram of simulating and calculating the value of the substate according to the strategy of determining the action according to the instant reward value in the implementation process of the present application;

[0032] Figure 15 A schematic diagram of simulating and obtaining the value of the first layer substate according to the value of the second layer substate in the implementation process of the present application;

[0033] Figure 16 A schematic diagram of the influence of different layer numbers m on the running time and total reward of the multi-layer value simulation algorithm in the implementation process of the present application;

[0034] Figure 17 A simulation case diagram in the implementation process of the present application;

[0035] Figure 18 A simulation experiment result diagram in the implementation process of the present application. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical scheme and advantages of the present application clearer and more apparent, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0037] Please refer to Figure 1 The present embodiment one provides a strategy-free milling tool path generation method, which comprises:

[0038] S1, obtaining a grid model of a part to be machined and determining a machining area on the grid model, taking a grid center point as a tool path point, converting a tool path generation problem of the machining area into a tool path point access sequence problem, and converting it into a reinforcement learning task;

[0039] S2, a Markov decision process model (MDP model) of the reinforcement learning task is established, wherein the agent in the MDP model is the tool, the environment is the state of the grid center point in the grid model, the state space of the tool in the environment and the corresponding action space are defined, and the immediate reward function of the tool action is set; after the tool performs an action according to the action space corresponding to the state space, the environment and the state space change and an immediate reward value corresponding to the action can be obtained according to the immediate reward function;

[0040] S3, the MDP model is solved based on the immediate reward value of the tool action, and the tool path of the part to be processed is generated.

[0041] The embodiment simplifies the part to be processed into a grid model, converts the tool path generation problem into a node access sequence problem on the grid model, and the node access sequence problem is a typical Markov decision process (MDP), so the tool path generation problem can be converted into a reinforcement learning task. The MDP model of the tool path generation can be established according to the process requirements of the numerical control milling process, specifically including setting the agent and the environment of the model, defining the state space and the action space, and setting the immediate reward function according to the process requirements of the milling process. The state space reflects the current state of the agent, and the action space reflects the current feasible action of the agent; after the agent performs an action, the state space, the action space and the environment are updated and changed respectively, and the immediate reward value of the action can be obtained through the immediate reward function.

[0042] The progress of the agent traversing the path points can be judged through the change of the environment, and when the agent traverses all the path points, the interaction between the agent and the environment ends, and the traversed path process is the generated tool path. The tool path can be judged according to the immediate reward value to obtain a better path. That is, the MDP model is solved according to the immediate reward value, and the tool path of the part to be processed is generated.

[0043] The technical problem to be solved by the embodiment is to reduce the dependence of the selection of policy parameters in the tool path generation process on the experience of CAM engineers, and to provide a strategy-free milling tool path generation method. The method can autonomously generate a tool path, and does not need to give a tool path generation strategy, so engineers do not need to be familiar with the characteristics and applicable conditions of various different tool path generation strategies, the technical and experience requirements of engineers are lower, and the tool path generation process can be simplified to a certain extent; at the same time, since the tool path generation process is no longer constrained by the given tool path generation strategy, the generated tool path has higher processing efficiency.

[0044] Figure 2The overall flowchart of the method of the embodiment is shown. The embodiment first divides a three-dimensional part into several layers by layered milling, and discretizes each layer into a square grid plane to obtain a grid model of the part, and converts the tool path generation problem into a reinforcement learning task; then establishes an MDP model of the reinforcement learning task according to the process requirements of the numerical control milling process, establishes the relationship between the time cost of the feed direction change and the angle and feed speed of the feed direction change through the machining experimental data, and designs the instant reward function accordingly; finally, the multi-layer value simulation algorithm is used to solve the reinforcement learning task, and the decision sequence of the agent is obtained, so as to obtain the tool path of the part.

[0045] The entire implementation process is as follows:

[0046] 1. Divide a three-dimensional part into several layers by layered milling, and discretize each layer into a square grid plane to obtain a grid model of the part. The center point of the square grid is used to construct the discrete tool path points of milling processing. Using this simplified model, the tool path can be defined as the order in which the milling tool visits the discrete tool path points on the grid plane, thereby converting the tool path generation problem into a node visit order problem on the grid plane.

[0047] The process of grid conversion is shown in Figure 3 For a given milling task of a part to be processed, first, the processing area and the non-processing area need to be distinguished, and the part contour is determined according to the processing allowance Δ and the tool radius R. The part contour is divided into an outer contour and an inner contour, and the enclosed area between them is the processing area. In addition, the area that is not part of the processing area is the non-processing area. Then, according to the size of the part and the cutting width, an empty grid area is constructed, where the length of each sub-grid is equal to the cutting width, and the cutting width is equal to the tool diameter. Next, the part features are mapped to the empty grid area. The grid included in the processing area is the processing grid, and the center point of the processing grid is the path point that needs to be traversed. In addition, the grid that is not included in the processing area is the non-processing grid. Finally, the tool path points are constructed according to the processing grid.

[0048] S1 in the above embodiment specifically includes: dividing the part to be processed into a plurality of processing layers by layered milling; obtaining a grid model of any processing layer, determining the contour of any processing layer according to a predetermined processing allowance and a tool radius, and determining the processing area of any processing layer based on the contour; taking the center point of the grid included in the processing area as a tool path point, thereby converting the tool path generation problem of any processing layer into a node visit order problem on the grid plane.

[0049] In Figure 3In the illustrated embodiment, the part contour is divided into an outer contour and an inner contour, with the enclosed area between them representing the area to be machined. The outer contour is formed by the part's outer boundary being inwardly contracted, with the amount of inward contraction being the sum of the machining allowance Δ and the tool radius R. The inner contour is formed by the part's inner boundary being outwardly expanded, with the amount of outward expansion being the sum of the machining allowance Δ and the tool radius R. In other embodiments, the part contour may include only the inner contour or only the outer contour, without specific limitation.

[0050] 2. Establish an MDP model for reinforcement learning tasks based on the process requirements of CNC milling.

[0051] The MDP model for reinforcement learning tasks mainly consists of the following parts:

[0052] (1) Agent and environment. For the tool path generation problem, the tool used to process a given part corresponds to the agent of the MDP model. In this embodiment, the part features and the area to be processed represented by the plane grid correspond to the environment of the MDP model, that is, in this embodiment, the environment is the state of the grid center points of the part area and the area to be processed in the grid model. Since there is only one element in the environment, the discrete tool path point, a feature matrix M can be created to represent the current state of the environment. The elements in the feature matrix correspond to the grid center points in the environment, and different numerical values ​​are used to represent the different states of the grid center points in the environment. For example Figure 4 As shown, 0 represents a cut path point, that is, the area where cutting has been completed, 1 represents an uncut path point, that is, the area that has not been cut, and -1 represents a non-processing path point, that is, the area that cannot be cut. The values ​​of the elements in the feature matrix are continuously updated as the intelligent agent interacts with the environment.

[0053] (2) State space and action space. For tool path generation, the state space of the tool is not only the information of the tool position, movement direction, etc., but also closely related to the state information of the surrounding environment. Therefore, the state space S of the tool in the environment is defined as:

[0054] S=[d,s0,s1,s2,s3,s4,s5,s6,s7,x,y];

[0055] Where d represents the current direction of the tool movement, ranging from 0 to 7, which represent the eight adjacent directions of a clockwise rotation from the top; s 0～7 Represents the environmental state information in eight adjacent directions around the discrete tool path point where the tool is located, s 0～7 The environmental state information of the eight adjacent directions of the discrete tool path point where the tool is located is rotated clockwise from the top. x and y represent the horizontal and vertical coordinates of the discrete tool path point where the tool is located, respectively. Figure 5 As shown in (a).

[0056] In the discrete grid plane, the tool path is the order of tool accessing discrete tool path points, when the tool moves from a path point a to a path point b, it is geometrically equivalent to the tool milling the material between path point a and path point b. The action space of the tool is the movement from the current tool path point to the adjacent tool path points in 8 directions and the tool lifting and feeding action. Define the action space A of the tool as:

[0057] A = [0, 1, 2, 3, 4, 5, 6, 7, 8];

[0058] In the formula, 0-7 represent the movement from the current tool path point to the adjacent tool path points in 8 directions, as shown in (b) of Figure 5 ; 8 represents the tool lifting and feeding action. In order to reduce the number of tool lifting and feeding, the tool is only allowed to perform lifting and feeding motion when there is no adjacent uncut path point in the 8 directions of the current tool path point, and a uncut path point and the direction with uncut path point are selected for feeding. At the same time, in order to reduce the search space of the agent, the action space of some special states should be processed. As shown in (c) of Figure 5 , when the tool is located at the boundary path point of the to-be-processed region, some actions will cause the tool to overcut, resulting in failure of part machining, therefore, the unactionable actions are excluded from the action space of these states to obtain the actionable action space.

[0059] (3) Instantaneous reward function. The instantaneous reward function R(s, a) of the agent is composed of three types of reward functions: the reward function of feed direction change, the reward function of tool lifting and feeding action, and the reward function of tool movement, which is the sum of the three reward functions. The design of the reward function of feed direction change and the reward function of tool movement is to generate a tool path that meets the geometric and technological requirements, and to ensure the smoothness and continuity of the tool path. The design of the reward function of tool lifting and feeding action is to reduce the tool idle stroke and shorten the total length of the tool path, and to improve the machining efficiency. The instantaneous reward value that can be obtained by all actions in the actionable action space of the agent in any state can be calculated by the instantaneous reward function. Although this instantaneous reward value does not directly determine the decision of the agent, it can reflect the immediate income of a certain action to a certain extent, and has a certain guiding effect on the decision of the agent.

[0060] In order to accurately represent the influence of different feed direction change amounts on machining efficiency in the immediate reward function, the reward function of the feed direction change is specifically obtained in the following manner: machining experiments are performed to measure the time cost of the feed direction change caused by different feed direction angle change amounts at different feed speeds, and the measurement results are shown in Table 1; the experimental data is fitted, for example, a binary polynomial fitting is performed to obtain the relationship between the time cost of the feed direction change and the feed direction angle change amount and the feed speed; and the reward function of the feed direction change is obtained based on the relationship. The time cost of the feed direction change is the difference between the machining time when the feed direction changes and the machining time when the feed direction does not change.

[0061] The machining experiments can specifically be: at any feed speed, a plurality of machining experiments with different feed direction angle change amounts are performed and compared with a comparison experiment in which the feed direction angle does not change; wherein the total length of the machining path in the plurality of machining experiments and the comparison experiment is the same, and the feed direction in any group of machining experiments with different feed direction angle change amounts can change by the same angle at least once, so that the difference between the machining time measured in any group of machining experiments with different feed direction angle change amounts and the machining time measured in the comparison experiment, divided by the number of angle changes, is the time cost when the feed direction change amount is the angle change amount in the experiment at a given feed speed.

[0062] Figure 6 represents the time cost caused by different feed speeds at the same feed direction angle change amount, and it can be known that when the angle of the feed direction change is constant, the time cost and the feed speed approximately present a linear function relationship. Figure 7 represents the time cost caused by different feed direction angle change amounts at the same feed speed, and it can be known that when the feed speed is constant, the time cost of the feed direction change and the angle of the feed direction change approximately present a quadratic function relationship. Therefore, the time cost f(v, ω) can be fitted using the following formula, wherein v represents the feed speed, ω represents the feed direction angle change amount, and f(v, ω) represents the time cost of the feed direction change caused by different feed speeds and different feed direction angle change amounts.

[0063] f(v, ω) = p 00 + p 10 v + p 01 ω + p 11 vω + p 02 ω 2 ;

[0064] The fitting results are shown in Table 1, and the coefficients in the formula are: Figure 8

[0065] p 00 = -3.684 (-6.326, -1.041); ​

[0066] p 10 =0.002091(0.0008873,0.003295);

[0067] p 01 =-0.08314(-0.1272,-0.03907);

[0068] p 11 =9.615e -5 (8.48e -5 ,0.0001075);

[0069] p 02 =0.00063(0.0004209,0.0008391);

[0070] Table 1 Experimental data of time cost (ms) at different feed speeds (mm / min) and feed direction change angles (°)

[0071]

[0072]

[0073] The reward function R1(s,a) for feed direction change gives the agent, i.e., the tool, a certain penalty (negative reward value) according to the angle of feed direction change, limiting the number and angle of tool feed direction changes. It is defined as follows:

[0074] R1(s,a)=-0.1*f(v,ω);

[0075] Assuming that the tool feed rate is constant at 2000 mm / min during the milling process, R1(s,a) can be defined as follows:

[0076]

[0077] Where Angle(s,a) is the angle between the current feed direction and the next feed direction, and its value range is [0°,180°]. Figure 9 As shown in the figure, when the agent is in state S, the current movement direction is 2. If the agent chooses action 2, the feed direction remains unchanged, and the reward value R1(s,a) it obtains is -0.049800; if the agent chooses action 3, the feed direction changes by 45°, and the reward value R1(s,a) it obtains is -0.668595; if the agent chooses action 4, the feed direction changes by 90°, and the reward value R1(s,a) it obtains is -1.542540.

[0078] The reward function R2(s, a) of the lifting and feeding motion gives the tool a negative reward value according to the lifting and feeding, that is, a larger negative reward value is given to the lifting and feeding of the tool, so as to limit the number of times of lifting and feeding of the tool, and is defined as shown in the following formula, wherein a≠8 represents that the lifting and feeding motion is not performed, and a=8 represents that the lifting and feeding motion is performed:

[0079]

[0080] As shown in Figure 10 , the agent is in state S, there is no adjacent uncut path point in the 8 directions of the current path point, and therefore the agent needs to perform the lifting and feeding motion, selects an uncut path point to feed, and the reward value R2(s, a) obtained is -10.

[0081] The reward function R3(s, a) of the tool movement is as follows: when the tool performs effective cutting, a certain positive reward value is given to the agent to guide the agent to continuously perform effective cutting; when the tool movement will generate idle stroke, a certain negative reward value is given to the agent to reduce the idle stroke of the tool. The definition is shown in the following formula:

[0082]

[0083] As shown in Figure 11 , the agent is in state S, if the agent selects action 5, the agent will move to the uncut path point, and the reward value R3(s, a) obtained is 4; if the agent selects action 4, the agent will move to the cut path point, and the reward value R3(s, a) obtained is -1.

[0084] The final instant reward function of the agent is composed of the above three types of instant reward functions, and is defined as shown in the following formula:

[0085] R(s, a)=R1(s, a)+R2(s, a)+R3(s, a);

[0086] As shown in Figure 12 , the agent is in state S, and the instant reward values that can be obtained by all actions in the action space of the agent can be calculated by the instant reward function R(s, a).

[0087] (4) State transition probability and termination state. Since the embodiment does not consider the influence of machine tool error, the state transition probability in the MDP model is constant 1, that is, given a certain state and action, the immediate reward, the next state and whether the environment terminates are determined. The maximum number of decisions of the agent can also be set. When the actual number of decisions of the agent reaches the maximum number of decisions, if the agent has not traversed all the tool path points at this time, the exploration of this round is ended in advance, so as to prevent the agent from falling into a dead loop in the process of interacting with the environment and avoid meaningless exploration of the agent. For example, the maximum number of decisions can be twice the number of tool path points of the environment. When the number of decisions of the agent is greater than twice the number of tool path points of the environment, if the agent has not traversed all the tool path points at this time, the exploration of this round is ended in advance, so as to prevent the agent from falling into a dead loop in the process of interacting with the environment. The path completion reward value can also be set. When the agent traverses all the tool path points, that is, when all the element values in the feature matrix M are less than or equal to 0, it indicates that the milling task of the given part is completed, the interaction process should be ended, and a large reward value such as 10 is given to the agent.

[0088] 3. The MDP model is solved by using a multi-layer value simulation algorithm to solve the reinforcement learning task, and a decision sequence of the agent is obtained, so as to obtain the tool path of the part to be processed. Due to the setting of the immediate reward function, the agent is prompted to preferentially select actions with less turning, less redundant cutting and less lifting of the tool; meanwhile, when there is no tool path point to be cut in the eight adjacent tool path points of the agent, the agent will perform the lifting and feeding action to find a tool path point that has not been cut as a new tool position. The above two points make the agent mostly perform effective cutting in the policy determined according to the immediate reward, and can give a path that traverses all the tool path points.

[0089] Specifically, the state value of the tool in any state is calculated according to the sum of the immediate reward values of all actions on the tool path formed from the any state as a starting point; from the state s t , the policy of determining actions by the immediate reward value is simulated to obtain the state value V(s t ):

[0090]

[0091] wherein V(s t ) represents the state value of the state s t , represents the tool path used for calculating the state value of the state s t , that is, the tool path determined by simulating the immediate reward value, that is, the policy of determining actions by the immediate reward value is simulated, and R tis the immediate reward value obtained by the agent interacting with the environment at time t, and γ is the reward discount factor, and π i represents a policy that determines actions according to the immediate reward value:

[0092]

[0093] where R(s t ,a t ) represents the immediate reward value obtained by performing action a t in state s t .

[0094] However, the path is generated by the policy (π i ) that determines actions according to the immediate reward value, and is not an excellent tool path. The machining efficiency of the tool path may be low, but the state value simulated by the policy (π i ) that determines actions according to the immediate reward value can reflect the state value of the optimal policy π * to some extent.

[0095] In order to obtain the state value of the optimal policy π * , a multi-layer value simulation algorithm is used. The action value of the tool performing any action in the current state is defined as the sum of the immediate reward value of the any action and the state value of the next layer of sub-states corresponding to the any action performed in the current state. The first layer of sub-states of the current state is the set of states after performing one step of the action in the current state. In this algorithm, when determining the action of a state, the action with the maximum action value in the next layer of sub-states is selected using the greedy strategy:

[0096]

[0097] where π g represents a policy that selects the action with the maximum action value according to the greedy algorithm, and Q(s t ,a t ) represents the action value of performing action a t in state s t . The action value of each action is calculated, and the action with the highest action value is selected using the greedy algorithm. As shown in Figure 13 , action 5 is a possible action in the current state, i.e., the state after action 5 is one of the first layer of sub-states of the current state. The state value of the state after action 5 needs to be calculated, then the immediate reward value of action 5 is calculated, and the action value of action 5 is obtained. The current state selects the next action according to the action value of each possible action.

[0098] There are two schemes to calculate the state value V(st+1 ), the first is the strategy to determine the action based on the immediate reward value (π i ) to simulate, that is, for each step, the strategy of the action with the largest immediate reward value is selected to obtain the simulation path. For example, Figure 14 As shown in the figure, the left figure represents a feasible action in the current state, that is, the end of the arrow in the left figure points to a sub-state in the first layer of sub-states of the current state. The right figure shows that the state value judgment of this sub-state is to use the immediate reward value to simulate and determine the tool path from this sub-state, and then calculate it based on the simulated path. The state value judgment strategy of this sub-state is the strategy of determining the action according to the immediate reward value (π i ). Starting from the first layer of sub-states, the action strategy is determined according to the immediate reward value (π i ) to simulate and guide the agent to complete the environmental task, thereby obtaining the state value of the first-level sub-state, as shown in the following formula:

[0099]

[0100] The state value of the first layer sub-state V(s t+1 ) Determine the action strategy based on the immediate reward (π i ) is used for simulation, which is the multi-layer value simulation algorithm when m=1.

[0101] The second solution is to follow the greedy strategy (π g ) selects the action with the largest action value in turn and guides the agent to simulate until the agent completes the environment task, thereby obtaining the state value of the first-level sub-state, as shown in the following formula:

[0102]

[0103] In the second solution, it is necessary to calculate the state value V(s) of the next layer of sub-states in any state when starting the path simulation from the first layer of sub-states. t+2 ), the next layer of sub-states is the state corresponding to the execution of one action in any state. Similar to the first layer of sub-states, two different strategies can be used to calculate the state value of the next layer of sub-states of any state in the first layer of sub-state path simulation. The first strategy is to use the immediate reward value to determine the action strategy (π i ), that is, the strategy of selecting the maximum immediate reward value to determine the action (π i ) simulates the path to calculate the state value of the next layer of sub-states of any state in the first layer of sub-state path simulation, and then simulates according to the state value of the next layer of sub-states to obtain the state value of the first layer of sub-states. Figure 15 As shown, combined Figure 14 The current state shown in the middle left picture, Figure 15The state highlighted in blue in the first figure is one of the first layer sub-states of the current state. As can be seen from the figure below the first figure, the action space of the first layer sub-state includes action 5 and action 6. The first step of the path relied on for calculating the state value of the first layer sub-state is to select the action with the maximum action value. Therefore, the action values of action 5 and action 6 need to be calculated, and further, the state values of the corresponding sub-states after action 5 and action 6 need to be calculated. The state values of the corresponding sub-states are determined by simulating the tool path from the sub-state using the immediate reward value, and then the state values are calculated based on the simulated path to obtain i As Figure 15 As can be seen from the two figures on the right in the middle box, the sum of the state value of the sub-state after action 5 from the first layer sub-state and the immediate reward value obtained by action 5 is the largest. Therefore, action 5 is selected as the first step of the path relied on for calculating the state value of the first layer sub-state, and the action is also selected according to the action value when determining the second step. The state value determination strategy of the next layer sub-state based on which the action value is calculated is the strategy of determining the action according to the immediate reward value. This process is repeated to simulate the path from the first layer sub-state, as shown in the figure below Figure 15 The state value of the first layer sub-state can be obtained, as shown in the lower figure. This scheme is the multi-layer value simulation algorithm when m = 2.

[0104] The second strategy is to select the action with the maximum action value again using the greedy strategy (π g ).

[0105] The above is the main process of the multi-layer value simulation algorithm. Multi-layer means that the strategy of determining the action according to the immediate reward value (π i ) is used to simulate the path for value calculation from a certain layer sub-state. That is, the m-layer value simulation algorithm is:

[0106] When m = 0, the action with the maximum immediate reward value is selected from the starting state to form the tool path. That is, in the current state, the action with the maximum immediate reward value is directly executed.

[0107] When m = 1, the action with the maximum action value in the current state is selected from the starting state to form the tool path. The action value of the action in the current state is calculated based on the state value of the first layer sub-state of the current state. The tool path relied on for calculating the state value of the first layer sub-state of the current state is the tool path simulated using the immediate reward value.

[0108] When m=n (n>1), the action with the maximum action value in the current state is selected from the starting state to form the tool path, wherein the action value of the action in the current state is calculated based on the state value of the first layer sub-state of the current state, the tool path used for calculating the state value of the first layer sub-state of the current state is the tool path determined by using the n-1 layer value simulation algorithm, any action in the tool path corresponding to the state value of the first layer sub-state of the current state is selected based on the action value, the tool path used for calculating the state value of the next layer sub-state based on the selected action is the tool path determined by using the n-2 layer value simulation algorithm, and the process is recursively performed until the tool path used for calculating the state value of the sub-state is the tool path determined by using the 0 layer value simulation algorithm, i.e., the tool path determined by using the immediate reward value simulation.

[0109] Specifically, m=2, and the two-layer value simulation algorithm is specifically as follows:

[0110] The action with the maximum action value in the current state is selected from the starting state to form the tool path, wherein the tool path used for calculating the state value of the first layer sub-state of the current state is the tool path determined by using the 1 layer value simulation algorithm, any action in the tool path corresponding to the state value of the first layer sub-state of the current state is selected based on the action value, the tool path used for calculating the state value of the next layer sub-state based on the selected action is the tool path determined by using the 0 layer value simulation algorithm, i.e., the tool path determined by using the immediate reward value simulation.

[0111] Further, when m=3, the three-layer value simulation algorithm is specifically as follows:

[0112] The action with the maximum action value in the current state is selected from the starting state to form the tool path, wherein the tool path used for calculating the state value of the first layer sub-state of the current state is the tool path determined by using the 2 layer value simulation algorithm, any action in the tool path corresponding to the state value of the first layer sub-state of the current state is selected based on the action value, the tool path used for calculating the state value of the next layer sub-state based on the selected action is the tool path determined by using the 1 layer value simulation algorithm;

[0113] The next layer sub-state based on which any action in the tool path corresponding to the state value of the first layer sub-state of the current state is selected is the sub-state corresponding to the state value path of the first layer sub-state, and any action in the tool path corresponding to the state value of the sub-state corresponding to the state value path of the first layer sub-state is selected according to the action value, and the tool path according to which the state value of the next layer sub-state based on which any action is selected is calculated is the tool path determined by using the 0-layer value simulation algorithm, that is, the tool path determined by using the immediate reward value simulation.

[0114] In the multi-layer value simulation algorithm, the strategy (π i ) of using the immediate reward value to determine the action at the sub-state of which layer is set, that is, m in the algorithm. When m is small, the estimation of the state value deviates from the value of the optimal strategy, but the simulation of fewer layers of sub-states reduces the running time of the algorithm; when m increases, more layers of sub-state simulation are used, and the operation time is significantly improved, but the estimation of the value is closer to the state value under the optimal strategy. Figure 16 The experimental results of different m values in a small-scale discrete grid of 9*9 are shown. It can be seen that when m increases, especially when m changes from 2 to 3, the calculation time significantly increases. When m changes from 0 to 1, the cumulative reward value obtained by the agent significantly increases, and when m is greater than 1, the cumulative reward value obtained by the agent tends to be stable and difficult to improve. Therefore, in the multi-layer value simulation algorithm, m is set to 2, so that the value of the state can be close to the state value under the optimal strategy, and the operation efficiency can be ensured.

[0115] The pseudo code of the multi-layer value simulation algorithm with m=2 is shown in the following table. In order to obtain accurate state values during simulation, the future reward value must be considered, so the reward discount factor γ=1.0 is set.

[0116]

[0117] In this embodiment, an improved version of the Monte Carlo method, the multi-layer value simulation algorithm, is used to solve the tool path generation problem. This method can quickly and accurately calculate the value of a state, thereby guiding the agent to complete the reinforcement learning task; compared with the Monte Carlo algorithm and the time difference algorithm, this method avoids the instability caused by random sampling; compared with the dynamic programming algorithm, this method reduces the calculation of the value of the state that will never appear in the processing process. Moreover, this method does not store a large amount of state values, and does not require a large amount of memory space and complex search operations on the environment.

[0118] 4、Benefits of the embodiment

[0119] In order to verify the effectiveness of the strategy-free milling tool path generation method proposed in this embodiment, the same experimental environment as in the first embodiment is used. Figure 17The cavity workpiece shown in the simulation experiment, the simulation experiment parameters are shown in Table 2. First, the part is layered by layered milling, and each layer is converted into a grid model; then the tool path is generated based on the grid model using the multi-layer value simulation algorithm.

[0120] The tool path generated by the method provided in the embodiment is compared with the tool path generated by the three common cutting modes (reciprocal, follow part, and follow perimeter) in NX to prove the effectiveness and advancement of the method provided in the embodiment. The tool paths generated by the four methods are simulated in the cutting simulation software CIMCO Edit 8.02.23, and the cutting results are shown in Figure 18 Figure 18 In (a), the method for generating a tool path for milling without strategy is provided in the embodiment, (b) is the NX reciprocal cutting mode, (c) is the NX follow part cutting mode, and (d) is the NX follow perimeter cutting mode. It can be observed that the tool paths generated by the four methods can complete the milling of the target part. The machining time, cutting time, trajectory length, and cutting distance of the tool paths generated by different methods are counted using the cutting simulation software CIMCO Edit 8.02.23, and the counting results are shown in Table 3. The improvement ratio is calculated based on the machining time. It can be seen that under the condition that the part model, machining allowance, feed speed, cutting width, tool diameter, and other cutting conditions are consistent, the tool path generated by the method for generating a tool path for milling without strategy provided in the embodiment has shorter cutting distance and trajectory length, less redundant cutting, and higher machining efficiency. Since there is no geometric-level tool path generation strategy, the tool does not need to be limited by the given strategy when generating the path, and can select a reasonable motion direction according to the current milling state of the workpiece and the motion state of the tool, so the generated tool path has less redundant path and higher machining efficiency.

[0121] Table 2 Simulation experiment parameters

[0122]

[0123] Table 3 Simulation experiment results

[0124]

[0125] ​Further, the second embodiment provides a strategy-free milling tool path generation system for implementing the method described in any of the above, which can be understood in conjunction with the above method. The system comprises: a grid establishment module, configured to obtain a grid model of a part to be machined and determine a region to be machined on the grid model, take a grid center point as a tool path point, convert a tool path generation problem of the region to be machined into a tool path point access sequence problem, and convert it into a reinforcement learning task; a model establishment module, configured to establish a Markov decision process model (MDP model) of the reinforcement learning task, wherein an agent in the MDP model is a tool, an environment is a state of a grid center point in the grid model, a state space and a corresponding action space of the tool in the environment are defined, an immediate reward function of the tool action is set, after the tool performs an action according to the action space corresponding to the state space, the environment and the state space change and an immediate reward value corresponding to the immediate reward function can be obtained; and a model solving module, configured to solve the MDP model based on the immediate reward value of the tool action, and generate a tool path of the part to be machined.

[0126] Those skilled in the art will easily understand that the above description is only the preferred embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A strategy-free milling tool path generation method, characterized in that: include: S1, obtaining a mesh model of the part to be machined and determining the area to be machined on the mesh model, using the mesh center point as the tool path point, and converting the tool path generation problem for the area to be machined into a tool path point access sequence problem, thereby converting it into a reinforcement learning task; S2, establishing a Markov decision process model (MDP model) for the reinforcement learning task, wherein the intelligent agent in the MDP model is the tool, and the environment is the state of the center point of the grid in the grid model. The state space of the tool in the environment and the corresponding action space are defined, and an immediate reward function is set for the tool action. After the tool performs an action in the action space corresponding to the state space, the environment and state space change and a corresponding immediate reward value can be obtained according to the immediate reward function. S3, solving the MDP model based on the instant reward value of the tool action to generate the tool path of the part to be processed; Immediate reward function for tool action Specifically, the reward function for changing the feeding direction , the reward function for the knife-lifting action And the reward function for tool movement the sum of; Among them, the reward function for changing the feed direction gives the tool a negative reward value according to the angle of the feed direction change, so as to limit the number and angle of the tool feed direction change; the reward function for lifting the tool and feeding the tool gives the tool a negative reward value according to the tool lifting and feeding, so as to limit the number of times the tool is lifted and fed; the reward function for tool movement is: when the tool performs effective cutting, the tool is given a positive reward value, and when the tool movement produces an empty stroke, the tool is given a negative reward value to reduce the tool's empty stroke.

2. The strategy-free milling tool path generation method according to claim 1, wherein: S1 specifically includes: Dividing the part to be processed into several layers to be processed by layered milling; Obtaining a mesh model of any of the layers to be processed, determining a contour of any of the layers to be processed according to a preset machining allowance and a tool radius, and determining a to-be-processed area of ​​any of the layers to be processed based on the contour; The grid center points contained in the to-be-processed area are used as tool path points, thereby converting the tool path generation problem of any to-be-processed layer into a node access sequence problem on the grid plane.

3. The strategy-free milling tool path generation method according to claim 1, wherein: S2 defines the state space of the tool in the environment S for: ； Where, d Indicates the current movement direction of the tool; s 0~7 Represents the environmental status information in 8 adjacent directions around the discrete tool path point where the tool is located; x and y Respectively represent the abscissa and ordinate of the discrete tool path point where the tool is located; Tool motion space A for: ； Where 0 to 7 represent the movement from the current tool path point to the adjacent tool path points in 8 directions, and 8 represents the tool lifting and feeding action.

4. The strategy-free milling tool path generation method according to claim 1, wherein: The reward function for changing the feeding direction is obtained in the following way: A machining experiment was conducted to measure the time cost of changing the feed direction caused by different feed direction angle changes at different feed speeds. The time cost of changing the feed direction is the difference between the machining time when the feed direction is changed and the machining time when the feed direction remains unchanged. Fitting the experimental data to obtain the relationship between the time cost of changing the feed direction and the feed speed and feed direction angle change; According to the relationship between the time cost of changing the feeding direction and the feeding speed and the amount of change in the feeding direction angle, the reward function of changing the feeding direction is obtained.

5. The strategy-free milling tool path generation method according to claim 4, wherein: The immediate reward function of the tool action is specifically: ； ； ； ； Where, v It represents the feed rate. It represents the change in the feed direction angle. It represents the time cost of changing the feed direction caused by the change in feed direction angle at different feed speeds. It means that the knife is not lifted and fed. It indicates that the knife is being lifted and fed.

6. The strategy-free milling tool path generation method according to claim 1, wherein: The MDP model also includes: Set the state transition probability to 1; And / or, set a maximum number of decisions for the agent. When the actual number of decisions made by the agent reaches the maximum number of decisions, if the agent has not traversed all tool path points at this time, end this round of exploration in advance to prevent the agent from falling into an infinite loop during the interaction with the environment.

7. The strategy-free milling tool path generation method according to claim 1, wherein: S3 specifically includes: solving the MDP model using a multi-layer value simulation algorithm to obtain the decision sequence of the intelligent agent, thereby obtaining the tool path of the part to be processed; Specifically, when the tool is in any state, the state value of any state is calculated based on the sum of the immediate reward values ​​of all actions on the tool path formed from any state as the starting point; the action value of the tool performing any action in the current state is the sum of the immediate reward value of any action and the state value of the next sub-state corresponding to the execution of any action from the current state; the first sub-state of the current state is the state set after executing one feasible action from the current state; m The layer value simulation algorithm is: when When , the actions with the largest immediate reward value are selected from the starting state to form the tool path; when When , the action with the largest action value in the current state is selected in sequence from the starting state to form a tool path, wherein the action value of the action in the current state is calculated based on the state value of the first-level sub-state of the current state, and the tool path based on which the state value of the first-level sub-state of the current state is calculated is the tool path determined by simulation using the immediate reward value; when When the tool path is formed, the action with the largest action value in the current state is selected from the starting state in sequence. The action value of the action in the current state is calculated based on the state value of the first layer sub-state of the current state. The tool path based on the state value calculation of the first layer sub-state of the current state is used n -1 layer value simulation algorithm solves the tool path determined by the state value of the first layer sub-state of the current state. Any step in the tool path corresponding to the state value of the current state is selected based on the action value. The tool path based on which the state value of the next layer sub-state based on which any step is selected is used. n -The tool path determined by the 2-layer value simulation algorithm is recursively solved until the state value of the sub-state is calculated based on the tool path determined by the 0-layer value simulation algorithm, that is, the tool path determined by the immediate reward value simulation.

8. The strategy-free milling tool path generation method according to claim 7, wherein: make ,At this time, the two-layer value simulation algorithm is specifically as follows: From the starting state, the actions with the largest action value in the current state are selected in sequence to form a tool path, wherein the tool path based on which the state value of the first-level sub-state of the current state is calculated is the tool path determined by solving the 1-level value simulation algorithm, and any action in the tool path corresponding to the state value of the first-level sub-state of the current state is selected based on the action value, and the tool path based on which the state value of the next-level sub-state on which any action is based is calculated is the tool path determined by solving the 0-level value simulation algorithm, that is, the tool path determined by simulation using the immediate reward value.

9. A strategy-free milling tool path generation system, characterized in that: The method for generating a tool path for strategy-free milling machining according to any one of claims 1 to 8 comprises: A mesh building module is used to obtain a mesh model of the part to be machined and determine the area to be machined on the mesh model. The mesh center point is used as the tool path point. The tool path generation problem for the area to be machined is converted into a tool path point access sequence problem, thereby converting it into a reinforcement learning task. A model building module is used to establish a Markov decision process model (MDP model) for reinforcement learning tasks, wherein the intelligent agent in the MDP model is a tool, and the environment is the state of the center point of the grid in the grid model. The state space of the tool in the environment and the corresponding action space are defined, and an immediate reward function is set for the tool action. After the tool performs an action in the action space corresponding to the state space, the environment and state space change and a corresponding immediate reward value can be obtained according to the immediate reward function. The model solving module is used to solve the MDP model based on the instant reward value of the tool action and generate the tool path of the part to be processed.

Images