Policy-free milling tool path generation method and system

Abstract

The invention belongs to the related technical field of numerical control machining and artificial intelligence, and discloses a strategy-free milling tool path generation method and system, and the method comprises the steps: obtaining a grid model of a to-be-machined part, determining a to-be-machined region, converting a tool path generation problem into a grid center point access sequence problem, and obtaining a grid center point access sequence; converting the task into a reinforcement learning task; an MDP model of the reinforcement learning task is established, an intelligent agent in the MDP model is a state of a cutter and an environment is a grid center point, a state space and a corresponding action space of the cutter are defined, and an instant reward function of cutter action is set; the MDP model is solved, and the tool path of the part to be machined is generated. According to the method, the reinforcement learning method and the tool path generation are effectively combined, the tool path generation problem is converted into the solution reinforcement learning task, a tool path generation strategy does not need to be given, and the tool path can be automatically generated, so that the manual dependence is reduced, and the machining efficiency is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field related to numerical control machining and artificial intelligence, and more specifically, relates to a method and system for generating a tool path for strategy-free milling machining. Background Art

[0002] CNC machine tools are mechanical equipment that can process complex parts with high precision. They are widely used in important manufacturing fields such as aerospace, mold manufacturing, and automobile manufacturing. Tool path planning for CNC machine tools is an important part of the CNC machining process, which directly affects the quality and efficiency of CNC machining, and even affects the service life of CNC machine tools.

[0003] A method for generating tool paths for three-axis equal residual height CNC machining of point cloud models is disclosed in the Chinese invention patent specification CN 110308702 B. The method directly calculates the residual height points on the tool contour by calculating the point cloud contour point set on the normal plane of the feed direction. The method can directly calculate the three-axis equal residual height CNC machining tool paths for the point cloud model. While meeting the residual height requirements, the line spacing is large and the total tool path length is small. A method for generating tool contact trajectories for equal residual heights of a triangular mesh model is disclosed in the Chinese invention patent specification CN108594764 B. The method divides the tool contacts in the tool contact trajectory into three types: convex tool contacts, concave tool contacts, and smooth tool contacts. The actual offset point is calculated based on the anisotropic offset in the tool contact tangent plane, and finally the machining residual height between tool trajectories is closer to the maximum allowable value, thereby improving the efficiency of CNC machining. A method for generating equal residual height tool paths for three-axis CNC machining of free-form surfaces is disclosed in the Chinese invention patent specification CN 116974237 A. This method does not require equal spacing of the surface and can directly and efficiently calculate the equal residual height tool paths for the free-form surfaces.

[0004] At present, the main tool path generation method is to use computer-aided manufacturing software (CAM) to plan the tool path, generate G code to control the movement of the tool, and then transfer the G code to the CNC machine tool for CNC machining. In mainstream CAM software such as NX and MasterCAM, many traditional tool path generation methods are integrated, among which the more common methods include the iso-planar method, the iso-parametric method, and the iso-cusp height method. The above three tool path generation methods require the input of a large number of parameters and options, including cutting parameters, geometric constraints, etc. The input of these processing parameters directly affects the processing quality and efficiency, which will hinder beginners, reduce processing efficiency and quality, and even reduce the service life of CNC machines or tools. At present, the setting of these parameters is heavily dependent on the experience of CAM engineers. At present, the research on tool path generation automation mainly focuses on traditional methods. The selection of strategy parameters in the tool path generation process is highly dependent on the experience of CAM engineers, and it is impossible to effectively improve the degree of intelligence. More advanced and adaptive tool path generation methods are needed to make up for the defects of traditional methods in processing quality, processing efficiency and intelligence. Summary of the invention

[0005] In view of the above defects or improvement needs of the prior art, the present invention provides a method and system for generating a strategy-free milling tool path. The technical problem to be solved is to reduce the dependence of strategy parameter selection in the tool path generation process on the experience of CAM engineers. The method can autonomously generate tool paths without the need for engineers to input strategy parameters, thereby reducing dependence on engineers' experience, which is conducive to improving intelligence and improving processing efficiency.

[0006] To achieve the above object, according to one aspect of the present invention, a method for generating a strategy-free milling tool path is provided, comprising:

[0007] S1, obtaining a mesh model of a part to be processed and determining a region to be processed on the mesh model, taking a mesh center point as a tool path point, converting a tool path generation problem of the region to be processed into a tool path point access sequence problem, thereby converting it into a reinforcement learning task;

[0008] S2, establish a Markov decision process model (MDP model) of the reinforcement learning task, wherein the intelligent agent in the MDP model is the tool, the environment is the state of the center point of the grid in the grid model, define the state space of the tool in the environment and the corresponding action space, set the immediate reward function of the tool action, after the tool performs an action according to the action space corresponding to the state space, the environment and the state space change and can obtain a corresponding immediate reward value according to the immediate reward function;

[0009] S3, solving the MDP model based on the instant reward value of the tool action to generate a tool path for the part to be processed.

[0010] According to another aspect of the present invention, there is provided a strategy-free milling tool path generation system, comprising:

[0011] A mesh building module is used to obtain a mesh model of the part to be processed and determine the area to be processed on the mesh model, using the center point of the mesh as the tool path point, and converting the tool path generation problem of the area to be processed into a tool path point access sequence problem, thereby converting it into a reinforcement learning task;

[0012] A model building module is used to build a Markov decision process model (MDP model) of a reinforcement learning task, wherein the intelligent agent in the MDP model is a tool, the environment is the state of the center point of the grid in the grid model, the state space of the tool in the environment and the corresponding action space are defined, and an immediate reward function of the tool action is set. After the tool performs an action according to the action space corresponding to the state space, the environment and the state space change and a corresponding immediate reward value can be obtained according to the immediate reward function;

[0013] The model solving module is used to solve the MDP model based on the instant reward value of the tool action and generate a tool path for the part to be processed.

[0014] In general, compared with the prior art, the above technical solutions conceived by the present invention provide a method and system for generating a tool path for strategy-free milling machining:

[0015] 1. This method effectively combines the reinforcement learning method with tool path generation. By converting the tool path generation problem into a reinforcement learning task, it does not need to give a tool path generation strategy, and can generate tool paths autonomously, thereby reducing manual dependence and improving the degree of intelligence while ensuring processing quality. In addition, since there is no tool path generation strategy at the geometric level, when generating the path, the tool does not need to be restricted by the given strategy, and can choose a reasonable movement direction according to the current state of the milling workpiece and the movement state of the tool. Therefore, the generated tool path has fewer redundant paths and higher processing efficiency, and can generate a tool path with higher processing efficiency;

[0016] 2. The MDP model takes into account the factors affecting the processing efficiency during the processing, and sets a comprehensive reward function, which can guide the decision-making of the intelligent agent to a certain extent;

[0017] 3. Using the multi-layer value simulation algorithm to calculate the tool path, the value of a state can be calculated quickly and accurately, thereby guiding the intelligent agent to complete the reinforcement learning task. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flow chart of the strategy-free milling tool path generation method provided by the present invention;

[0019] Figure 2 It is an overall flow chart of the present invention;

[0020] Figure 3 A schematic diagram of a grid conversion process during the implementation of the present invention;

[0021] Figure 4 Schematic diagram of the corresponding relationship between the environment and the feature matrix M during the implementation of the present invention, (a) the initial state of the environment; (b) the tool makes a decision, moves to a new state, and the feature matrix is ​​updated; (c) the environment is terminated;

[0022] Figure 5 Schematic diagram of the state space and action space of the agent during the implementation of the present invention, (a) agent state; (b) agent action space; (c) special state processing;

[0023] Figure 6 A relationship diagram between the time cost of changing the feeding direction and the feeding speed during the implementation of the present invention;

[0024] Figure 7 A graph showing the relationship between the time cost and the angle of the feed direction change during the implementation of the present invention;

[0025] Figure 8 A graph showing the relationship between the time cost of changing the feeding direction and the feeding speed and angle during the implementation of the present invention;

[0026] Fig. 9 A schematic diagram of a reward function for changing the feeding direction during the implementation of the present invention;

[0027] Fig.10 It is a schematic diagram of the reward function of the knife lifting and cutting action during the implementation of the present invention;

[0028] Fig.11 A schematic diagram of a reward function for tool movement during the implementation of the present invention;

[0029] Fig.12This is a schematic diagram of the final instant reward function during the implementation of the present invention;

[0030] Fig.13 It is a schematic diagram of selecting actions according to the value of the actions during the implementation of the present invention;

[0031] Fig.14 A schematic diagram of sub-state value simulation calculation for a strategy of determining an action according to an instant reward value during the implementation of the present invention;

[0032] Fig.15 A schematic diagram of the value of the first layer sub-state obtained by simulating according to the value of the second layer sub-state during the implementation of the present invention;

[0033] Fig.16 This is a schematic diagram of the impact of different number of layers m on the running time and overall reward of the multi-layer value simulation algorithm during the implementation of the present invention;

[0034] Fig.17 This is a simulation case diagram during the implementation of the present invention;

[0035] Fig.18 This is a diagram of simulation experiment results during the implementation of the present invention. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0037] See also Figure 1 The first embodiment of the present invention provides a strategy-free milling tool path generation method, and the strategy-free milling tool path generation method includes:

[0038] S1, obtaining a mesh model of a part to be processed and determining a region to be processed on the mesh model, taking a mesh center point as a tool path point, converting a tool path generation problem of the region to be processed into a tool path point access sequence problem, thereby converting it into a reinforcement learning task;

[0039] S2, establish a Markov decision process model (MDP model) of the reinforcement learning task, wherein the intelligent agent in the MDP model is the tool, the environment is the state of the center point of the grid in the grid model, define the state space of the tool in the environment and the corresponding action space, set the immediate reward function of the tool action, after the tool performs an action according to the action space corresponding to the state space, the environment and the state space change and can obtain a corresponding immediate reward value according to the immediate reward function;

[0040] S3, solving the MDP model based on the instant reward value of the tool action to generate a tool path for the part to be processed.

[0041] This embodiment simplifies the part to be processed into a grid model, and transforms the tool path generation problem into a node access order problem on the grid model. The node access order problem is a typical Markov decision process (MDP), so the tool path generation problem can be transformed into a reinforcement learning task. An MDP model for tool path generation can be established according to the process requirements of the CNC milling process, specifically including setting the agent and environment of the model, defining the state space and action space, and setting an immediate reward function according to the process requirements of the milling process. The state space reflects the current state of the agent, and the action space reflects the current movable actions of the agent; after the agent performs an action, the state space, action space and environment will be updated and changed respectively, and the immediate reward value of the action can be obtained through the immediate reward function.

[0042] The process of the agent traversing the path points can be judged by the changes in the environment. When the agent traverses all the path points, the interaction between the agent and the environment ends, and the traversed path process is the generated tool path. At the same time, the tool path can be judged according to the instant reward value to obtain a better path. That is, the MDP model is solved according to the instant reward value to generate the tool path of the part to be processed.

[0043] The technical problem to be solved by this embodiment is to reduce the dependence of strategy parameter selection in the tool path generation process on the experience of CAM engineers, and to provide a strategy-free milling tool path generation method. This method can independently generate tool paths without the need for a given tool path generation strategy, so engineers do not need to be familiar with the characteristics and applicable conditions of various tool path generation strategies. The requirements on engineers' skills and experience are relatively low, and the tool path generation process can be simplified to a certain extent. At the same time, since the tool path generation process no longer needs to be constrained by a given tool path generation strategy, the generated tool path processing efficiency is higher.

[0044] Figure 2The overall flow chart of the method of this embodiment is shown. This implementation case first divides the three-dimensional part into several layers by layered milling, and discretizes each layer into a square grid plane to obtain a grid model of the part, and converts the tool path generation problem into a reinforcement learning task; then, the MDP model of the reinforcement learning task is established according to the process requirements of the CNC milling process, and the relationship between the time cost of changing the feed direction and the angle and feed speed of the feed direction change is established through processing experimental data, and the instant reward function is designed accordingly; finally, the multi-layer value simulation algorithm is used to solve the reinforcement learning task, and the decision sequence of the intelligent agent is obtained, thereby obtaining the tool path of the part.

[0045] The entire implementation process is as follows:

[0046] 1. Divide the three-dimensional part into several layers by layered milling, and discretize each layer into a square grid plane to obtain the mesh model of the part. The discrete tool path points for milling are constructed from the center points of the square grid. Using this simplified model, the tool path can be defined as the order in which the milling tool visits the discrete tool path points on the grid plane, thereby converting the tool path generation problem into a node access order problem on the grid plane.

[0047] The process of grid conversion is as follows Figure 3 As shown in the figure, for a given milling task of a part to be processed, it is first necessary to distinguish between the processing area and the non-processing area, determine the part contour according to the machining allowance Δ and the tool radius R, and divide the part contour into an outer contour and an inner contour. The closed area between them is the area to be processed, and the area outside is the non-processing area. Then, according to the part size and the cutting width, an empty grid area is constructed, in which the side length of each sub-square grid is equal to the cutting width, and the cutting width is equal to the tool diameter. Next, the part features are mapped to the empty grid area, the grid contained in the area to be processed is the grid to be processed, and the center point of the grid to be processed is the path point that needs to be traversed, and the grids outside are non-processing grids, that is, the grids that are half-contained or not contained in the area to be processed are non-processing grids. Finally, the tool path points are constructed based on the processing grid.

[0048] That is, S1 in the above embodiment specifically includes: dividing the part to be processed into several layers to be processed by layered milling; obtaining a mesh model of any of the layers to be processed, determining the contour of any of the layers to be processed according to a preset machining allowance and a tool radius, and determining the processing area of ​​any of the layers to be processed based on the contour; using the mesh center point contained in the processing area as the tool path point, thereby converting the tool path generation problem of any of the layers to be processed into a node access order problem on the mesh plane.

[0049] exist Figure 3In the illustrated embodiment, the part contour is divided into an outer contour and an inner contour, and the enclosed area between them is the area to be processed. The outer contour is formed by the outer boundary of the part being retracted inward, and the amount of retraction is the sum of the machining allowance Δ and the tool radius R; the inner contour is formed by the inner boundary of the part being expanded outward, and the amount of expansion is the sum of the machining allowance Δ and the tool radius R. In other embodiments, the part contour may also include only the inner contour or the outer contour, which is not specifically limited.

[0050] 2. Establish the MDP model of reinforcement learning task according to the process requirements of CNC milling process.

[0051] The MDP model for reinforcement learning tasks mainly consists of the following parts:

[0052] (1) Agent and environment. For the tool path generation problem, the tool used to process a given part corresponds to the agent of the MDP model. In this embodiment, the part features and the area to be processed represented by the plane grid correspond to the environment of the MDP model, that is, in this embodiment, the environment is the state of the grid center points of the part area and the area to be processed in the grid model. Since there is only one element in the environment, namely the discrete tool path point, a feature matrix M can be created to represent the current state of the environment. The elements in the feature matrix correspond to the grid center points in the environment, and different numerical values ​​are used to represent the different states of the grid center points in the environment. For example Figure 4 As shown, 0 represents a cut path point, that is, the area where cutting has been completed, 1 represents an uncut path point, that is, the area where cutting has not been completed, and -1 represents a non-processing path point, that is, the area that cannot be cut. The values ​​of the elements in the feature matrix are continuously updated as the agent interacts with the environment.

[0053] (2) State space and action space. For tool path generation, the state space of the tool is not only the information of the tool position, movement direction, etc., but also closely related to the state information of the surrounding environment. Therefore, the state space S of the tool in the environment is defined as:

[0054] S=[d,s0,s1,s2,s3,s4,s5,s6,s7,x,y];

[0055] Where d represents the current moving direction of the tool, ranging from 0 to 7, where 0 to 7 represent the eight adjacent directions of a clockwise rotation from the top; s 0～7 Represents the environmental state information in eight adjacent directions around the discrete tool path point where the tool is located, s 0～7 The environmental state information of the eight adjacent directions of the discrete tool path point where the tool is located is rotated clockwise from the top; x and y represent the horizontal and vertical coordinates of the discrete tool path point where the tool is located, respectively, such as Figure 5 As shown in (a).

[0056] In the discrete grid plane, the tool path is the order in which the tool visits the discrete tool path points. When the tool moves from a path point a to a path point b, it is geometrically equivalent to the tool milling away the material between path point a and path point b. The tool's motion space is the movement from the current tool path point to the adjacent tool path points in 8 directions and the tool's lifting and feeding action. The tool's motion space A is defined as:

[0057] A=[0,1,2,3,4,5,6,7,8];

[0058] Where 0 to 7 represent the movement from the current tool path point to the adjacent tool path point in 8 directions, such as Figure 5 As shown in (b); 8 represents the tool lifting and feeding action. In order to reduce the number of tool lifting and feeding, the tool is restricted to lift and feed only when there are no adjacent uncut path points in the 8 directions of the current tool path point, and select an uncut path point and a direction with uncut path points for feeding. At the same time, in order to reduce the search space of the intelligent agent, the action space of some special states should be processed. Figure 5 As shown in (c) in the figure, when the tool is located at the path point at the boundary of the area to be processed, certain actions will cause the tool to overcut, resulting in part processing failure. Therefore, the inoperable actions are excluded from the action space of these states to obtain its operable action space.

[0059] (3) Instant reward function. The agent's instant reward function R(s,a) consists of three types of reward functions: the reward function for feed direction change, the reward function for tool lift and feed, and the reward function for tool movement. It is the sum of the three reward functions. The reward function for feed direction change and tool movement is designed to generate a tool path that meets the geometric process requirements and ensures the smoothness and continuity of the tool path. The reward function for tool lift and feed is designed to reduce the tool's idle stroke, shorten the total length of the tool path, and improve processing efficiency. When the agent is in any state, the instant reward value that can be obtained for all actions in its movable action space can be calculated through the instant reward function. Although the instant reward value does not directly determine the agent's decision, it can reflect the instant benefit of a certain action to a certain extent and has a certain guiding role in the agent's decision.

[0060] In order to accurately represent the impact of different feed direction changes on processing efficiency in the instant reward function, the reward function of feed direction change is obtained in the following way: a processing experiment is conducted to measure the time cost of feed direction change caused by different feed direction angle changes at different feed speeds, and the measurement results are shown in Table 1; and the experimental data is fitted, for example, a binary polynomial is fitted to obtain the relationship between the time cost of feed direction change and the feed direction angle change and feed speed; and the reward function of feed direction change is obtained based on this relationship. The time cost of feed direction change is the difference between the processing time when the feed direction changes and the processing time when the feed direction remains unchanged.

[0061] The processing experiment can be specifically as follows: at any feed speed, multiple groups of processing experiments with different feed direction angle changes are carried out and compared with the comparison experiment with the feed direction angle unchanged; wherein, the total length of the processing paths in the multiple groups of processing experiments and the comparison experiment is the same, and the feed direction in any group of processing experiments with the feed direction angle change can be changed at least once according to the same angle, so that the difference between the processing time measured by the processing experiment of any group of processing experiments with the feed direction angle change and the processing time measured by the comparison experiment, divided by the number of angle changes, is the time cost when the feed direction change is the angle change in the experiment at the given feed speed of the experiment.

[0062] Figure 6 It represents the time cost caused by different feed speeds under the same feed direction angle change. It can be seen that when the angle of feed direction change is constant, the time cost and feed speed are approximately in a linear function relationship. Figure 7 It represents the time cost caused by different feed direction angle changes at the same feed speed. It can be seen that when the feed speed is constant, the time cost of the feed direction change is approximately a quadratic function of the angle of the feed direction change. Therefore, the following formula can be used to fit the time cost f(v,ω), where v represents the feed speed, ω represents the feed direction angle change, and f(v,ω) represents the time cost of the feed direction change caused by different feed direction angle changes at different feed speeds.

[0063] f(v,ω)=p 00 +p 10 v+p 01 ω+p 11 vω+p 02 ω 2 ;

[0064] The fitting results are as follows Figure 8 As shown, the coefficients in the formula are:

[0065] p 00 =-3.684(-6.326,-1.041);

[0066] p 10 =0.002091(0.0008873,0.003295);

[0067] p 01 =-0.08314(-0.1272,-0.03907);

[0068] p 11 =9.615e -5 (8.48e -5 ,0.0001075);

[0069] p 02 =0.00063(0.0004209,0.0008391);

[0070] Table 1 Experimental data of time cost (ms) at different feed speeds (mm / min) and feed direction change angles (°)

[0071]

[0072]

[0073] The reward function R1(s,a) for changing the feed direction gives the agent, i.e. the tool, a certain penalty (negative reward value) according to the angle of the feed direction change, limiting the number and angle of the tool feed direction change, and is defined as follows:

[0074] R1(s,a)=-0.1*f(v,ω);

[0075] Assuming that during the milling process, the feed rate of the tool is constant at 2000 mm / min, then R1(s,a) can be defined as follows:

[0076]

[0077] Where Angle(s,a) is the angle between the current feed direction and the next feed direction, and its value range is [0°,180°]. Fig. 9 As shown in the figure, when the agent is in state S, the current movement direction is 2. If the agent chooses action 2, the feed direction remains unchanged, and the reward value R1(s,a) obtained is -0.049800; if the agent chooses action 3, the feed direction changes by 45°, and the reward value R1(s,a) obtained is -0.668595; if the agent chooses action 4, the feed direction changes by 90°, and the reward value R1(s,a) obtained is -1.542540.

[0078] The reward function R2(s,a) of the tool lifting and feeding action gives a negative reward value to the tool according to the tool lifting and feeding, that is, a larger negative reward value is given to the tool lifting and feeding to limit the number of times the tool lifts and feeds. It is defined as shown in the following formula, where a≠8 means that the tool lifting and feeding action is not performed, and a=8 means that the tool lifting and feeding action is performed:

[0079]

[0080] like Fig.10 As shown in the figure, when the agent is in state S, there are no adjacent uncut path points in the eight directions of the current path point. Therefore, the agent needs to lift the knife and move forward, and select an uncut path point to move forward. The reward value R2(s,a) it obtains is -10.

[0081] The reward function R3(s,a) for tool movement is: when the tool performs effective cutting, a certain positive reward value is given to the agent to guide the agent to continuously perform effective cutting; when the tool movement will produce an empty stroke, a certain negative reward value is given to the agent to reduce the empty stroke of the tool. The definition is as follows:

[0082]

[0083] like Fig.11 As shown in the figure, the agent is in state S. If the agent chooses action 5, the agent will move to the uncut path point, and the reward value R3(s,a) it obtains is 4; if the agent chooses action 4, the agent will move to the cut path point, and the reward value R3(s,a) it obtains is -1.

[0084] The agent's final immediate reward function is composed of the above three types of immediate reward functions, and is defined as follows:

[0085] R(s,a)=R1(s,a)+R2(s,a)+R3(s,a);

[0086] like Fig.12 As shown in the figure, the agent is in state S, and the immediate reward values ​​that can be obtained for all actions in its action space can be calculated by the immediate reward function R(s,a).

[0087] (4) State transition probability and termination state. Since the present embodiment does not consider the influence of machine tool errors, the state transition probability in the MDP model is always 1, that is, given a certain state and action, the immediate reward, the next state and the result of whether to terminate the environment given by the environment are determined. The maximum number of decisions of the agent can also be set. When the actual number of decisions of the agent reaches the maximum number of decisions, if the agent has not traversed all tool path points at this time, the current round of exploration is terminated in advance to prevent the agent from falling into an infinite loop during the interaction with the environment and avoid meaningless exploration of the agent. For example, the maximum number of decisions can be twice the number of all tool path points in the environment. When the number of decisions of the agent is greater than twice the number of all tool path points in the environment, if the agent has not traversed all tool path points at this time, the current round of exploration should be terminated in advance to prevent the agent from falling into an infinite loop during the interaction with the environment. A path completion reward value can also be set. When the agent has traversed all tool path points, that is, when all element values ​​in the feature matrix M are less than or equal to 0, it means that the milling task of the given part is completed, the interaction process should be ended, and a larger reward value, such as 10, should be given to the agent.

[0088] 3. Use the multi-layer value simulation algorithm to solve the MDP model of the reinforcement learning task, obtain the decision sequence of the agent, and thus obtain the tool path of the part to be processed. Due to the setting of the immediate reward function, the agent is prompted to give priority to actions with fewer turns, less redundant cutting, and less tool lifting; at the same time, when there are no tool path points to be cut among the 8 adjacent tool path points of the agent, the agent will perform a tool lifting and feeding action, looking for a tool path point that has not been cut as the new tool position. The above two points make it possible for the agent to perform effective cutting in most cases in the strategy determined by the immediate reward, and to provide a path that traverses all tool path points.

[0089] Specifically, when the tool is in any state, the state value of the state is calculated according to the sum of the instant reward values ​​of all actions on the tool path formed from the state as the starting point; t At the beginning, the strategy of determining the action with the immediate reward value is simulated, and the state value V(s t )for:

[0090]

[0091] Among them, V(s t ) indicates state s t The state value of Indicates state s t The tool path based on which the state value is calculated is the tool path determined by the instant reward value simulation, that is, the strategy of determining the action by the instant reward value is simulated. tis the instantaneous reward value obtained by the agent's interaction with the environment at time t, γ is the reward discount factor, and π i Represents a strategy for determining actions based on immediate reward values:

[0092]

[0093] Among them, R(s t ,a t ) indicates state s t Next, perform action a t The instant reward value that can be obtained.

[0094] However, this path only determines the action strategy based on the immediate reward value (π i ) is not an excellent tool path, and its processing efficiency may be relatively low, but the strategy of determining the action based on the immediate reward value (π i ) The value of the state generated by the simulation can reflect the optimal strategy π to a certain extent * The status value.

[0095] In order to obtain the optimal strategy π * The state value is determined by using a multi-layer value simulation algorithm. The action value of the tool performing any action in the current state is defined as the sum of the immediate reward value of the action and the state value of the next layer of sub-states corresponding to the execution of the action from the current state; the first layer of sub-states of the current state is the state set after executing one actionable action from the current state. In this algorithm, when determining the action of a state, a greedy strategy is used to select the action with the largest action value in the next layer of sub-states:

[0096]

[0097] Among them, π g represents a strategy of selecting the action with the largest action value according to the greedy algorithm, Q(s t ,a t ) indicates state s t Next, perform action a t Calculate the action value of each actionable action, that is, all actions in the actionable action space of the current state, and then use the greedy algorithm to select the action with the highest action value. Fig.13 As shown, action 5 is a feasible action in the current state, that is, the state after action 5 is a state in the first-level sub-state of the current state. It is necessary to calculate the state value of the state after action 5, and then calculate the immediate reward value of action 5, and then obtain the action value of action 5. The current state selects the next action according to the action value of each feasible action.

[0098] There are two ways to calculate the state value V(s) of the first layer sub-state:t+1 ), the first is the strategy to determine the action based on the immediate reward value (π i ) is used for simulation, that is, the action strategy with the largest immediate reward value is selected for each step to obtain the simulation path. For example, Fig.14 As shown in the figure, the left figure represents a feasible action in the current state, that is, the end of the arrow in the left figure points to a sub-state in the first layer of sub-states of the current state. The right figure shows that the state value judgment of this sub-state is to use the immediate reward value to simulate and determine the tool path from this sub-state, and then calculate it based on the simulated path. The state value judgment strategy of this sub-state is the strategy to determine the action according to the immediate reward value (π i ). Starting from the first layer of sub-states, the action strategy is determined according to the immediate reward value (π i ) to simulate and guide the agent to complete the environmental task, thereby obtaining the state value of the first-level sub-state, as shown in the following formula:

[0099]

[0100] The state value V(s) of the first layer sub-state t+1 ) The strategy for determining actions based on immediate rewards (π i ) is used for simulation, which is the multi-layer value simulation algorithm when m=1.

[0101] The second solution is to follow the greedy strategy again (π g ) selects the action with the largest action value in turn and guides the agent to simulate until the agent completes the environment task, thereby obtaining the state value of the first-level sub-state, as shown in the following formula:

[0102]

[0103] In the second scheme, it is necessary to calculate the state value V(s) of the next layer of sub-states in any state when starting the path simulation from the first layer of sub-states. t+2 ), the next layer of sub-states is the state corresponding to the execution of one step of action in any state. Similar to the first layer of sub-states, two different strategies can also be used to calculate the state value of the next layer of sub-states of any state in the first layer of sub-state path simulation. The first strategy is to use the immediate reward value to determine the action strategy (π i ), that is, the strategy of selecting the maximum immediate reward value to determine the action (π i ) performs simulation path calculation of the state value of the next layer of sub-states of any state in the first layer of sub-state path simulation, and then performs simulation according to the state value of the next layer of sub-states, thereby obtaining the state value of the first layer of sub-states. Fig.15 As shown, combined Fig.14 The current state shown in the middle left picture, Fig.15The state highlighted in blue on the first figure is a state in the first-level sub-state of the current state. As can be seen from the figure below the first figure, the action space of the first-level sub-state includes actions 5 and 6. The first step in the path based on which the state value of the first-level sub-state is calculated is to select the action with the largest action value. Therefore, the action values ​​of actions 5 and 6 need to be calculated. Further, the state value of the corresponding sub-state after actions 5 and 6 needs to be calculated. The state value of the corresponding sub-state is determined by simulating the tool path using the immediate reward value starting from the sub-state, and then calculating π based on the simulated path. i ,like Fig.15 As shown in the two figures on the right in the middle box; after calculation, the sum of the sub-state value after taking action 5 from the first-level sub-state plus the immediate reward value obtained from action 5 is the largest, that is, the action value of taking action 5 from the first-level sub-state is the largest, so action 5 is selected as the first step in the path based on which the state value of the first-level sub-state is calculated. When determining the second step, the action is also selected based on the action value. The state value determination strategy of the next-level sub-state based on the action value is the strategy of determining the action based on the immediate reward value. This process is repeated to simulate the path corresponding to the first-level sub-state, as shown in Fig.15 As shown in the figure below, the state value of the first layer of sub-states can be obtained. This scheme is also the multi-layer value simulation algorithm when m=2.

[0104] The second strategy is to use the greedy strategy again (π g ) selects the action with the largest action value, but this requires calculating the state value of the next layer of sub-states of any state in the next layer of sub-state path simulation corresponding to any state in the first layer of sub-state path simulation.

[0105] The above is the main process of the multi-layer value simulation algorithm. Multi-layer refers to starting from a certain layer of sub-state and using the immediate reward value to determine the action strategy (π i ) simulates the path to calculate the value. That is, the m-layer value simulation algorithm is:

[0106] When m=0, the action with the largest immediate reward value is selected from the starting state to form the tool path, that is, in the current state, the action with the largest immediate reward value is directly executed;

[0107] When m=1, the action with the largest action value in the current state is selected in sequence from the starting state to form a tool path, wherein the action value of the action in the current state is calculated based on the state value of the first-level sub-state of the current state, and the tool path based on which the state value of the first-level sub-state of the current state is calculated is the tool path determined by simulation using the immediate reward value;

[0108] When m=n (n>1), the actions with the largest action value in the current state are selected in sequence from the starting state to form a tool path, wherein the action value of the action in the current state is calculated based on the state value of the first-level sub-state of the current state, and the tool path used for calculating the state value of the first-level sub-state of the current state is the tool path determined by solving the n-1-level value simulation algorithm, and any action in the tool path corresponding to the state value of the first-level sub-state of the current state is selected based on the action value, and the tool path used for calculating the state value of the next-level sub-state on which any action is based is the tool path determined by solving the n-2-level value simulation algorithm, and this process is repeated recursively until the tool path used for calculating the state value of the sub-state is the tool path determined by solving the 0-level value simulation algorithm, that is, the tool path determined by simulation using the immediate reward value.

[0109] Specifically, let m=2. At this time, the two-layer value simulation algorithm is as follows:

[0110] From the starting state, the actions with the largest action value in the current state are selected in sequence to form a tool path, wherein the tool path based on which the state value of the first-level sub-state of the current state is calculated is the tool path determined by solving the 1-level value simulation algorithm, and any action in the tool path corresponding to the state value of the first-level sub-state of the current state is selected based on the action value, and the tool path based on which the state value of the next-level sub-state on which any action is selected is the tool path determined by solving the 0-level value simulation algorithm, that is, the tool path determined by simulation using the immediate reward value.

[0111] Furthermore, when m=3, the three-layer value simulation algorithm is specifically as follows:

[0112] From the starting state, actions with the largest action value in the current state are selected in sequence to form a tool path, wherein the tool path based on which the state value of the first-level sub-state of the current state is calculated is the tool path determined by solving the 2-level value simulation algorithm, and any action in the tool path corresponding to the state value of the first-level sub-state of the current state is selected based on the action value, and the tool path based on which the state value of the next-level sub-state based on which any action is selected is the tool path determined by solving the 1-level value simulation algorithm;

[0113] Define that the next sub-state on which any action in the tool path corresponding to the state value of the first-level sub-state of the current state is based is the sub-state corresponding to the state value path of the first-level sub-state, and any action in the tool path corresponding to the state value of the sub-state corresponding to the state value path of the first-level sub-state is selected based on the action value. The tool path used for calculating the state value of the next-level sub-state on which any action is based is the tool path determined by the 0-level value simulation algorithm, that is, the tool path determined by the instant reward value simulation.

[0114] In the multi-layer value simulation algorithm, it is necessary to set the strategy (π i ), which is m in the algorithm. When m is small, the estimate of the state value will deviate from the value of the optimal strategy, but fewer layers of sub-state simulation will reduce the algorithm running time; when m increases, there will be more layers of sub-state simulation, the calculation time will increase significantly, but the value estimate will be closer to the state value under the optimal strategy. Fig.16 The experimental results of different m values ​​in a small discrete grid of 9*9 are shown. It can be seen that when m increases, especially when m changes from 2 to 3, the calculation time is significantly improved. When m changes from 0 to 1, the cumulative reward value obtained by the agent is significantly improved. When m is greater than 1, the cumulative reward value obtained by the agent tends to be stable and difficult to improve. Therefore, in the multi-layer value simulation algorithm, m=2 is set so that the value of the state can be close to the state value under the optimal strategy while ensuring the computational efficiency.

[0115] The pseudo code of the multi-layer value simulation algorithm with m=2 is shown in the following table. In order to obtain accurate state value during simulation, future reward value must be taken seriously, so the reward discount factor γ is set to 1.0.

[0116]

[0117] In this embodiment, an improved version of the Monte Carlo method, the multi-layer value simulation algorithm, is used to solve the tool path generation problem. This method can quickly and accurately calculate the value of a state, thereby guiding the intelligent agent to complete the reinforcement learning task; compared with the Monte Carlo algorithm and the temporal difference algorithm, this method avoids the instability caused by random sampling; compared with the dynamic programming algorithm, this method reduces the value of the state that will not appear in the calculation process. And this method does not store a large amount of state value, does not require a large amount of memory space and complex search operations on the environment.

[0118] 4. Beneficial effects of this embodiment

[0119] In order to verify the effectiveness of the strategy-free milling tool path generation method proposed in this embodiment, the following Fig.17The cavity workpiece shown in the figure is simulated, and the simulation experiment parameters are shown in Table 2. First, the part is layered by layered milling, and each layer is converted into a mesh model; then the tool path is generated based on the mesh model using a multi-layer value simulation algorithm.

[0120] The tool paths generated by the method proposed in this embodiment are compared with the tool paths generated by the three commonly used cutting modes (reciprocating, following parts, and following periphery) in NX to prove the effectiveness and advancement of the method proposed in this embodiment. The tool paths generated by the four methods are simulated in the cutting simulation software CIMCO Edit 8.02.23. The cutting results are shown in Figure 2. Fig.18 As shown, Fig.18 (a) is the strategy-free milling tool path generation method provided in this embodiment, (b) is the NX reciprocating cutting mode, (c) is the NX follow-part cutting mode, and (d) is the NX follow-periphery cutting mode. It can be observed that the tool paths generated by the four methods can complete the milling of the target parts. The cutting simulation software CIMCO Edit8.02.23 is used to count the processing time, cutting time, trajectory length and cutting distance of the tool paths generated by different methods. The statistical results are shown in Table 3, where the improvement ratio is calculated based on the processing time. It can be seen that while keeping the cutting conditions such as part model, machining allowance, feed speed, cutting width, tool diameter, etc. consistent, the tool path generated by the strategy-free milling tool path generation method proposed in this embodiment has a shorter cutting distance and trajectory length, less redundant cutting, and higher processing efficiency. Since there is no tool path generation strategy at the geometric level, when generating the path, the tool does not need to be restricted by the given strategy, but can choose a reasonable movement direction according to the current state of the milling workpiece and the motion state of the tool, so the generated tool path has fewer redundant paths and higher processing efficiency.

[0121] Table 2 Simulation experiment parameters

[0122]

[0123] Table 3 Simulation results

[0124]

[0125] Furthermore, the second embodiment of the present invention provides a strategy-free milling tool path generation system, which is used to implement any of the methods described above. The system can be understood by reference to the above methods, and the system includes: a grid establishment module, which is used to obtain a grid model of a part to be processed and determine the area to be processed on the grid model, and use the grid center point as the tool path point to convert the tool path generation problem of the area to be processed into a tool path point access sequence problem, thereby converting it into a reinforcement learning task; a model establishment module, which is used to establish a Markov decision process model (MDP model) of the reinforcement learning task, wherein the intelligent agent in the MDP model is a tool, and the environment is the state of the grid center point in the grid model. The state space of the tool in the environment and the corresponding action space are defined, and an immediate reward function for the tool action is set. After the tool performs an action in the action space corresponding to the state space, the environment and the state space change and a corresponding immediate reward value can be obtained according to the immediate reward function; a model solving module, which is used to solve the MDP model based on the immediate reward value of the tool action to generate a tool path for the part to be processed.

[0126] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A strategy-free milling tool path generation method, characterized in that: include: S1, obtaining a mesh model of a part to be processed and determining a region to be processed on the mesh model, taking a mesh center point as a tool path point, converting a tool path generation problem of the region to be processed into a tool path point access sequence problem, thereby converting it into a reinforcement learning task; S2, establish a Markov decision process model (MDP model) of the reinforcement learning task, wherein the intelligent agent in the MDP model is the tool, the environment is the state of the center point of the grid in the grid model, define the state space of the tool in the environment and the corresponding action space, set the immediate reward function of the tool action, after the tool performs an action according to the action space corresponding to the state space, the environment and the state space change and can obtain a corresponding immediate reward value according to the immediate reward function; S3, solving the MDP model based on the instant reward value of the tool action to generate a tool path for the part to be processed.

2. The strategy-free milling tool path generation method according to claim 1, characterized in that: S1 specifically includes: Dividing the to-be-processed part into a plurality of to-be-processed layers by means of layered milling; Acquire a mesh model of any of the layers to be processed, determine the contour of any of the layers to be processed according to a preset processing allowance and a tool radius, and determine a processing area of ​​any of the layers to be processed based on the contour; The grid center point contained in the area to be processed is used as the tool path point, so as to transform the tool path generation problem of any layer to be processed into a node access sequence problem on the grid plane.

3. The strategy-free milling tool path generation method according to claim 1, characterized in that: The state space S of the tool in the environment is defined in S2 as: S=[d,s0,s1,s2,s3,s4,s5,s6,s7,x,y]; Where d represents the current movement direction of the tool; s 0～7 Represents the environmental state information in eight adjacent directions around the discrete tool path point where the tool is located; x and y represent the horizontal coordinate and vertical coordinate of the discrete tool path point where the tool is located, respectively; The action space A of the tool is: A＝[0,1,2,3,4,5,6,7,8]； Where 0 to 7 represent the movement from the current tool path point to the adjacent tool path points in 8 directions, and 8 represents the tool lifting and feeding action.

4. The strategy-free milling tool path generation method according to claim 1, characterized in that: The instantaneous reward function R(s,a) of the tool action is specifically the sum of the reward function R1(s,a) of the feed direction change, the reward function R2(s,a) of the tool lifting and feeding action, and the reward function R3(s,a) of the tool movement; Among them, the reward function for changing the feed direction gives the tool a negative reward value according to the angle of the feed direction change, so as to limit the number and angle of the tool feed direction change; the reward function for lifting the tool for feeding gives the tool a negative reward value according to the tool lifting for feeding, so as to limit the number of times the tool is lifted for feeding; the reward function for tool movement is: when the tool performs effective cutting, a positive reward value is given to the tool, and when the tool movement produces empty stroke, a negative reward value is given to the tool, so as to reduce the empty stroke of the tool.

5. The strategy-free milling tool path generation method according to claim 4, characterized in that: The reward function for changing the feeding direction is obtained in the following way: A processing experiment was conducted to measure the time cost of changing the feed direction caused by different feed direction angle changes at different feed speeds, where the time cost of changing the feed direction is the difference between the processing time when the feed direction is changed and the processing time when the feed direction remains unchanged; Fitting the experimental data to obtain the relationship between the time cost of changing the feeding direction and the feeding speed and the change in the feeding direction angle; According to the relationship between the time cost of changing the feeding direction and the feeding speed and the change in the feeding direction angle, the reward function of changing the feeding direction is obtained.

6. The strategy-free milling tool path generation method according to claim 5, characterized in that: The immediate reward function of the tool action is specifically: R(s,a)=R1(s,a)+R2(s,a)+R3(s,a); R1(s,a)=-0.1*f(v,ω); In the formula, v represents the feed speed, ω represents the change in feed direction angle, f(v,ω) represents the time cost of changing the feed direction caused by different feed direction angle changes at different feed speeds, a≠8 means that the tool lifting and feeding action is not performed, and a=8 means that the tool lifting and feeding action is performed.

7. The strategy-free milling tool path generation method according to claim 1, characterized in that: The MDP model also includes: Set the state transition probability to 1; And / or, a maximum number of decisions of the agent is set. When the actual number of decisions of the agent reaches the maximum number of decisions, if the agent has not traversed all tool path points at this time, this round of exploration is terminated in advance to prevent the agent from falling into an infinite loop during the interaction with the environment.

8. The strategy-free milling tool path generation method according to claim 1, characterized in that: S3 specifically includes: solving the MDP model using a multi-layer value simulation algorithm to obtain a decision sequence of the intelligent agent, thereby obtaining a tool path of the part to be processed; Specifically, it is defined that when the tool is in any state, the state value of any state is calculated according to the sum of the immediate reward values ​​of all actions on the tool path formed from any state as the starting point; the action value of the tool performing any action in the current state is the sum of the immediate reward value of any action and the state value of the next layer of sub-state corresponding to the execution of any action from the current state; the first layer of sub-state of the current state is the state set after executing one feasible action from the current state; The m-layer value simulation algorithm is: When m = 0, the actions with the largest immediate reward value are selected from the starting state to form the tool path; When m=1, the action with the largest action value in the current state is selected in sequence from the starting state to form a tool path, wherein the action value of the action in the current state is calculated based on the state value of the first-level sub-state of the current state, and the tool path based on which the state value of the first-level sub-state of the current state is calculated is the tool path determined by simulation using the immediate reward value; When m=n (n>1), the actions with the largest action value in the current state are selected in sequence from the starting state to form a tool path, wherein the action value of the action in the current state is calculated based on the state value of the first-level sub-state of the current state, and the tool path used for calculating the state value of the first-level sub-state of the current state is the tool path determined by solving the n-1-level value simulation algorithm, and any action in the tool path corresponding to the state value of the first-level sub-state of the current state is selected based on the action value, and the tool path used for calculating the state value of the next-level sub-state on which any action is based is the tool path determined by solving the n-2-level value simulation algorithm, and this process is repeated recursively until the tool path used for calculating the state value of the sub-state is the tool path determined by solving the 0-level value simulation algorithm, that is, the tool path determined by simulation using the immediate reward value.

9. The strategy-free milling tool path generation method according to claim 8, characterized in that: Let m = 2. At this time, the two-layer value simulation algorithm is as follows: From the starting state, the actions with the largest action value in the current state are selected in sequence to form a tool path, wherein the tool path based on which the state value of the first-level sub-state of the current state is calculated is the tool path determined by solving the 1-level value simulation algorithm, and any action in the tool path corresponding to the state value of the first-level sub-state of the current state is selected based on the action value, and the tool path based on which the state value of the next-level sub-state on which any action is selected is the tool path determined by solving the 0-level value simulation algorithm, that is, the tool path determined by simulation using the immediate reward value.

10. A strategy-free milling tool path generation system, characterized in that: include: A mesh building module is used to obtain a mesh model of the part to be processed and determine the area to be processed on the mesh model, using the center point of the mesh as the tool path point, and converting the tool path generation problem of the area to be processed into a tool path point access sequence problem, thereby converting it into a reinforcement learning task; A model building module is used to build a Markov decision process model (MDP model) of a reinforcement learning task, wherein the intelligent agent in the MDP model is a tool, the environment is the state of the center point of the grid in the grid model, the state space of the tool in the environment and the corresponding action space are defined, and an immediate reward function of the tool action is set. After the tool performs an action according to the action space corresponding to the state space, the environment and the state space change and a corresponding immediate reward value can be obtained according to the immediate reward function; The model solving module is used to solve the MDP model based on the instant reward value of the tool action and generate a tool path for the part to be processed.

Images