A numerical control machining tool shaft vector motion trajectory planning method based on Q-learning

Abstract

The application discloses a kind of Q-learning-based numerical control processing tool shaft vector motion trajectory planning methods, comprising: by C space expression technology, tool shaft vector is compressed from three dimensions to two dimensions, and grid division is carried out, based on no interference, no singular machining requirements distinguish feasible domain and forbidden domain;Using Q-learning algorithm combines reward feedback mechanism, searches optimal tool shaft path in feasible domain, and through state, action space definition and reward function design, gradually improve the smoothness of path and the stability of machine tool motion;Reward function comprehensively considers path safety, tool kinematics characteristics and tool shaft vector change amplitude and other factors, to strengthen the optimization planning of tool shaft trajectory;Finally, realize the global optimization of tool shaft vector path through Q value iteration update.The application effectively solves the calculation complexity and local optimal problem of tool posture optimization under complex working conditions, significantly improves the adaptability and efficiency of five-axis machining path planning.

Description

Technical Field

[0001] The present invention relates to the technical field of numerical control machining, and in particular to a method for planning a vector motion trajectory of a numerical control machining tool axis based on Q-learning. Background Art

[0002] In five-axis CNC machining, the smoothness of tool posture directly impacts process stability and quality. Smooth tool posture changes not only prevent machine tool overload and vibration caused by sudden posture changes, but also significantly improve the quality of the machined surface, thereby enhancing overall machining efficiency and accuracy. Maintaining smooth tool posture changes is particularly crucial when machining complex curved surfaces. This ensures optimal contact between the tool and the workpiece, preventing degradation of machining quality or equipment wear caused by drastic posture changes.

[0003] At present, there are two main mainstream methods for tool posture smoothness planning: the adjustment method based on preset rules and the global planning method based on the feasible domain.

[0004] For example, Chinese patent document with publication number CN107335847A discloses a processing method for constraining tool posture by cutting efficiency; Chinese patent document with publication number CN115857429A discloses a method for generating a smooth path for a tool of a five-axis machine tool.

[0005] Adjustment methods based on preset rules pre-generate tool postures according to certain constraints, ensuring no interference or anomalies, and then combine local optimization to achieve posture adjustment. This method is simple and intuitive, but it cannot achieve global optimization and requires multiple adjustments for different workpieces, resulting in low efficiency and poor adaptability.

[0006] Global planning methods based on a feasible region are based on a feasible region of tool pose angles without interference or singularity at each tool contact point, achieving global smoothing optimization within this feasible region. In theory, this approach is highly universal, allowing for the selection of additional optimization objectives (such as cutting forces or dynamic characteristics) without the need for subsequent adjustments. However, its computational complexity leads to lengthy planning times, limiting its practical application.

[0007] In order to solve performance issues, heuristic algorithms are widely used in such optimization problems. However, heuristic algorithms often face limitations such as local optimal traps, dependence on domain knowledge, and poor adaptability when solving complex optimization problems, making them difficult to expand in high-dimensional and uncertain environments.

[0008] In contrast, reinforcement learning effectively balances exploration and exploitation through a reward mechanism, has stronger adaptability, and does not rely on explicit domain knowledge, so it can more flexibly cope with dynamically changing environments. This makes reinforcement learning exhibit strong robustness and global search capability in complex optimization problems, becoming an effective alternative to traditional heuristic algorithms. SUMMARY

[0009] To solve the problems in the background art, the application provides a Q-learning-based numerical control machining tool axis vector motion trajectory planning method, which can efficiently explore the optimal trajectory of the tool axis vector, thereby achieving global optimization in complex machining scenarios and providing a new solution for tool posture smoothness planning in five-axis numerical control machining.

[0010] A Q-learning-based numerical control machining tool axis vector motion trajectory planning method, comprising the following steps:

[0011] S1, by C space representation technology, the tool axis vector is compressed from three-dimensional space to two-dimensional polar coordinate space, and is grid divided;

[0012] S2, the grid after grid division is state discriminated, and is divided into feasible region and forbidden region;

[0013] S3, a reinforcement learning model is constructed; wherein the state space is defined as the grid set of the tool contact point feasible region, the action space is defined as the action set of the agent from the current grid point to all grid points in the next feasible region, and the reward function comprehensively considers the safety of the path, the kinematics characteristics of the tool and the change amplitude of the tool axis vector;

[0014] S4, a Q-learning algorithm is used in combination with value iteration update, and the optimal tool axis vector motion trajectory is searched through interaction with the environment and reward feedback.

[0015] In step S1, the tool axis vector is compressed from three-dimensional space to two-dimensional polar coordinate space by C space representation technology, specifically:

[0016] In the machine tool machining process, the tool axis vector is represented by a three-dimensional vector (i, j, k) in the workpiece coordinate system, and when the tool axis vector is mapped to the polar coordinate form, only two angle parameters (α, β) are used., wherein:

[0017]

[0018] In the formula, α represents the direction angle of the tool axis vector in the horizontal plane, and β represents the included angle of the tool axis vector with the vertical direction.

[0019] In step S1, the specific process of grid division is:

[0020] The two-dimensional polar coordinate space (a, b) is discretized at a certain resolution, a is divided into n α intervals, and b is divided into n β intervals; each grid center point corresponds to a discrete tool axis vector (a i , b i ), and these grid points constitute the state space of the tool posture.

[0021] In step S2, for each grid point of the state space, it is determined whether it satisfies the constraints of no interference and no singularity, and is marked, and the grid points satisfying the constraints of no interference and no singularity are divided into a feasible region, and the remaining grid points are divided into a forbidden region.

[0022] In step S3, when defining the state space, the feasible region grid set of the tool contact point is taken as the state space, where the feasible region grid of each tool contact point is described by two discretized variables a and b, representing the set of inclination angle and azimuth angle; the state space S is defined as the grid pair of adjacent tool contact point feasible regions, represented as:

[0023]

[0024] where k represents the current tool contact point number, S k represents the kth state, S k is represented as:

[0025] S k = {(a k , b k ), (a k+1 , b k+1 )}

[0026] wherein, is the feasible region grid corresponding to the kth tool contact point, (a k+1 , b k+1 ) is the feasible region grid corresponding to the k+1th tool contact point.

[0027] In step S3, the action space is defined as the set of all possible actions of the agent from the current grid point to all grid points in the next feasible region:

[0028] A k = {(a k+1 , b k+1 ) e feasible region}.

[0029] In step S3, the reward function is defined as follows:

[0030]

[0031] where r t(s, a) represents the immediate reward obtained by transferring to the next state s from the selected action a at time t, the negative of the Euclidean distance between the grid points of adjacent feasible regions, as the reward value.

[0032] The specific process of step S4 is:

[0033] Initialization parameters: randomly initialize the Q value table Q(S, A); the learning rate a determines the influence degree of new information on Q value update, and is taken as 0.1; the discount factor γ determines the importance of future rewards, and is taken as 0.9;

[0034] Selection of the starting state of the agent: randomly select a feasible grid pair of the k-1th and kth contact points as the starting point S k ;

[0035] Action exploration strategy: in order to balance exploration and utilization, the ε-greedy exploration strategy is adopted, and the agent selects the action according to the following rules under the current state S k :

[0036]

[0037] Wherein, ε is taken as 0.1, indicating the exploration probability; A k represents the set of all available actions under the current state S k , and Q(S k , A') represents the Q value of the action A' under the state S

[0038] State transition and reward calculation: according to the selected action a, transfer to the next state S', and obtain the immediate reward r(S, a) at the same time;

[0039] Q value table update:

[0040]

[0041] In the formula, Q(s, a) is the Q value of the action a under the current state s; r is the immediate reward obtained by taking the action a under the current state s; γ is the discount factor, which determines the weight of future rewards; is the maximum Q value of all possible actions in the next state s'; a is the learning rate, which determines the influence degree of new information on Q value update;

[0042] Path generation: starting from the starting state, the agent selects actions according to the current Q value table to generate a continuous grid point sequence between the contact points; with the iterative update of the Q value table, the accuracy and optimization effect of path planning are gradually improved, and finally the optimal tool axis vector path meeting the smoothness and safety requirements is obtained; when the update amplitude of the Q value table is lower than the predetermined threshold, the iteration is stopped, and the optimal path result is output.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] (1) The present invention significantly reduces the complexity of the path planning problem through C-space compression and grid discretization technology.

[0045] (2) The present invention adopts a reinforcement learning path planning algorithm based on Q-learning. By balancing exploration and utilization, it effectively avoids the defect of traditional heuristic algorithms that are prone to falling into local optimality, and ensures the global optimality of tool axis vector planning.

[0046] (3) The present invention comprehensively considers the path smoothness, safety and kinematic characteristics of the machine tool, and ensures the smoothness and stability of the tool movement by designing the corresponding reward function, thereby improving the processing quality and equipment operation reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a flow chart of path planning using the Q-learning algorithm in the present invention.

[0048] Figure 2 It is a two-dimensional discrete grid representation of the feasible domain.

[0049] Figure 3 This is a graph showing the changes in total rewards during Qleaning training.

[0050] Figure 4 Schematic diagram of tool axis vector path planning in four feasible regions. DETAILED DESCRIPTION

[0051] The present invention will be described in further detail below with reference to the accompanying drawings and examples. It should be noted that the following examples are intended to facilitate understanding of the present invention and do not have any limiting effect on the present invention.

[0052] The present invention compresses the tool axis vector from three dimensions to two dimensions through C-space expression technology, and divides the two-dimensional space into grids. Based on processing requirements such as no interference and no singularity, the grid is state-judged to distinguish between feasible domains and forbidden domains. Through C-space compression and grid discretization, the complexity of the path planning problem is greatly reduced. During the planning process, the Q-learning algorithm searches for the optimal tool axis path in the feasible domain and combines the reward feedback of the current grid state to strengthen the mapping relationship between the environmental state and the path planning. The algorithm gradually improves the smoothness and stability of the tool vector path in iterative learning, ensures that the path planning meets the smoothness, and thus realizes the optimal tool axis trajectory planning under complex working conditions. The specific technical solutions adopted by the present invention are as follows:

[0053] S1, C space compression and grid division

[0054] S1.1, C-space representation of 3D tool axis vector

[0055] In the process of machine tool machining, the tool axis vector can be represented by a three-dimensional vector (i, j, k) in the workpiece coordinate system. In order to simplify the calculation, the tool axis vector is mapped to the polar coordinate form, that is, only two angle parameters (a, b) are used, where:

[0056]

[0057] In the formula, a represents the directional angle of the tool axis vector in the horizontal plane, and b represents the angle between the tool axis vector and the vertical direction.

[0058] S1.2, meshing

[0059] The two-dimensional polar coordinate space (a, b) is discretized according to a certain resolution, a is divided into n α intervals, and b is divided into n β intervals. The two-dimensional grid is numbered from the lower left corner, using the row-by-row numbering rule, that is, numbered from left to right and from bottom to top. The grid G i,j represents the jth grid in the feasible region of the ith tool contact point. Each grid can be represented by a grid center point, and the grid center point corresponds to a discrete tool axis vector (a i , b i ). These grid center points constitute the state space of the tool posture.

[0060] S1.3, discrimination of feasible region and forbidden region

[0061] For each grid, it is judged whether it meets the constraints of no interference and no singularity. Based on the following assumptions: if the tool axis vectors represented by the four vertices of a grid all meet the no interference and no singularity conditions, the grid is feasible, and is marked as a feasible region, as shown in Figure 2 .

[0062] S2, creation of reinforcement learning model

[0063] S2.1, definition of state space

[0064] The core of reinforcement learning is that the agent learns the rules of the environment and optimizes the decision through interaction with the environment. In the task of tool axis vector path planning, the goal of the agent is to plan a tool axis vector path for a set of continuous tool contact points, and the environment corresponds to the entire machining process. Let the set of continuous n tool contact points be CL = {CL1, CL2, …, CL n}, where each tool contact point CL i = (x i , y i , z i ) represents the position coordinates of the tool center. Corresponding to each tool contact point CLi The feasible region of its tool pose can be defined as CO i is represented by a discretized grid. Therefore, the state space can be represented as a set of tool contact point feasible regions CO = {CO1, CO2, …, CO n}.

[0065] To facilitate the fast learning and decision-making of the agent and reduce the complexity of the path planning problem, each feasible region CO i is divided into a grid, and the state space is converted into a finite and discrete form. In addition, to introduce the acceleration constraint, the state input s is defined as a pair of adjacent tool contact point feasible regions (CO i-1 , CO i ), that is, s = (CO i-1 , CO i ). Through this design, the agent can comprehensively consider the dynamic constraints between the current tool contact point and the adjacent tool contact point, thereby achieving the optimization planning of the tool axis vector path.

[0066] In this embodiment, the grid set of the tool contact point feasible region is taken as the state space, where each tool contact point feasible region grid is described by two discrete variables a and β, representing the set of inclination angles and azimuth angles, respectively. The state space can be defined as a pair of adjacent tool contact point grids, denoted as:

[0067]

[0068] where k represents the current tool contact point number, S k represents the kth state, and S k is represented as:

[0069] S k = {(a k , β k ), (a k+1 , β k+1 )}

[0070] wherein is the feasible region grid corresponding to the kth tool contact point, and (a k+1 , β k+1 ) is the feasible region grid corresponding to the k+1th tool contact point.

[0071] S2.2, Action space definition

[0072] The action space is defined as the set of all possible grids in the next feasible region that the agent can choose after selecting a grid in the current feasible region. Specifically, the action space describes the target grids that the agent can reach from the current state (i.e., the current grid), and these target grids belong to the next feasible region.

[0073] Through this design, the agent can explore different paths in each decision-making step, thereby achieving optimal planning of the tool axis vector. The action space can be represented as:

[0074]

[0075] where CO = {CO1, CO2, …, CO n} represents the set of all feasible regions, g i-1 and g i represent the grid points in the feasible region pair (CO i-1 , CO i ) corresponding to the adjacent two tool contact points for any state input s, and g i+1 represents the grid point in the next feasible region CO i+1 .

[0076] In this embodiment, the action space is defined as the set of actions that the agent can take from the current grid point to the next grid point:

[0077] A k = {(α k+1 , β k+1 ) ∈ feasible region}

[0078] S2.3, Reward function design

[0079] In the reinforcement learning model, the reward function is a core component that directly determines the learning direction of the agent and the final optimization effect. The design of the reward function needs to consider the constraints and optimization objectives comprehensively to ensure the safety, feasibility, and smoothness of the path planning.

[0080] S2.3.1, Ensure path safety

[0081] In numerical control machining, once the path enters the forbidden domain, it may cause tool collision or singularity problems, causing uncontrollable and even irreversible effects on the machining process. Therefore, the planned tool axis vector trajectory must always be within the feasible region.

[0082] The reward function is designed as follows:

[0083] When the path enters the forbidden domain, a large negative reward is given:

[0084] R 域 = -100

[0085] Through this negative reward design, the reinforcement learning model can effectively avoid dangerous paths and ensure the safety of the tool axis trajectory.

[0086] S2.3.2, Meet the kinematics characteristics of each axis of the machine tool

[0087] The feasible region of tool axis vector is defined in the workpiece coordinate system, but its mapping from the workpiece coordinate system to the machine tool coordinate system involves a nonlinear transformation. This makes the smoothness in the workpiece coordinate system unable to fully reflect the smoothness of the rotation angle in the machine tool coordinate system. Therefore, it is necessary to constrain the rotation angular velocity and acceleration of each axis in the machine tool coordinate system in the reward function. For the tool axis vector in the workpiece coordinate system selected by the current state s and the tool axis vector selected by the next action a According to the inverse kinematics IK, it is transformed into the tool axis vector in the machine tool coordinate system Then the central difference method is used to obtain the angular velocity and angular acceleration expressions of the A and C rotation angles:

[0088]

[0089] where f is the feed speed, which is generally a constant value during machining; L is the distance between adjacent tool contact points. In order to meet the performance requirements of the machine tool, the rotation angular velocity and angular acceleration need to be limited in the following intervals:

[0090]

[0091] where * represents the A or C axis.

[0092] The reward function is designed as follows:

[0093] If the next action leads to angular velocity or angular acceleration exceeding the predetermined limit interval, a large negative reward is given:

[0094] R 动 = -100

[0095] Through this design, the agent can preferentially select the tool axis vector that meets the kinematic constraints during path planning, thereby avoiding machine tool overload or unstable motion caused by over-limit.

[0096] S2.3.3, ensure the smoothness of the tool axis vector trajectory

[0097] The change amplitude of the tool axis vector between adjacent tool position points is an important indicator for evaluating the smoothness of the path. Excessive change amplitude may cause trajectory discontinuity, affecting the machining quality and efficiency. Therefore, in order to ensure the smoothness of the machining path, it is necessary to minimize the change of the tool axis vector between adjacent tool position points.

[0098] The negative value of the Euclidean distance between adjacent feasible region grid points is used as an index to measure the magnitude of the tool axis vector change, when the Euclidean distance between two adjacent tool position points is small (i.e. the magnitude of the tool axis vector change is small), a higher positive reward can be given to encourage the agent to choose a smooth path. Conversely, if the Euclidean distance is large, a larger negative reward is given, so as to avoid the path from appearing mutation. Let the coordinates of two adjacent feasible region grid points be g i =(x i ,y i ) and g i+1 =(x i+1 ,y i+1 ).

[0099] The reward function is designed as follows:

[0100] The negative number of the Euclidean distance between adjacent feasible region grid points is used as the reward value:

[0101]

[0102] S2.3.4, reward function definition summary

[0103]

[0104] S3, path planning using Q-learning algorithm, as Figure 1 shown.

[0105] S3.1, initialization parameters

[0106] Randomly initialize the Q value table Q(S, A) to 0 for easy calculation; the learning rate alpha determines the influence of new information on Q value update, and is taken as 0.1; the discount factor gamma determines the importance of future rewards, and is taken as 0.9.

[0107] S3.2, decision-making process of agent

[0108] S3.2.1, starting state selection

[0109] Randomly select the feasible grid pair of the k-1th and kth tool contact points as the starting point S k .

[0110] S3.2.2, action exploration strategy

[0111] In order to balance exploration and utilization, the epsilon-greedy exploration strategy is adopted, and the agent selects action under the current state S k according to the following rules:

[0112]

[0113] where ε is 0.1, representing the exploration probability; A k represents the set of all available actions in the current state, Q(S k , A') represents the Q value of action A' in state S k .

[0114] S3.2.3, State transition and reward calculation

[0115] According to the selected action a, the next state S' is reached, and the immediate reward r(S, a) is obtained.

[0116] S3.2.4, Q value table update

[0117] In the process of reinforcement learning, the agent continuously updates the Q value table through continuous interaction with the environment to optimize path planning decisions. The core of Q-learning is to learn how to choose actions through the following update formula and iteratively adjust the Q value table:

[0118]

[0119] In the formula, Q(s, a) is the Q value of action a in the current state s; r is the immediate reward obtained by taking action a in the current state s; γ is the discount factor, which determines the weight of future rewards; is the maximum Q value of all possible actions in the next state s'; α is the learning rate, which determines the degree of influence of new information on Q value update.

[0120] S3.2.5, Action selection

[0121] In the process of Q value update, the agent needs to balance the relationship between exploration and exploitation. Exploitation means that the agent always chooses the action with the maximum Q value, completely based on existing experience to optimize the path, with high learning efficiency, but easy to fall into local optimum. Exploration means that the agent always randomly selects actions, with low learning efficiency, but can ensure global optimization.

[0122] In order to balance exploration and exploitation, many strategies have been proposed.

[0123] S3.2.6, Iterative update of Q value table until convergence

[0124] Q-learning updates the Q value table through repeated iterations to optimize the strategy until the change in Q value meets the convergence condition. In each iteration, the agent selects an action based on the current state and obtains feedback through interaction with the environment, constantly correcting the value, thereby optimizing path planning. With the progress of iteration, the accuracy and stability of path planning gradually improve. Finally, the agent will find the optimal tool axis trajectory that meets the optimization goal within the feasible region.

[0125] To verify the effect of the present application, in the present example, four consecutive feasible domain grids are selected for path planning, simulating the complex tool posture planning task in five-axis NC machining. In order to more clearly verify the tool axis vector path optimization effect based on the Q-learning algorithm, the following experimental conditions and parameter settings are designed.

[0126] (1) Experimental conditions and parameter settings

[0127] Grid division

[0128] The two-dimensional polar coordinate space (a, b) of the tool axis vector after C space compression is divided into a 10x10 grid, and the center point of each grid corresponds to a discrete tool axis vector. Four feasible domains define the grid range under the requirements of no interference and no singular machining, and mark the feasible grid points inside. The agent plans the optimal trajectory of the tool from the initial position to the target position in these feasible domains.

[0129] Reinforcement learning parameter settings

[0130] The number of iterations is 2000, the learning rate a = 0.1, which ensures that the agent can quickly absorb new environmental information; the discount factor g = 0.9, which gives a higher weight to future rewards to ensure the global optimality of the path; the action exploration strategy chooses the e-greedy exploration strategy, with an initial value of e = 1, which decays exponentially to 0.1 as the number of iterations increases, balancing exploration and exploitation.

[0131] (2) Experimental results and analysis

[0132] Figure 3 is the total reward value curve during the Q-learning exploration process, and the final result is shown in Figure 4 , which shows the optimal path of the tool between the four grids. The optimal path is [36, 36, 36, 66, 36], and the maximum total reward value is -6.

[0133] The experiment shows that in the initial 700 iterations, the agent is in the exploration phase, resulting in large fluctuations in the reward value. As the number of iterations increases, the agent gradually reduces the rotation of negative paths, and the reward value steadily improves. After 1000 iterations, it tends to be stable, indicating that the agent has found the globally optimal path.

[0134] The above-described embodiments detail the technical solutions and beneficial effects of the present application. It should be understood that the above-described embodiments are only specific embodiments of the present application and are not intended to limit the present application. Any modifications, supplements, and equivalent replacements made within the scope of the principles of the present application should be included within the protection scope of the present application.

Claims

1. A Q-learning based numerical control machining tool axis vector motion trajectory planning method, characterized in that, The method comprises the following steps: S1, by C space representation technique, the tool axis vector is compressed from three-dimensional space to two-dimensional polar coordinate space, and is meshed and divided; S2, the state space after meshing and dividing is judged, and is divided into feasible region and forbidden region; S3, a reinforcement learning model is constructed; wherein, the state space is defined as the grid set of the tool contact point feasible region, the action space is defined as the action set of the agent from the current grid point to all grid points in the next feasible region, and the reward function comprehensively considers the safety of the path, the kinematics characteristics of the tool and the change amplitude of the tool axis vector; When defining the state space, the set of feasible region grids of the tool contact points are taken as the state space, where each feasible region grid of the tool contact points is described by two discretized variables a and b, representing the set of the inclination angle and the azimuth angle, respectively; the state space defined as the grid pair of the adjacent tool contact point feasible regions, denoted as: where k denotes the current tool contact number, S k represents the kth state, S k is represented as: S k ＝{(α k ,β k ),(α k+1 ,β k+1 )} wherein (a k , b k ) is the feasible region grid corresponding to the kth tool contact point, and (a k+1 , b k+1 ) is the feasible region grid corresponding to the k+1th tool contact point; The reward function is defined as follows: where r t (s, a) denotes the immediate reward obtained by transitioning to the next state s from the selected action a at time t, denotes the negative of the Euclidean distance between the grid points of adjacent feasible regions, which is taken as the reward value; S4, a Q-learning algorithm is used in combination with value iteration update, and the optimal tool axis vector motion trajectory is searched through interaction with the environment and reward feedback; the specific process is as follows: Initialize parameters: randomly initialize the Q value table Q(S,A); the learning rate alpha determines the influence degree of new information on Q value update, and is 0.1; the discount factor gamma determines the importance of future rewards, and is 0.9; Selection of the initial state of the agent: randomly select a feasible grid pair of the (k-1)th and kth knife contact points as the starting point S k ; Action exploration strategy: To balance exploration and exploitation, an ε-greedy exploration strategy is adopted, in which the agent chooses an action a at a state S k Next, the agent selects an action a according to the following rule Wherein, ε takes the value of 0.1, representing the exploration probability; A k represents the set of all optional actions under the current state, Q(S k ,A') represents the Q value of the state S k action A'. State transition and reward calculation: according to the selected action a, the next state S' is transferred, and the immediate reward r(S,a) is obtained; Q value table update: In the formula, Q(s, a) is the Q value of taking action a in current state s; r is the immediate reward obtained by taking action a in current state s; γ is a discount factor, which determines the weight of future rewards; is the maximum Q value of all possible actions in the next state s'; and α is a learning rate, which determines the influence of new information on the update of the Q value. Path generation: starting from the initial state, the agent selects actions according to the current Q value table to generate a continuous grid point sequence between the tool contact points; with the iterative update of the Q value table, the accuracy and optimization effect of the path planning are gradually improved, and the optimal tool axis vector path meeting the smoothness and safety requirements is finally obtained; when the update amplitude of the Q value table is lower than the predetermined threshold, the iteration is stopped, and the optimal path result is output.

2. The Q-learning based CNC machining tool axis vector motion trajectory planning method according to claim 1, characterized in that, In step S1, the tool axis vector is compressed from three-dimensional space to two-dimensional polar coordinate space by C space representation technique, specifically: In the machine tool machining process, the tool axis vector is represented by a three-dimensional vector (i,j,k) in the workpiece coordinate system, and when the tool axis vector is mapped to the polar coordinate form, only two angle parameters (alpha, beta) are used, wherein: In the formula, alpha represents the directional angle of the tool axis vector in the horizontal plane, and beta represents the included angle of the tool axis vector with the vertical direction.

3. The Q-learning based CNC machining tool axis vector motion trajectory planning method according to claim 2, characterized in that, In step S1, the specific process of meshing and dividing is as follows: The two-dimensional polar coordinate space (a, b) is discretized at a certain resolution, a is divided into n α intervals, and b is divided into n β intervals; each grid center point corresponds to a discrete tool axis vector (a i , b i ), and these grid points constitute the state space of the tool posture.

4. The Q-learning based CNC machining tool axis vector motion trajectory planning method according to claim 1, characterized in that, In step S2, for each grid point of the state space, it is judged whether it meets the constraints of no interference and no singularity, and is marked, and the grid points meeting the constraints of no interference and no singularity are divided into feasible region, and the remaining grid points are divided into forbidden region.

5. The Q-learning based CNC machining tool axis vector motion trajectory planning method according to claim 1, characterized in that, In step S3, the action space is defined as the action set of the agent from the current grid point to all grid points in the next feasible region: A k = {(α k+1 ,β k+1 ) e feasible region}.

Images