Machine learning device, control device and machine learning method

The machine learning device optimizes IIR filter coefficients in servo control systems by representing the transfer function's zero point and pole in polar coordinates, addressing the inefficiency of existing methods and reducing settling times.

DE102019210397B4Active Publication Date: 2024-11-14FANUC LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
DE102019210397
Authority / Receiving Office
DE · DE
Patent Type
Patents
Current Assignee / Owner
Priority Date
2018-07-17
Filing Date
2019-07-15
Publication Date
2024-11-14
Estimated Expiration
2039-07-15

AI Technical Summary

Technical Problem

Existing servo control systems using IIR filters for machine learning in servo motors lack efficient methods to optimize coefficients, leading to prolonged settling times during machine learning processes.

Method used

A machine learning device that optimizes the coefficients of an IIR filter by representing the transfer function's zero point and pole in polar coordinates, using reinforcement learning to search and learn the radius and angle within a predetermined range, thereby optimizing the coefficients of the IIR filter.

Benefits of technology

This approach significantly shortens the settling time required for machine learning, enhancing the efficiency of servo control systems by minimizing position errors and optimizing the IIR filter coefficients.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

Machine learning device (200) configured to perform machine learning in connection with optimizing the coefficients of a transfer function of an IIR filter (1092, 1102) with respect to a servo control device (100) configured to control a servo motor (300) configured to drive an axis of a machine tool, robot, or industrial machine by means of a feed control having a feed calculation unit (109, 110) with the IIR filter (1092, 1102), wherein the machine learning device represents a zero point at which the transfer function of the IIR filter (1092, 1102) is zero and a pole at which the transfer function diverges to infinity in polar coordinates using a radius r and an angle θ respectively, and searches for and learns the radius r and the angle θ within a predetermined search range.in order to carry out the optimization of the coefficients of the transfer function of the IIR filter (1092, 1102).
Need to check novelty before this filing date? Find Prior Art

Description

BACKGROUND OF THE INVENTIONField of the invention

[0001] The present invention relates to a machine learning device that performs machine learning in connection with the optimization of a coefficient of an infinite impulse response (IIR) filter with respect to a servo control device that controls a servo motor driving an axis of a machine tool, robot or industrial machine, wherein the feed control of a forward feed computation unit is used which includes the IIR filter and relates to a control device comprising the machine learning device and a machine learning method. Related technology

[0002] A servo control device that uses a feed control of a feed calculation unit with an IIR filter is disclosed, for example, in patent document 1. Patent document 1 discloses an invention relating to a control device of a servo motor and discloses that a speed feed unit (corresponding to a position feed unit of the present embodiment, which will be described later) comprises a speed feed calculator (corresponding to a differentiator of the present embodiment, which will be described later) and a speed feed filter, and an IIR filter can be used as a speed feed filter (see paragraph

[0080] and the like).

[0003] Patent document 1: Japanese unexamined patent application, Publication No. JP 2003 - 23 785 A.

[0004] DE 10 2012 104 194 A1 discloses a robot with a robot mechanism unit comprising a sensor and a control unit. The control unit comprises a normal control unit that controls the operation of the robot mechanism unit, and a learning control unit which, when the robot mechanism unit is operated with a velocity command given by multiplying a teaching velocity, specified in a task program, by a velocity change ratio, performs a learning process to calculate a learning correction value from a result detected by the sensor. This correction value approximates a trajectory or position of a control target in the robot mechanism unit to a target trajectory or position, or reduces the vibrations of the control target.

[0005] RC NONGPIUR et al., "Improved Design Method for Nearly Linear-Phase IIR Filters Using Constrained Optimization," IEEE Transactions on Signal Processing, Vol. 61, No. 4, 2013, pp. 895-906, discloses a new optimization method for the design of nearly linear linear-phase IIR digital filters. The group delay deviation is minimized under the condition that the passband ripple and stopband attenuation are within a prescribed specification, and either a prescribed or an optimized group delay can be achieved.

[0006] ALTINTAS, Y, et al.: Machine tool feed drives, CIRP annals, Vol. 60, 2011, pp. 779-796 deals with the design and control of feed drive systems for machine tools, and in particular machine tools that utilize the principles of friction, rolling elements, hydrostatics and magnetic levitation.

[0007] KOBER, J., et al.: Reinforcement learning in robotics: A survey, Int. Journal of Robotics Research (IJRR), Vol. 32, 2013, pp. 1238-1274, discusses key problems and major challenges in robot reinforcement learning, as well as its most significant successes. The role of algorithms, representations, and prior knowledge in achieving these successes is examined. SUMMARY OF THE INVENTION

[0008] Patent document 1 discloses a feed control that uses an IIR filter, but does not disclose a machine element for optimizing a coefficient of a transfer function of an IIR filter.

[0009] An objective of the present invention is to provide a machine learning device capable of shortening the settling time of machine learning in order to optimize a coefficient of a transfer function of an IIR filter with respect to a servo control device that controls a servo motor using the feed control of a forward feed computation unit with the IIR filter, and to provide a control device comprising the machine learning device and a machine learning method.

[0010] (1) A machine learning device according to the present invention is a machine learning device (for example, a machine learning device 200 to be described later) configured to perform machine learning in connection with the optimization of the coefficients of a transfer function of an IIR filter with respect to a servo control device (for example, a servo control device 100 to be described later) configured to control a servo motor (for example, a servo motor 300 configured to drive an axis of a machine tool, robot, or industrial machine using feed control by a feed calculation unit (for example, a speed feed calculation unit 109 or a position feed calculation unit 110 to be described later) with the IIR filter (for example, an IIR filter 1092 or 1102 to be described later).wherein the machine learning device represents a zero point at which the transfer function of the IIR filter is zero, and a pole at which the transfer function diverges continuously in polar coordinates using a radius r and an angle θ, and searches for and learns the radius r and the angle θ within a predetermined search range in order to optimize the coefficients of the transfer function of the IIR filter.

[0011] (2) In the machine learning device according to (1) the search range of the radius r can be defined on the basis of a damping factor and the search range of the angle θ on the basis of a vibration suppression frequency.

[0012] (3) In the machine learning device according to (1) or (2), the search for the zero point can be carried out before the search for the pole.

[0013] (4) In the machine learning device according to any of paragraphs (1) to (3), the pole can be set when searching for the zero point.

[0014] (5) In the machine learning device according to one of (1) to (4), the search for the angle θ can be carried out before the search for the radius r.

[0015] (6) In the machine learning device according to one of (1) to (5), the radius r can be set to a fixed value when searching for the angle θ.

[0016] (7) In the machine device according to one of (1) to (6), the zero point can be represented by a complex number and a complex conjugate thereof.

[0017] (8) In the machine learning device according to any of (1) to (7) the feed calculation unit may be a speed feed calculation unit or a position feed calculation unit.

[0018] (9) In the machine learning device according to any of paragraphs (1) to (8) the feed calculation unit may be a velocity feed calculation unit, the machine learning device may furthermore include a position feed calculation unit with an IIR filter, and the optimization of the transfer function of the IIR filter of the velocity feed calculation unit may be performed before the optimization of the transfer function of the IIR filter of the position feed calculation unit.

[0019] (10) The machine learning device according to any one of paragraphs (1) to (9) may further comprise: a state information acquisition unit (for example, a state information acquisition unit 201 to be described later) configured to acquire state information from the servo control device, which includes a servo state with at least one position error and a transfer function of the feed calculation unit, by causing the servo control device to execute a predefined machining program; an action information output unit (for example, an action information output unit 203 to be described later) configured to output action information, including setting information for the coefficients of the transfer function, contained in the state information to the servo control device;a reward output unit (for example, a Reward Output Unit 2021, which will be described later) configured to output a value of a reinforcement learning reward based on the position error contained in the state information; and a value function update unit (for example, a Value Function Update Unit 2022, which will be described later) configured to update an action value function based on the value of the reward output unit, the state information, and the action information.

[0020] (11) In the machine learning device according to (10) the reward output unit can output the value of the reward based on an absolute value of the position error.

[0021] (12) The machine learning device according to (10) or (11) may further comprise: an optimization action information output unit (for example, an optimization action information output unit 205 to be described later) configured to generate and output correction information of the coefficients of the transfer function of the feed computation unit based on the value function updated by the value function update unit.

[0022] (13) A control device according to the present invention is a control device comprising: the machine learning device according to any one of elements (1) to (12); and a servo control device configured to control a servo motor configured to drive an axis of a machine tool, robot or industrial machine which uses feed control by a forward feed calculation unit with an IIR filter.

[0023] (14) In the control device according to (13) the machine learning device can be incorporated into the servo control device.

[0024] (15) A machine learning method according to the present invention is a machine learning method of a machine learning device (for example, a machine learning device 200 to be described later) configured to perform machine learning in connection with the optimization of the coefficients of a transfer function of an IIR filter with respect to a servo control device (for example, a servo control device 100 to be described later) configured to control a servo motor (for example), a servo motor 300 configured to control an axis of a machine tool, robot, or industrial machine using feed control by a feed calculation unit (for example, a velocity feed calculation unit 109 or a position feed calculation unit 110 to be described later) with the IIR filter (for example, an IIR filter 1092 or 1102).(which will be described later) to drive, wherein the machine learning method represents a zero point where the transfer function of the IIR filter is zero, and a pole where the transfer function diverges infinitely in polar coordinates using a radius r and an angle θ, and searches for and learns the radius r and the angle θ within a predetermined search range in order to optimize the coefficients of the transfer function of the IIR filter.

[0025] According to the present invention, it is possible to shorten the settling time of machine learning for optimizing a coefficient of a transfer function of an IIR filter. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 is a block diagram that represents a configuration example of a control device according to an embodiment of the present invention. Fig. Figure 2 is a block diagram illustrating a section of a machine tool with a servo motor, serving as an example of a control target of a servo control device. Fig. Figure 3 is a diagram describing the operation of a servo motor when a machining shape is an octagon. Fig. Figure 4 is a diagram describing the operation of a motor where a machining shape is a shape in which the corners of an octagon are alternately replaced by arcs. Fig. Figure 5 is an explanatory diagram of a complex plane illustrating a search area from a pole to a zero point. Fig. Figure 6 is a block diagram representing a machine learning device according to the present embodiment. Fig. Figure 7 is a flowchart illustrating the operation of the machine learning device according to the present embodiment. Fig. Figure 8 is a flowchart illustrating the operation of an optimization action information output unit of the machine learning device according to the present embodiment. Fig. Figure 9 is a block diagram representing a control device according to another embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0026] In the following, an embodiment of the present invention is described with reference to the drawings. Fig. Figure 1 is a block diagram illustrating a configuration example of a control device according to an embodiment of the present invention. A Fig. The control device 10 shown comprises a servo control device 100 and a machine learning device 200.

[0027] A servomotor 300 is a control target of the servo control device 100 and is used, for example, in a machine tool, a robot, or an industrial machine. The servo control device 100 can be provided as part of a machine tool, a robot, or an industrial machine together with the servomotor 300.

[0028] First, the servo control device 100 is described. The servo control device 100 comprises a position command generation unit 101, a subtractor 102, a position control unit 103, an adder 104, a subtractor 105, a speed control unit 106, an adder 107, an integrator 108, a speed-feed calculation unit 109, and a position-feed calculation unit 110. The speed-feed calculation unit 109 comprises a double differentiator 1091 and an IIR filter 1092. Furthermore, the position-feed calculation unit 110 comprises a differentiator 1101 and an IIR filter 1102.

[0029] The position command generation unit 101 generates a position command value and outputs the generated position command value to the subtractor 102, the speed-feed calculation unit 109, the position-feed calculation unit 110, and the machine learning device 200. The subtractor 102 calculates a difference between the position command value and a feedback position and outputs the difference as a position error to the position control unit 103 and the machine learning device 200.

[0030] The position command generation unit 101 generates a position command value based on a program for operating the servomotor 300. The servomotor 300 is, for example, integrated into a machine tool. If a table with a workpiece mounted on it moves in an X-axis and a Y-axis direction within a machine tool, the servo control device 100 and the servomotor 300 are integrated into the machine tool. Fig. The servomotor 300 shown in Figure 1 is provided in the X-axis direction and the Y-axis direction, respectively. If the table is moved in directions of three or more axes, the servo control device 100 and the servomotor 300 are provided in the respective axis directions. The position command generation unit 101 sets a feed rate and generates a position command value, thus obtaining a machining shape defined by the machining program.

[0031] The position control unit 103 outputs a value which is obtained by multiplying a position gain Kp with the position error to the adder 104 as the target speed value.

[0032] The adder 104 adds the target speed and the output value (the position feed term) of the position feed calculation unit 110 and outputs the result to the subtractor 105 as a forward-controlled target speed. The subtractor 105 calculates the difference between the output of the adder 104 and a feedback speed measurement value and outputs the difference to the speed control unit 106 as a speed error.

[0033] The speed control unit 106 adds a value obtained by multiplying and integrating an integral gain K1v with the speed error and a value obtained by multiplying a proportional gain K2v with the speed error, and outputs an addition result to the adder 107 as the torque setpoint.

[0034] The adder 107 adds the torque setpoint and an output value (the speed feed term) of the speed feed calculation unit 109 and outputs the addition result to the servo motor 300 as a feed-controlled torque setpoint for driving the servo motor 300.

[0035] The rotary angular position of the servomotor 300 is detected by a rotary encoder, which serves as a position sensing unit assigned to the servomotor 300, and a speed value is input as speed feedback to the subtractor 105. The speed value is integrated by the integrator 108 as a position value, and the position value is input as position feedback to the subtractor 102.

[0036] The double differentiator 1091 of the velocity-feed calculation unit 109 differentiates the position command value twice and multiplies one differentiation result by a constant α. The IIR filter 1092 performs an IIR filtering process at the output of the double differentiator 1091, represented by a transfer function VFF(z) in Expression 1 (see Formula 1 below), and outputs the processing result as a velocity forward term to the adder 107. The coefficients a1, a2, b0 to b2 in Expression 1 are coefficients of the transfer function of the IIR filter 1092. Although the denominator and numerator of the transfer function VFF(z) are quadratic functions in this example, they are not specifically restricted to quadratic functions but could be cubic or higher-order functions. VFF(z)=b0+b1z−1+b2z−21+a1z−1+a2z−2

[0037] The differentiator 1101 of the position-feed calculation unit 110 differentiates the position command value and multiplies the differentiation result by a constant β. The IIR filter 1102 performs an IIR filtering process, represented by a transfer function PFF(z) in expression 2 (see formula 2 below), at the output of the differentiator 1101 and outputs the processing result to the adder 104 as a position-feed term. The coefficients c1, c2, and d0 to d2 in expression 2 are the coefficients of the transfer function of the IIR filter 1102. Although the denominator and numerator of the transfer function PFF(z) are quadratic functions in this example, they are not specifically restricted to quadratic functions but could be cubic or higher-order functions. PFF(z)=d0+d1z−1+d2z−21+c1z−1+c2z−2

[0038] The servo control device 100 is configured in this way.

[0039] Next, a control target 400 including the servo motor 300, which is controlled by the servo control device 100, is described. Fig. Figure 2 is a block diagram depicting a section of a machine tool with the servomotor 300, which is an example of the control target 400 of the servo control device 100. The servo control device 100 causes the servomotor 300 to move the table 303 by means of a coupling mechanism 302 in order to machine a workpiece mounted on the table 303. The coupling mechanism 302 comprises a coupling 3021 coupled to the servomotor 300 and a ball screw 3023 attached to the coupling 3021, and a nut 3022 is screwed into the ball screw 3023. When the servomotor 300 rotates, the nut 3022 screwed into the ball screw 3023 moves in the axial direction of the ball screw 3023. With the movement of the nut 3022, the table 303 moves.

[0040] The rotary angular position of the servomotor 300 is detected by the rotary encoder 301, which serves as a position sensing unit and is associated with the servomotor 300. As described above, the detected signal is used as velocity feedback. The detected signal is integrated by the integrator 108 and serves as position feedback. An output of a linear scale 304, which is attached to one end of the ball screw 3023 to detect a distance traveled by the ball screw 3023, can be used as position feedback. In addition, position feedback can be generated using an accelerometer. <Maschinenlernvorrichtung 200>

[0041] The machine learning device 200 learns the coefficients of the transfer function of the IIR filter 1092 of the speed-feed calculation unit 109 and the coefficients of the transfer function of the IIR filter 1102 of the position-feed calculation unit 110 by executing a predefined machining program (hereinafter also referred to as the "learning machining program"). A machining shape defined by the learning machining program is, in this case, an octagon or a shape in which the vertices of an octagon are alternately replaced by arcs. Here, the machining shape defined by the learning machining program is not limited to these machining shapes but can be other machining shapes.

[0042] Fig. Figure 3 is a diagram describing the operation of a motor when a machining shape is an octagon. Fig. Figure 4 is a diagram describing the operation of a motor where a machining mode is a mode in which the corners of an octagon are alternately replaced by arcs. Fig. 3 and Fig. 4 assumes that a table is moved in the X and Y directions so that a workpiece (a workpiece) is machined in a clockwise direction.

[0043] If the processing shape is an octagon, as in Fig. As shown in Figure 3, the rotational speed of a motor moving the table in the Y-axis direction decreases at corner position A1, while the rotational speed of a motor moving the table in the X-axis direction increases. The direction of rotation of the motor moving the table in the Y-axis direction is reversed at corner position A2, and the table moves linearly in the opposite direction in the Y-axis direction. Furthermore, the motor moving the table in the X-axis direction rotates at the same speed and in the same direction from position A1 to position A2 and from position A2 to position A3.

[0044] The rotational speed of the motor that moves the table along the Y-axis increases at corner position A3, while the rotational speed of the motor that moves the table along the X-axis decreases. The direction of rotation of the motor that moves the table along the X-axis is reversed at corner position A4, and the table moves linearly in the opposite direction along the X-axis. Furthermore, the motor that moves the table along the Y-axis rotates at the same speed and in the same direction from position A3 to position A4 and from position A4 to the next corner position.

[0045] If the processing shape is one in which the corners of an octagon are alternately replaced by arcs, as in Fig. As shown in Figure 4, the rotational speed of a motor moving the table in the Y-axis direction decreases at corner position B1, while the rotational speed of a motor moving the table in the X-axis direction increases. The direction of rotation of the motor moving the table in the Y-axis direction is reversed at arc position B2, and the table moves linearly in the opposite direction in the Y-axis direction. Furthermore, the motor moving the table in the X-axis direction rotates at the same speed and in the same direction from position B1 to position B3. This contrasts with the case where the machining shape is an octagon, which is shown in Fig. As shown in Figure 3, the rotational speed of the motor that moves the table in the direction of the Y-axis gradually decreases as it approaches position B2, the rotation stops at position B2, and the rotational speed gradually increases as it moves away from position B2, so that a machining shape of an arc is formed before and after position B2.

[0046] The rotational speed of the motor moving the table along the Y-axis increases at corner position B3, while the rotational speed of the motor moving the table along the X-axis decreases. The direction of rotation of the motor moving the table along the X-axis is reversed at arc position B4, and the table moves linearly in the opposite direction along the X-axis. Furthermore, the motor moving the table along the Y-axis rotates at the same speed and in the same direction from position B3 to position B4 and from position B4 to the next corner position. The rotational speed of the motor moving the table along the X-axis gradually decreases as it approaches position B4, stops rotating at position B4, and gradually increases as it moves away from position B4, thus creating a machining profile of an arc before and after position B4.

[0047] In the present embodiment, it is assumed that the machine learning device 200 performs machine learning in connection with the optimization of the coefficients of the transfer function of the IIR filter 1092 of the speed-feed calculation unit 109 and the coefficients of the transfer function of the IIR filter 1102 of the position-feed calculation unit 110 by evaluating vibrations when changing a rotational speed during linear control at positions A1 and A3 and positions B1 and B3 of the machining shape defined by the learning machining program and investigates the influence on a position error.Machine learning in connection with the optimization of the coefficients of the transfer function of the IIR filter is not limited to the velocity-feed calculation unit and the position-feed calculation unit, but can also be applied to a current-feed calculation unit with an IIR filter, which is provided, for example, in the implementation of the current feed of a servo control device.

[0048] The machine learning device 200 is described in more detail below. It is assumed that the machine learning device 200 of the present embodiment performs reinforcement learning in connection with the optimization of the coefficients of the transfer function of the speed-feed calculation unit 109 and the position-feed calculation unit 110, which form a speed circuit and a position circuit, respectively, in the servo control device 100, as will be described later as an example of machine learning. Furthermore, the machine learning in the present invention is not limited to reinforcement learning; the present invention can also be applied to a case of performing other types of machine learning (for example, supervised learning).

[0049] Before describing the individual functional blocks contained in the machine learning device 200, a basic mechanism of reinforcement learning will first be described. An agent (corresponding to the machine learning device 200 in the present embodiment) observes an environmental state and selects a specific action. The environment then changes based on the action. A specific reward is granted according to the environmental change, and the agent learns to select (decide on) a better action. While supervised learning represents a completely correct answer, the reward in reinforcement learning often represents a fragmented value based on changes in a part of the environment. Therefore, the agent learns to select an action in such a way as to maximize the overall reward in the future.

[0050] In this way, reinforcement learning acquires a method for learning a suitable action based on the reciprocal effect of an action on the environment (i.e., an action to maximize the reward to be obtained in the future) by learning an action. This means that, in the present embodiment, such an action, which has an effect on the future, is obtained, for example, an action to select action information to reduce a positional error.

[0051] Although any learning method is used here as reinforcement learning, the following description uses Q-learning, a method for learning a value Q(S,A) for selecting an action A under a given environmental state S, as an example. One goal of Q-learning is to select an action A with the highest value Q(S,A) as the optimal action among the actions A that can be performed in a given state S.

[0052] At the initial point in time when Q-learning begins, the correct value of Q(S,A) for a combination of state S and action A is not known at all. Therefore, the agent learns the correct value of Q(S,A) by selecting different actions A under a given state S and making a better choice of action based on the rewards granted for the selected actions A.

[0053] Since the agent wants to maximize the total compensation received over the future, Q-learning aims to ultimately establish a relationship Q (S, A) = E [Σ (γ t ) r t] to achieve. Here, E[ ] represents an expected value, t represents time, γ is a parameter called the discount factor, which will be described later, rt is a reward at time t, and Σ is the sum at time t. In this expression, the expected value is a value expected when the state has been changed according to an optimal action. However, since it is unclear which action would be optimal in the Q-learning process, reinforcement learning is performed while searching for an optimal action through various actions. An updated expression of such a value Q(S,A) can be represented by the following expression 3 (Equation 3). Q(St+1,At+1)←Q(St,At)+α(rt+1+γmaxAQ(St+1,A)−Q(St,At))

[0054] In expression 3, St shows an environmental state at time t and A t an action at time t. Through action A t does the state change in S t+1rt+1 signifies a reward obtained by changing the state. Furthermore, a term with max is a multiplication of the Q-value by γ when an action A with the highest Q-value known at that moment under state S is performed. t+1 was selected. Here, γ is a parameter of 0 < γ ≤ 1 and is called the discount rate. Furthermore, α is a learning factor and lies in the range 0 < α ≤ 1.

[0055] Expression 3 denotes a procedure for updating a value Q(St,At) of an action A t in a state St based on a returning reward rt+1 when action A t is executed. This update expression indicates that when the value max a Q(S t+1 ,A) the best action in the next state S t+1 , which is part of an action A t is assigned, is greater than the value Q(St,At) of an action A tIn state St, Q(St,At) is increased, and in other states, Q(St,At) is decreased. That is, the value of a particular action in a given state approaches the value of the best action in the next state associated with that action. However, this difference varies depending on the discount rate γ and the reward r. t+1 In contrast, the update expression has a structure such that the value of the best action in a given state essentially affects the value of an action in a state prior to that state.

[0056] A Q-learning method for creating a Q(S,A) table for all state-action pairs (S,A) is known for performing the learning process. However, it can take a considerable amount of time for the Q-learning process to be completed if the values ​​of Q(S,A) for all state-action pairs are to be calculated, because the number of states is too large.

[0057] Thus, Q learning can utilize an existing technique known as Deep Q Network (DQN). Specifically, an agent can compute the value of Q(S,A) by constructing a value function Q using a suitable neural network and approximating the value function Q with the corresponding neural network by adjusting the network's parameters. Using DQN makes it possible to reduce the computation time for Q learning. Details of DQN are disclosed, for example, in the following non-patent document. <nicht-patent-dokument>

[0058] MNIH, Volodymyr [ua]: Human-level control through deep reinforcement learning, Nature, Vol. 518, 2015, H. 7540, pp. 529-533, URL: https: / / storage.googleapis.com / deepmindmedia / dqn / DQNNaturePaper.pdf [accessed on 2018-07-12].

[0059] The machine learning device 200 performs the Q-learning described above. The machine learning device 200 performs machine learning (hereinafter referred to as learning) of the transfer function of the IIR filter 1092 of the velocity-feed calculation unit 109 and the transfer function of the IIR filter 1102 of the in Fig. 1. Position feed calculation unit 110 shown.

[0060] The machine learning device 200 performs the learning of the coefficients of the transfer function of the IIR filter 1092 on an inner loop (the inner loop) as well as of the IIR filter 1102 before learning the coefficients of the transfer function of the IIR filter 1102. In particular, the machine learning device 200 sets the coefficients of the transfer function of the IIR filter 1102 and learns the optimal values ​​of the coefficients of the transfer function of the IIR filter 1092. Subsequently, the machine learning device 200 fixes the coefficients of the transfer function of the IIR filter 1092 to the optimal values ​​obtained through learning and learns the coefficients of the transfer function of the IIR filter 1102.By performing the learning of the coefficients of the transfer function of the IIR filter 1092 before learning the coefficients of the transfer function of the IIR filter 1102, it is possible to perform a learning in connection with the optimization of the coefficients of the transfer function of the IIR filter 1102 under the condition of the velocity-feed term optimized by the learning (the output of the IIR filter 1092).

[0061] Although learning the coefficients of the transfer function of IIR filter 1092 and learning the coefficients of the transfer function of IIR filter 1102 can be performed simultaneously, if the learning is performed concurrently, the amount of information processed for machine learning increases and the settling time of the machine learning increases.

[0062] As described above, the machine learning device 200 first performs machine learning of the coefficients of the transfer function of the IIR filter 1092 of the speed-feed calculation unit 109. The machine learning device 200 learns a value Q for selecting an action A to set the coefficients a1, a2, and b0 to b2 of the transfer function VFF(z) of the IIR filter 1092, which are associated with a state S, where state S is a servo state, such as commands and feedback comprising the coefficients a1, a2, and b0 to b2 of the transfer function VFF(z) of the IIR filter 1092 of the speed-feed calculation unit 109, and the position error information and the position command of the servo control device 100, which were acquired by executing the learning machining program.In particular, the machine learning device 200, according to the embodiment of the present invention, sets the coefficients of the transfer function VFF(z) of the IIR filter 1092 by searching within a predetermined range for a radius r and an angle θ that represent a zero point and a pole of the transfer function VFF(z) in polar coordinates, respectively, in order to learn the radius r and the angle θ. The pole is the value of z at which the transfer function VFF(z) is infinite, and the zero point is the value of z at which the transfer function VFF(z) is 0. Therefore, the coefficients in the numerator of the transfer function VFF(z) have been modified as follows. b0+b1z−1+b2z−2=b0(1+(b1 / b0)z−1+(b2 / b0)z−2)

[0063] In the following, (b1 / b0) and (b2 / b0) are denoted by b1' and b2', respectively, unless explicitly stated otherwise. The machine learning device 200 learns the radius r and the angle θ that minimize the positional error in order to set the coefficients a1, a2, b1', and b2' of the transfer function VFF(z). The coefficient b0 can be obtained by machine learning after, for example, the radius r and the angle θ have been set to the optimal values ​​r0 and θ0. The coefficient b0 can be learned simultaneously with the angle θ. Furthermore, the coefficient b o be learned simultaneously with the radius r.

[0064] The coefficients c1, c2, and d0 to d2 of the transfer function PFF(z) of the IIR filter 1102 are then learned using a procedure similar to that used for the transfer function VFF(z) of the IIR filter 1092. Although the learning of the coefficients of the transfer function VFF(z) of the IIR filter 1092 for the velocity-feed calculation unit 109 is described below, the learning of the coefficients of the transfer function PFF(z) of the IIR filter 1102 for the position-feed calculation unit 110 is performed in a similar manner.

[0065] The machine learning device 200 observes the state information S, including the servo state such as commands and feedback, including position commands and position error information from the servo control device 100 at positions A1 and A3 and / or positions B1 and B3 of the machining form. It executes the learning machining program based on the values ​​of the coefficients a1, a2, and b0 to b2 of the transfer function VFF(z) of the IIR filter 1092 to determine action A. The machine learning device 200 receives a reward for each execution of action A. The machine learning device 200 trial and error searches for the optimal action A such that the sum of rewards is maximized in the future. This allows the machine learning device 200 to determine an optimal action A (i.e.,The machine learning device 200 selects the values ​​of the optimal zero point and pole of the transfer function VFF(z) of the IIR filter 1092 with respect to state S, including the servo state such as commands and feedback, including position commands and position error information of the servo control device 100. These values ​​are acquired by executing the learning program based on the coefficient values ​​calculated from the zero point and pole values ​​of the transfer function VFF(z) of the IIR filter 1092. The direction of rotation of the servo motor in the X-axis and Y-axis directions does not change at positions A1 and A3, as well as positions B1 and B3, so that the machine learning device 200 can learn the zero point and pole values ​​of the transfer function VFF(z) of the IIR filter 1092 in linear operation.

[0066] That is, it is possible to select an action A (i.e., the values ​​of the zero point and pole of the transfer function VFF(z) of the IIR filter 1092) that minimizes the position error detected by the execution of the learning machining program by selecting an action A that maximizes the value of Q from the actions A applied to the transfer function VFF(z) of the IIR filter 1092 associated with a particular state S, based on the value function Q learned by the machine learning device 200.

[0067] The following describes a method for learning the radius r and the angle θ, which represent the zero point and the pole of the transfer function VFF(z) of the IIR filter 1092, which minimize the position error in polar coordinates in order to obtain the coefficients a1, a2, b1' and b2' of the transfer function VFF(z), and a method for obtaining the coefficient b0.

[0068] The machine learning device 200 sets a pole, z, at which the transfer function VFF(z) in expression 1 is infinite, and a zero point, z, at which the transfer function VFF(z) is equal to 0, obtained from the IIR filter 1092. The machine learning device 200 multiplies the denominator and the numerator in expression 1 by z. 2 , to obtain expression 4 (given by equation 4 below) to obtain the pole and the zero point. VFF(z)=b0(z2+b1'z+b2')z2+a1z+a2

[0069] The pole is z where the denominator of expression 4 is equal to 0 (i.e., z2+a1z+a2=0), and the zero point is z where the numerator of expression 4 is equal to 0 (i.e., z2+b1'z+b2'=0).

[0070] In the present embodiment, the pole and the zero point are represented in polar coordinates, and the machine learning device searches for the pole and the zero point in polar coordinates. The zero point is important for vibration suppression, and the machine learning device 200 first fixes the pole and sets the coefficients b1' (=-re iθ re -iθ ) and b2' (=r 2 ) one, which are calculated when z=re iθ and the complex conjugate z*=re -iθ in the numerator (z 2 +b1'z+b2') are the zero point (the angle θ lies in a predetermined range, and 0≤r≤1) as the coefficients of the transfer function VFF(z) to search for the zero point re iθ The optimal coefficients b1' and b2' are determined using polar coordinates. The radius r depends on a damping factor, and the angle θ on a vibration damping frequency. The zero point can then be set to an optimal value, and the value of the coefficient b0 can be determined. Finally, the pole of the transfer function VFF(z) is represented in polar coordinates, and the value re is determined. iθ For the pole represented in polar coordinates, a method similar to the one used for the zero point is sought. In this way, it is possible to determine the values ​​of the optimal coefficients a1 and a2 in the denominator of the transfer function VFF(z). With a fixed pole and the coefficients in the numerator of the transfer function VFF(z), it suffices to be able to suppress a high-frequency-specific gain, and the pole corresponds, for example, to a second-order low-pass filter. Thus, for example, a transfer function of a second-order low-pass filter is represented by the expression 5 (referred to below as mathematical). ω is a peak gain frequency of the filter. 1s² + 2ωs + ω²

[0071] If the pole is a third-order low-pass filter, the third-order low-pass filter can be formed by providing three first-order low-pass filters whose transfer function is represented by 1 / (1+Ts) (where T is a time constant of the filter). The third-order low-pass filter can be formed by combining the first-order low-pass filter with the second-order low-pass filter in expression 5. The transfer function in the z-domain is obtained by a bilinear transformation of the transfer function in the s-domain.

[0072] Although the pole and zero point of the transfer function VFF(z) can be sought simultaneously, it is possible to reduce the number of machine learning operations and shorten the learning time if the pole and zero point are sought and learned separately.

[0073] The search areas of the pole and the origin can be limited to predefined search areas, indicated by hatched areas, by defining the radius r in the range 0≤r≤1, for example in a complex plane in Fig. 5 and the angle θ is defined in a responsible frequency range of a speed control loop. The upper limit of the frequency range can be set to 200 Hz, since the oscillation generated by the resonance of the speed control loop is, for example, approximately 200 Hz. Although the search range is determined by the resonance characteristics of a control target such as a machine tool, since the angle θ at approximately 250 Hz corresponds to a 90° value at a sampling time of 1 msec, with an upper limit of the frequency range of 200 Hz, a search range of the angle θ is defined as in the complex plane of Fig. 5. By limiting the search area to a predefined range, it is possible to reduce the amount of machine learning and shorten the settling time of the machine learning.

[0074] When searching for the origin in polar coordinates, the coefficient b0 is first set to, for example, 1, and the radius r to an arbitrary value in the range of 0≤r≤1, and the angle θ in the search range of Fig. 5 is set up using trial and error in order to determine the coefficients b1' (=-re iθ -re -iθ ) and b2' (=r 2 ) so that z and the complex conjugate z* are the zero point of (z 2 +b1'z+b2'). The initial setting value of the angle θ is determined in the Fig. The search range shown in Figure 5 is set. The machine learning device 200 transmits the setting information of the obtained coefficients b1' and b2' as action A to the IIR filter 1092 and sets the coefficients b1' and b2' in the numerator of the transfer function VFF(z) of the IIR filter 1092. The coefficient b0 is set to 1, for example, as described above. If such an ideal angle θ0 is determined that maximizes the value of the value function Q by the machine learning device 200, which performs the learning to search for the angle θ, the angle θ is fixed to the angle θ0 and the radius r is varied to thereby determine the coefficients b1' (=-re jθ -re -iθ The coefficients b1' and b2' (=r2) are set in the numerator of the transfer function VFF(z) of the IIR filter 1092. By learning the search for the radius r, an optimal radius r0 is determined that maximizes the value of the value function Q. The coefficients b1' and b2' are set using the angle θ0 and the radius r0, then the learning process is performed with respect to b0, determining the coefficients b0, b1', and b2' in the numerator of the transfer function VFF(z).

[0075] If the pole is sought in polar coordinates, the learning process can be performed similarly to the denominator of the transfer function VFF(z). First, the radius r is set to a value within the range (for example, 0 ≤ r ≤ 1), and the angle θ is sought within the search range, similar to the search for the origin, in order to determine an ideal angle θ of the pole of the transfer function VFF(z) of the IIR filter 1092 through learning. Then, the angle θ is fixed to the specified angle, and the radius r is sought and learned, thus determining the ideal angle θ and the ideal radius r of the pole of the transfer function VFF(z) of the IIR filter 1092. The optimal coefficients a1 and a2 are then determined according to the ideal angle θ and the ideal radius r of the pole. As described above, the radius r depends on the damping factor, and the angle θ depends on a vibration suppression frequency.Therefore, it is preferable to learn the angle θ before the radius r in order to suppress vibrations.

[0076] In this way, by searching for the radius r and the angle θ that represent the zero point and the pole of the transfer function VFF(z) of the IIR filter 1092 in polar coordinates, so that the position error is minimized, it is possible to perform the optimization of the coefficients a1, a2, b0, b1' and b2' of the transfer function VFF(z) more efficiently than by learning the coefficients a1, a2, b0, b1' and b2' directly.

[0077] When the coefficient b0 of the transfer function VFF(z) of the IIR filter 1092 is learned, for example, the initial value of the coefficient b0 is set to 1, and then the coefficient b0 of the transfer function VFF(z) contained in action A is incrementally increased or decreased. The initial value of the coefficient b0 is not limited to 1; it can be set to any value. The machine learning device 200 returns a reward based on a position error for each execution of action A and adjusts the coefficient b0 of the transfer function VFF(z) to an ideal value that maximizes the value of the value function Q by increasingly learning to trial and error for the optimal action A, thus maximizing the total future reward.Although the learning of the coefficient b0 in this example is performed after the learning of the radius r, the coefficient b0 can be learned simultaneously with the angle θ and the radius r. While the radius r, the angle θ, and the coefficient b0 can be learned concurrently, it is possible to reduce the amount of machine learning required and shorten the machine learning settling time by learning these coefficients separately.

[0078] When the learning process for optimizing the coefficients of the transfer function VFF(z) of the IIR filter 1092 is complete, the learning process for optimizing the coefficients of the transfer function PFF(z) of the IIR filter 1102 is then performed. The machine learning device 200 multiplies the denominator and numerator in expression 2 by Z 2 , to obtain expression 6 (given by equation 6 below) to obtain the pole and the zero point. PFF(z)=d0(z2+d1'z+d2')z2+c1z+c2 d1' and d2' in expression 6 correspond to (d1 / d o ) or (d2 / d o Subsequently, the radius r and the angle θ, which represent the zero point and the pole of the transfer function PFF(z) of the IIR filter 1102 in polar coordinates, are learned in order to perform an optimization of the coefficients c1, c2 and d0 to d2 of the transfer function PFF(z) similar to the case of learning the radius r and the angle θ, which represent the zero point and the pole of the transfer function VFF(z) of the IIR filter 1092. Therefore, a description of the latter is omitted.

[0079] Fig. Figure 6 is a block diagram illustrating the machine learning device 200 of the present embodiment. While the learning of the coefficients of the transfer function VFF(z) of the IIR filter 1092 of the velocity-feed calculation unit 109 is described later, the learning of the coefficients of the transfer function of the IIR filter 1102 of the position-feed calculation unit 110 is carried out similarly. As shown in Fig. As shown in Figure 6, the machine learning device 200 for performing the reinforcement learning described above comprises a state information acquisition unit 201, a learning unit 202, an action information output unit 203, a value function storage unit 204, and an optimization action information output unit 205. The learning unit 202 comprises a reward output unit 2021, a value function update unit 2022, and an action information generation unit 2023.

[0080] The state information acquisition unit 201 acquires the state S from the servo control device 100, which includes servo states such as commands and feedback with position commands and position error information from the servo control device 100. This state S is acquired by executing the learning machining program based on the values ​​of the coefficients a1, a2, and b0 to b2 of the transfer function VFF(z) of the IIR filter 1092 of the velocity-feed calculation unit 109 of the servo control device 100. The state information S corresponds to a state S of the environment in Q-learning. The state information acquisition unit 201 outputs the acquired state information S to the learning unit 202.Furthermore, the state information acquisition unit 201 acquires the angle θ and the radius r, which represent the zero point and the pole in polar coordinates, and the corresponding coefficients a1, a2, b1' and b2' from the action information generation unit 2023, in order to store it, and outputs the angle θ and the radius r, which represent the zero point, and the pole, which corresponds to the coefficients a1, a2, b1' and b2', which were acquired by the servo control device 100 in polar coordinates to the learning unit 202.

[0081] The initial settings of the transfer function VFF(z) of the IIR filter 1092 at the time when Q-learning begins are preset by a user. In the present embodiment, the user-set initial settings of the coefficients a1, a2, and b0 to b2 of the transfer function VFF(z) of the IIR filter 1092 are then optimized by gain learning of the search for the radius r and the angle θ, which represent the origin and the pole in polar coordinates, as described above. The coefficient α of the double differentiator 1091 of the velocity-feed calculation unit 109 is set to a fixed value, such as α=1. Furthermore, the initial settings in the denominator of the transfer function VFF(z) of the IIR filter 1092 are adjusted to the values ​​shown in Equation 5. (the transfer function in the z-domain converted by bilinear transformation).Furthermore, as with the initial setting values ​​of the coefficients b0 to b2 in the numerator of the transfer function VFF(z), b0=1, the radius r can be set to a value within the range 0≤r≤1 and the angle θ to a value within the specified search range. The initial setting values ​​of the position-feed calculation unit 110 are set similarly. Additionally, regarding the coefficients a1, a2, and b0 to b2, and the coefficients c1, c2, and d0 to d2, if an operator pre-sets the machine tool, machine learning can be performed using the values ​​of the radius r and the angle θ, which represent the zero point and the pole of the set transfer function in polar coordinates as initial values.

[0082] Learning Unit 202 is a unit that learns the value Q(S,A) when a specific action A is selected under a specific environmental state S. Here, for example, action A sets the coefficient b0 to 1 and calculates the correction information for the coefficients b1' and b2' in the numerator of the transfer function VFF(z) of IIR filter 1092, based on the correction information of the radius r and the angle θ, which represent the zero point of the transfer function VFF(z) in polar coordinates. The following description provides an example case where, for instance, the coefficient b0 is initially set to 1 and the action information A is the correction information for the coefficients b1' and b2'.

[0083] The reward output unit 2021 is a unit that calculates a reward when action A is selected under a given state S. Here, a set (a position error set) of position errors that are state variables of state S is denoted by PD(S), and a set of position errors that are state variables with respect to state information S' that are modified by state S' due to action information A is denoted by PD(S'). Furthermore, the position error value in state S is a value calculated based on a predefined evaluation function f(PD(S)). Functions can be used as the evaluation function f: A function that calculates an integrated value of an absolute value of a positional error. ∫|e|dt A function that calculates an integrated value by weighting an absolute value of a positional error over time. ∫t|e|dt A function that calculates an integrated value of a 2n-th power (n is a natural number) of an absolute value of a position error. ∫e2ndt(n is a natural number) A function that calculates a maximum value of an absolute value of a positional error. Max{|e|}

[0084] In this case, the reward output unit 2021 sets the value of a reward to a negative value if the position error value f(PD(S')) of the servo control device 100, which is operated based on the velocity-feed calculation unit 109, after the correction with respect to the state information S' corrected by the action information A, is greater than the position error value f(PD(S)) of the servo control device 100, which is operated based on the velocity-feed calculation unit 109, before the correction with respect to the state information S, before it is corrected by the action information A.

[0085] On the other hand, the reward output unit 2021 sets the reward value to a positive value if the position error value f(PD(S')) of the servo control device 100, which is operated based on the velocity-feed calculation unit 109, after correction with respect to the state information S' corrected by the action information A, is smaller than the position error value f(PD(S)) of the servo control device 100, which is operated based on the velocity-feed calculation unit 109, before correction with respect to the state information S, before it is corrected by the action information A.Furthermore, the reward output unit 2021 can set the reward value to zero if the position error value f(PD(S')) of the servo control device 100, which is operated based on the velocity-feed calculation unit 109, is corrected after the correction with respect to the state information S' corrected by the action information A is equal to the position error value f(PD(S)) of the servo control device 100, which is operated based on the velocity-feed calculation unit 109, before the correction with respect to the state information S by the action information A.

[0086] Furthermore, if the position error value f(PD(S')) in state S' after execution of action A becomes greater than the position error value f(PD(S)) in the previous state S, the negative value can be increased proportionally. That is, the negative value can be increased according to the degree of increase in the position error value. Conversely, if the position error value f(PD(S')) in state S' after execution of action A becomes less than the position error value f(PD(S)) in the previous state S, the positive value can be increased proportionally. That is, the positive value can be increased according to the degree of decrease in the position error value.

[0087] The Value Function Update Unit 2022 updates the value function Q stored in the Value Function Memory Unit 204 by performing a Q-Lemen based on the state S, the action A, the state S' when action A is applied to state S, and the value of the compensation calculated as described above. The value function Q can be updated through online learning, batch learning, or mini-batch learning. Online learning is a learning method in which a specific action A is applied to a current state S, and the value function Q is updated immediately when the current state S transitions to a new state S'. Batch learning is a learning method in which, by applying a specific action A to a current state S and repeating the transition of state S to a new state S', the value function Q is updated with all the collected learning data.Mini-batch learning is a learning method that is an interface between online learning and batch learning and involves updating the value function Q when a certain amount of learning data is collected.

[0088] The action information generation unit 2023 selects action A in the Q-learning process with respect to the current state S. The action information generation unit 2023 generates action information A and outputs the generated action information A to the action information output unit 203 to perform an operation (corresponding to action A of Q-learning) to correct the coefficients b1' and b2' of the transfer function VFF(z) of the IIR filter 1092 of the servo control device 100 in the Q-learning process, based on the radius r and the angle θ, which represent the origin in polar coordinates. Specifically, the action information generation unit 2023 increases or decreases the angle θ received from the state information acquisition unit 201 within the range specified in the Q-learning unit 201. Fig. 5 search range shown in a state in which the coefficients a1, a2 and b0 of the transfer function VFF(z) in expression 4 are fixed and the radius r received by the state information acquisition unit 201 with the zero point of z set in the numerator (z 2 +b1'z+b2') as re iθ , for example, to find the origin in polar coordinates. Furthermore, z is set as the origin and its complex conjugate z* using the fixed radius z and the increased or decreased angle θ, and new coefficients b1' and b2' are calculated based on the origin.

[0089] If state S transitions to state S' by increasing or decreasing the angle θ and readjusting the coefficients b1' and b2' of the transfer function VFF(z) of the IIR filter 1092, and a plus reward (a positive reward) is offered in return, the action information generation unit 2023 can choose a policy in which an action A' that further reduces the value of the position error, for example by increasing or decreasing the angle θ similarly to the previous action, is selected as the next action A'.

[0090] In contrast, if a negative reward is offered in return, the Action Information Generation Unit 2023 can choose a policy whereby an action A' that results in the value of the position error being smaller than the previous value, for example by decreasing or increasing the angle θ as opposed to the previous action, is selected as the next action A'.

[0091] The Action Information Generation Unit 2023 can select a policy in which action A' is selected according to a known procedure, such as a greedy method in which an action A' with the highest value Q(S,A) is selected among the values ​​of the currently estimated actions A, or an ε-greedy method in which an action A' is randomly selected with a certain low probability ε and an action A' with the highest value Q(S,A) is selected in other cases.

[0092] In the search for the angle θ, an ideal angle θ0, which maximizes the value of Q, is determined by learning using optimization action information (which will be described later) from the optimization action information output unit 205. The action information generation unit 2023 fixes the angle θ to the angle θ0 for the search for the radius r in the range 0≤r≤1 and sets the coefficients b1' and b2' in the numerator of the transfer function VFF(z) of the IIR filter 1092 similarly to the search for the angle θ. If the search for the radius r continues and an ideal radius r0 maximizing the value of Q is determined by learning from the optimization action information output unit 205, which will be described later, the action information generation unit 2023 determines the optimal coefficients b1' and b2' in the numerator.Then, as described above, the optimal values ​​of the coefficients in the numerator of the transfer function VFF(z) are learned by learning the coefficient b0.

[0093] The Action Information Generation Unit 2023 then searches for the coefficients of the transfer function relative to the numerator of the transfer function VFF(z) based on the radius r and the angle θ, which represent the pole in polar coordinates as described above. The Lemen sets the radius r and the angle θ, which represent the pole in polar coordinates, by gain learning similar to the numerator of the transfer function VFF(z) of the IIR filter 1092. In this case, the radius r is learned after learning the angle θ, similar to the numerator of the transfer function VFF(z). Since the learning procedure is similar to the case of finding the zero point of the transfer function VFF(z), a detailed description of the latter is omitted.

[0094] The action information output unit 203 is a unit that transmits the action information A, output by the learning unit 202, to the servo control device 100. As described above, the servo control device 100 fine-tunes the current state S (i.e., the currently set radius r and angle θ, which represent the zero point of the transfer function VFF(z) of the IIR filter 1092 in polar coordinates) based on the action information in order to transition to the next state S' (i.e., the coefficients b1' and b2' of the transfer function VFF(z) of the IIR filter 1092 corresponding to the corrected zero point).

[0095] The value function storage unit 204 is a storage device that stores the value function Q. The value function Q can be stored as a table (hereinafter referred to as the action value table) for each saturation set and each action A, for example. The value function Q stored in the value function storage unit 204 is updated by the value function update unit 2022. Furthermore, the value function Q stored in the value function storage unit 204 can be shared with other machine learning devices 200. When the value function Q is shared by a large number of machine learning devices 200, because reinforcement learning can be distributed across the respective machine learning devices 200, it is possible to improve the efficiency of reinforcement learning.

[0096] The optimization action information output unit 205 generates the action information A (hereinafter referred to as "optimization action information"), which causes the velocity-feed calculation unit 109 to perform an operation to maximize the value Q(S,A) based on the value function Q, which is updated by the value function update unit 2022, which performs Q-learning. More precisely, the optimization action information output unit 205 acquires the value function Q stored in the value function storage unit 204. As described above, the value function Q is updated by the value function update unit 2022, which performs Q-learning. The optimization action information output unit 205 generates the action information based on the value function Q and outputs the generated action information to the servo control device 100 (the IIR filter 1092 of the velocity-feed calculation unit 109).The optimization action information includes information that corrects the coefficients of the transfer function VFF(z) of the IIR filter 1092 by learning the angle θ, the radius r and the coefficient b0 similar to the action information output by the action information output unit 203 in the Q-learning process.

[0097] In the servo control device 100, the coefficients of the transfer function with respect to the numerator of the transfer function VFF(z) of the IIR filter 1092 are corrected based on action information derived from the angle θ, the radius r, and the coefficient b0. After optimizing the coefficients in the numerator of the transfer function VFF(z) of the IIR filter 1092 using the operations described above, the machine learning device 200 performs an optimization of the coefficients in the denominator of the transfer function VFF(z) of the IIR filter 1092 by learning the angle θ and the radius r in a manner similar to the optimization process.Subsequently, the machine learning device 200 can, similar to the learning and optimization of the coefficients of the transfer function VFF(z) of the IIR filter 1092, perform the learning and optimization of the coefficients of the transfer function PFF(z) of the IIR filter 1102 by learning the angle θ, the radius r, and the coefficient d0, and operate in such a way as to reduce the position error value. As described above, the use of the machine learning device 200 according to the present invention makes it possible to simplify the setting of the parameters of the velocity-feed calculation unit 109 and the position-feed calculation unit 110 of the servo control device 100.

[0098] The following describes the functional blocks of the servo controller 100 and the machine learning device 200. To implement these functional blocks, the servo controller 100 and the machine learning device 200 each include an arithmetic processing unit, such as a central processing unit (CPU). The servo controller 100 and the machine learning device 200 each also include an auxiliary storage device, such as a hard disk drive (HDD), which stores various control programs, such as application software or an operating system (OS), and a main storage device, such as main memory (RAM), to store data that is temporarily needed when the arithmetic processing unit executes a program.

[0099] In each of the servo control devices 100 and the machine learning device 200, the arithmetic processing device reads an application or operating system from the auxiliary memory device and inserts the read application software and the operating system into the main memory device to perform arithmetic processing based on the read application software or the operating system. The arithmetic processing device also controls various types of hardware provided in each device based on the arithmetic result. In this way, the functional blocks of the present embodiment are realized. That is, the present embodiment can be realized through the cooperation of hardware and software.

[0100] Since the Machine Learning Device 200 involves a large amount of computation related to machine learning, graphics processing units (GPUs) can be mounted on a personal computer and used for arithmetic processing related to machine learning using a technique called General Purpose Computing on Graphics Processing Units (GPGPUs). This enables high-speed processing. Furthermore, to perform even faster processing, the Machine Learning Device 200 can build a computer cluster consisting of a large number of computers equipped with such GPUs and perform parallel processing with the numerous computers in the cluster.

[0101] Subsequently, the operation of the machine learning device 200 during Q-learning according to the present embodiment is described with reference to the flowcharts of Fig. 7 described. The flowchart of Fig. Figure 7 is a flowchart relating to learning the angle θ, radius r and coefficient b0 in order to determine the coefficients b0 to b2 with respect to the transfer function in the numerator of the transfer function VFF(z) of the IIR filter 1092 of the velocity-feed calculation unit 109. Although the following flowchart describes the learning of the angle θ, the radius r and the coefficient b0, which represent the zero point of the transfer function VFF(z) for determining the coefficients b0 to b2 with respect to the transfer function in the numerator of the transfer function VFF(z) of the IIR filter 1092 of the velocity-feed calculation unit 109 in polar coordinates as an example, the learning of the angle θ and the radius r, which represent the pole for determining the coefficients a1 and a2 with respect to the transfer function in the denominator in polar coordinates, can be carried out according to a similar flowchart.Since the flowchart of learning the angle θ, the radius r and the coefficient d0, which define the zero point and the pole of the transfer function PFF(z) for determining the coefficients c1, c2 and d0 to d2 of the transfer function PFF(z) of the IIR filter 1102 of the subsequently carried out position-feed calculation unit 110 in polar coordinates is similar to that in . Fig. Since the flowchart shown in section 7 is presented, its description is omitted.

[0102] In step S11, the state information acquisition unit 201 acquires the state information S from the servo control device 100. The acquired state information S is output to the value function update unit 2022 and the action information generation unit 2023. As described above, the state information S is information corresponding to the Q-learning state and comprises the coefficients a1, a2, and b0 to b2 of the transfer function VFF(z) of the IIR filter 1091 at the time of step S11. The state information S0 at an initial time at which the learning begins is an initial setting value, and the coefficient b0 with respect to the numerator of the transfer function VFF(z) of the IIR filter 1092 and the coefficients a1 and a2 with respect to the denominator are set to fixed values.In this way, a position error, the PD(S) set according to a given feed rate and a machining shape of a circle, is detected when the coefficients a1, a2 and b0 to b2 of the transfer function VFF(z) of the IIR filter 1092 are set to initial values.

[0103] The position error value PD(S0) in state S0 is obtained by the subtractor 102 at the point in time when Q-learning begins, initially by actuating the servo control device 100 according to a learning machining program. The position command generation unit 101 issues position commands sequentially according to a predefined machining shape (for example, a machining shape of an octagon) defined by the machining program. For example, a position command value corresponding to the machining shape of an octagon is issued by the position command generation unit 101 and output to the subtractor 102, the speed-feed calculation unit 109, the position-feed calculation unit 110, and the machine learning device 200.The subtractor 102 outputs a difference between the target position value and the acquisition position output by the integrator 108 at positions A1 and A3 and positions B1 and B3 of the machining form to the machine learning device 200 as a position error PD(S0). In the machine learning device 200, the difference between the target position value and the acquisition position output by the integrator 108 at positions A1 and A3 and positions B1 and B3 of the machining form can be extracted as a position error PD(S0).

[0104] In step S12, the action information generation unit 2023 generates new action information A in the order of angle θ, radius r, and coefficient b0 based on radius r and angle θ, which, as described above, represent the origin in polar coordinates. The generated new action information A is then output to the servo control device 100 via the action information output unit 203. The action information generation unit 2023 outputs the new action information A based on the directive described above. The servo control device 100, having received the action information A, drives a machine tool with the servo motor 300 according to state S', which is obtained by correcting the coefficients b1' and b2' of the transfer function VFF(z) of the IIR filter 1092 associated with the current state S, based on the received action information.As described above, the action information corresponds to action A in Q-learning.

[0105] In step S13, the state information acquisition unit 201 records the position error PD(S') in the new state S' and the coefficients b1' and b2' of the transfer function VFF(z) of the IIR filter 1092 from the subtractor 102. In this way, the state information acquisition unit 201 records the position error set PD(S') corresponding to the processing shape of an octagon (specifically positions A1 and A3 and positions B1 and B3 of the processing shape) according to the coefficients of the transfer function VFF(z) in state S'. The recorded state information is output to the reward output unit 2021.

[0106] In step S14, the reward output unit 2021 determines a magnitude relationship between the position error value f(PD(S')) in state S' and the position error value f(PD(S)) in state S and sets the reward to a negative value in step S15 if f(PD(S')) > f(PD(S)). <f(PD(S)), setzt die Belohnungsausgabeeinheit 2021 die Belohnung in Schritt S16 auf einen positiven Wert. Wenn f(PD(S'))=f(PD(S)), setzt die Belohnungsausgabeeinheit 2021 die Belohnung in Schritt S17 auf Null. Die Belohnungsausgabeeinheit 2021 kann eine Gewichtung der negativen und positiven Belohnungswerte vornehmen.

[0107] When any of steps S15, S16, and S17 finish, the value function update unit 2022 updates the value function Q stored in memory unit 204 based on the value of the compensation calculated in step S18. Although the online updates in step S18 are shown as examples, batch updates or mini-batch updates can be performed instead. Subsequently, the flow returns to step S11 if step S19 determines that learning the angle θ is incomplete, and the flow proceeds to step S20 if learning is complete. Similarly, if step S20 determines that learning the radius r is incomplete, the flow returns to step S21 if learning is complete.The flow then returns to step S11 if step S21 determines that the learning of the coefficient b0 is incomplete, and the flow terminates when the learning is complete. Afterward, the flow returns again to step S11, and the processes described above for steps S11 to S21 are repeated, with the value function Q settling to a corresponding value. The learning of the angle θ, the radius r, and the coefficient can terminate if the processes described above are repeated for a predetermined number of times or for a predetermined period.

[0108] Due to the in relation to Fig. In contrast to the functionality described in Section 7 and similar sections, the machine learning device 200 described as an example in the present embodiment offers the advantage that it is possible to further reduce the learning time required to optimize the coefficients of the transfer functions of the IIR filters 1092 and 1102 by finding and learning, within a predefined range, the values ​​of the radius r and the angle θ, which represent the zero point and the pole of the transfer functions of the IIR filters 1092 and 1102 in polar coordinates. Next, a process during the generation of the optimization action information by the optimization action information output unit 205 is described with reference to the flowchart of Fig. As described in section 8, the information output unit 205 of the optimization action first captures the value function Q stored in the memory unit 204 in step S31. As described above, the value function Q is updated by the value function update unit 2022, which performs Q learning.

[0109] In step S32, the output unit 205 generates the optimization action information based on the value function Q and outputs the generated optimization action information to the IIR filter 1092 of the servo control device 100. Using the operations described above, the machine learning device 200 optimizes the angle θ, the radius r, and the coefficient b0, which represent the zero point for determining the coefficients in the numerator of the transfer function VFF(z) of the IIR filter 1092 in polar coordinates, and then optimizes the angle θ and the radius r, which represent the pole for determining the coefficients a1 and a2 with respect to the transfer function in the denominator in polar coordinates.Using similar operations, the machine learning device 200 performs learning with respect to the angle θ and the radius r, which represent the zero point and the pole of the transfer function PFF(z) in polar coordinates, and performs learning and optimization with respect to the coefficient d0 to determine the coefficients c1, c2 and d0 to d2 of the transfer function PFF(z) of the IIR filter 1102.

[0110] In the present embodiment, the machine learning device 200 can be used due to the information relating to Fig. The processes described in section 8 generate optimization action information based on the value function Q obtained through learning, and the servo control device 100 can facilitate the adjustment of the currently set coefficients a1, a2, and b0 to b2 of the transfer function VFF(z) of the IIR filter 1092 and reduce the position error value based on the optimization action information. Furthermore, the position error value can be reduced even further when the velocity-feed coefficients are set to the initial values ​​for larger dimensions and the machine learning device 200 performs the learning. Regarding the adjustment of the coefficients c1, c2, and d0 to d2 of the transfer function PFF(z) of the IIR filter 1102, the position error value can be reduced similarly to the adjustment of the coefficients a1, a2, and b0 to b2 of the transfer function VFF(z) of the IIR filter 1092.

[0111] In the present embodiment, the reward output unit 2021 calculated the reward value by comparing the evaluation function value f(PD(S)) of the position error in state S, calculated based on the predefined evaluation function f(PD(S)) using the position error PD(S) in state S as input, with the evaluation function value f(PD(S')) of the position error in state S', calculated based on the evaluation function f(PD(S')) using the position error PD(S') in state S' as input. However, the reward output unit 2021 can add an element other than the position error when calculating the reward value.For example, the machine learning device 200 can add at least one of the position errors output by the subtractor 102 to a difference between a speed feedback and a position-feed-controlled speed command and a position-feed-controlled torque command output by the adder 107, in addition to the position error output by the subtractor 102.

[0112] Furthermore, either the position feed calculation unit or the speed feed calculation unit can be provided as the feed calculation unit. In this case, if only the position feed calculation unit is provided, the double differentiator 1091, the IIR filter 1092, and the adder 107, for example, are not required. The servo control unit of the servo control device described above and the components contained in the machine learning device can be implemented by hardware, software, or a combination thereof. The servo control method, which is carried out by the interaction of the components contained in the servo control device described above, can also be implemented by hardware, software, or a combination thereof. Here, implementation by software means that it occurs when a computer reads and executes a program.

[0113] Programs can be stored on various types of non-volatile, computer-readable media and made available to a computer. Non-volatile, computer-readable media include various types of physical storage media. Examples of non-volatile, computer-readable media are magnetic recording media (such as flexible disks and hard disk drives), magneto-optical recording media (such as magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, semiconductor memory (such as mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), Flash ROMs, and RAM (Random Access Memory)). Furthermore, programs can be delivered to a computer via various types of temporary, computer-readable media.

[0114] The embodiment described above is a preferred embodiment of the present invention. However, the scope of the present invention is not limited to this embodiment; rather, the present invention can be represented in various modifications without departing from the spirit of the present invention. <Modifikation, bei der die Servosteuervorrichtung die Maschinenlernvorrichtung umfasst>

[0115] Although the machine learning device 200 is configured as a device separate from the servo control device 100 in the embodiments described above, some or all functions of the machine learning device 200 can be implemented by the servo control device 100. <Freiheit in der Systemkonfiguration>

[0116] Fig. Figure 9 is a block diagram representing a control device according to another embodiment of the present invention. As in Fig. Figure 9 shows a control device 10A comprising n servo control devices 100-1 to 100-n (n is a natural number of 2 or more), n machine learning devices 200-1 to 200-n, and a network 500. n is an arbitrary natural number. Each of the n servo control devices 100-1 to 100-n corresponds to the one shown in Figure 9. Fig. 1 servo control device 100 shown. Each of the n machine learning devices 200-1 to 200-n corresponds to the one shown in Figure 1. Fig. 1 machine learning device 200 shown.

[0117] The servo controller 100-1 and the machine learning device 200-1 are coupled one-to-one and communicatively connected. The servo controllers 100-2 to 100-n and the machine learning devices 200-2 to 200-n are similarly connected to the servo controller 100-1 and the machine learning device 200-1. Although the n pairs of servo controllers 100-1 to 100-n and the machine learning device 200-1 to 200-n are connected via the network 500 in Fig. Since 9 are connected, the n pairs of servo control devices 100-1 to 100-n and the machine learning devices 200-1 to 200-n can each be directly connected via interfaces. A plurality of n pairs of servo control devices 100-1 to 100-n and machine learning devices 200-1 to 200-n can, for example, be provided in the same system or in different systems.

[0118] The 500 network, for example, is a local area network (LAN) set up in a facility, the internet, a public telephone network, or a combination thereof. The specific communication scheme of the 500 network, as well as whether the network uses a wired or wireless connection, and so on, are not particularly restricted.

[0119] In the Fig. Although the machine learning devices 200-1 to 200-n and the servo control devices 100-1 to 100-n shown in Figure 9 can be connected as a one-to-one correspondence, a machine learning device 200-1 can be connected via the network 500 to a multitude of servo control devices 100-1 to 100-m (m <n oder m=n) kommunizierbar verbunden sein und ein Maschinenlernen der Servosteuervorrichtungen 100-1 bis 100-m durchführen. In diesem Fall kann ein verteiltes Verarbeitungssystem eingesetzt werden, bei dem die jeweiligen Funktionen der Maschinenlernvorrichtung 200-1 gegebenenfalls auf eine Vielzahl von Servern verteilt sind. Die Funktionen der Maschinenlernvorrichtung 200-1 können durch die Verwendung einer virtuellen Serverfunktion oder dergleichen in einer Cloud realisiert werden.If there are a large number of machine learning devices 200-1 to 200-n, each corresponding to a large number of servo control devices 100-1 to 100-n with the same type name, specification, or series, the machine learning devices 200-1 to 200-n can be configured to share the learning results. In this way, another optimal model can be created.

[0120] Although the embodiment of the present invention describes, as an example, the case in which the transfer functions of the IIR filters 1092 and 1102 are quadratic functions, the transfer function is not limited to the quadratic function as described above in the embodiment. The transfer function can be a cubic function or a higher-order function. EXPLANATION OF REFERENCE SYMBOLS 10, 10A control device 100 servo control device 101 Position command generation unit 102 subtractors 103 Position control unit 104 Adders 105 subtractors 106 Speed ​​control unit 107 Adders 108 Integrator 109 Speed-Feed Calculation Unit 110 position feed calculation unit 200 machine learning devices 201 Condition Information Acquisition Unit 202 Learning Unit 203 Action Information Output Unit 204 Value function memory unit 205 Optimization Action Information Output Unit 300 servo motor 400 Tax target 500 network

Claims

[1] A machine learning device (200) configured to perform machine learning related to optimizing the coefficients of a transfer function of an IIR filter (1092, 1102) with respect to a servo control device (100) configured to control a servo motor (300) configured to drive an axis of a machine tool, a robot, or an industrial machine by means of a feed controller having a feed calculation unit (109, 110) with the IIR filter (1092, 1102), wherein the machine learning device represents a zero point at which the transfer function of the IIR filter (1092, 1102) is zero and represents a pole at which the transfer function diverges to infinity in polar coordinates using a radius r and an angle θ, respectively, and searches for the radius r and the angle θ within a predetermined search range and learns,in order to optimize the coefficients of the transfer function of the IIR filter (1092, 1102). [2] The machine learning apparatus (200) according to claim 1, wherein the search range of the radius r is defined based on a damping factor, and the search range of the angle θ is defined based on a vibration suppression frequency. [3] The machine learning apparatus (200) of claim 1 or 2, wherein the search for the zero point is performed before the search for the pole. [4] The machine learning device (200) according to any one of claims 1 to 3, wherein the pole is fixed in the search for the zero point. [5] The machine learning apparatus (200) of any one of claims 1 to 4, wherein the search for the angle θ is performed before the search for the radius r. [6] The machine learning device (200) according to any one of claims 1 to 5, wherein the radius r is set to a fixed value when searching for the angle θ. [7] The machine learning apparatus (200) according to any one of claims 1 to 6, wherein the zero point is represented by a complex number and a conjugate complex number thereof. [8] The machine learning apparatus according to any one of claims 1 to 7, wherein the feed calculation unit is a speed feed calculation unit or a position feedforward calculation unit. [9] Machine learning device (200) according to one of claims 1 to 8, wherein the feed calculation unit is a speed-feed calculation unit, the machine learning device further comprises a position feed calculation unit with an IIR filter (1092, 1102), and the optimization of the transfer function of the IIR filter (1092, 1102) of the speed-feed calculation unit is carried out before the optimization of the transfer function of the IIR filter (1092, 1102) of the position-feed calculation unit. [10] Machine learning device (200) according to one of claims 1 to 9, further comprising: a state information acquisition unit (201) configured to acquire, from the servo control device (100), state information including a servo state including at least a position error and a transfer function of the feed calculation unit by causing the servo control device (100) to execute a predetermined machining program; an action information output unit (203) configured to output action information including setting information of the coefficients of the transfer function included in the state information to the servo control device (100); a reward output unit (2021) configured to output a value of a reward for reinforcement learning based on the position error included in the state information; and a value function updating unit (2022) configured to update an action value function based on the value of the reward issued by the reward issuing unit (2021), the state information, and the action information. [11] The machine learning device (200) according to claim 10, wherein the reward output unit (2021) outputs the value of the reward based on an absolute value of the position error. [12] Machine learning device (200) according to claim 10 or 11, further comprising: an optimization action information output unit configured to generate and output correction information of the coefficients of the transfer function of the feed calculation unit based on the value function updated by the value function updating unit. [13] Control device comprising: the machine learning device (200) according to any one of claims 1 to 12; and a servo control device (100) configured to control a servo motor (300) configured to drive an axis of a machine tool, a robot, or an industrial machine using feed control by a feed calculation unit having an IIR filter (1092, 1102). [14] The control device according to claim 13, wherein the machine learning device (200) is included in the servo control device (100). [15] A machine learning method of a machine learning device (200) configured to perform machine learning with respect to the optimization of coefficients of a transfer function of an IIR filter (1092, 1102) with respect to a servo control device (100) configured to control a servo motor (300) configured to drive an axis of a machine tool, a robot, or an industrial machine with a feed controller having the IIR filter (1092, 1102), the machine learning method comprising representing a zero point at which the transfer function of the IIR filter (1092, 1102) is zero and a pole at which the transfer function diverges to infinity in polar coordinates using a radius r and an angle θ, and searching and learning the radius r and the angle θ within a predetermined search range,in order to optimize the coefficients of the transfer function of the IIR filter (1092, 1102).

Citation Information

Patent Citations

  • Control device and control method of servomotor

    JP2003023785A

  • Robots and spot welding robots with learning control function

    DE102012104194A1