Machine learning device, control system and machine learning method

Abstract

A machine learning device (130, 130A) that performs machine learning of optimizing coefficients of a filter (110) provided in a motor control device (100) that controls the rotation of a motor (127) based on measurement data of an external measuring instrument (300) provided outside the motor control device (100) and a control command input to the motor control device (100), the device (130, 130A) comprising: a state information acquisition unit (131) that acquires state information including the measurement data, the control command and the coefficients of the filter (110); an action information output unit (133) that outputs action information including adjustment information of the coefficients included in the state information to the filter (110); a reward output unit (1321) that outputs a reward value for reinforcement learning using a value function based on a difference between the measurement data and the control command; and a value function updating unit (1322) that updates an action value function based on the reward value output by the reward output unit (1321), the state information, and the action information.

Description

Background of the inventionField of the invention

[0001] The present invention relates to a machine learning apparatus that performs machine learning of optimizing coefficients of a filter provided in a motor control apparatus that controls the rotation of a motor, a machine tool, a robot, or an industrial machine, and relates to a control system including the machine learning apparatus and a machine learning method. Related technology

[0002] A device that automatically adjusts characteristics of a filter is disclosed, for example, in Patent Document 1 and Patent Document 2. Patent Document 1 discloses a servo actuator that superimposes an AC signal obtained by sweeping frequencies on a speed command signal during a tuning mode, detects an amplitude of a torque command signal obtained by a speed control unit as a result of the superimposition, and sets the frequency of the torque command signal when the rate of change of the amplitude changes from positive to negative as the center frequency of a notch filter.

[0003] Patent Document 2 discloses a servo actuator including a speed feedback loop that controls the speed of a motor, and in which a notch filter means is inserted into the speed feedback loop to remove mechanical resonance. The servo actuator includes: data collecting means that collects data indicative of the frequency characteristics of the speed feedback loop; moving average means that calculates a moving average of the data collected by the data collecting means; comparing means that compares the data obtained by the moving average with the data obtained by the data collecting means to extract the resonance characteristics of the speed feedback loop.and a notch filter adjusting means that adjusts the frequency and Q value of the notch filter means based on the resonance characteristics extracted by the comparing means;

[0004] DE 10 2012 104 194 A1 concerns a robot and spot welding process with self-learning control function.

[0005] US 2015 / 0 355 628 A1 concerns a servo control that controls a machine tool or another machine.

[0006] US 2003 / 0 158 709 A1 relates to a device and a method for parameter estimation.

[0007] KOBER, J.; BAGNELL, JA; PETERS, J.: Reinforcement learning in robotics: A survey. Int. Journal of Robotics Research (IJRR), Vol. 32, 2013, pp. 1238-1274 - ISSN 1741-3176 concerns reinforcement learning for robots. Patent Document 1: JP H05-19 858 A Patent document 2: JP 2009- 104 439 A Overview of the invention

[0008] The servo actuator of Patent Document 1 adjusts the characteristics of the notch filter using the torque command value signal, and the servo actuator of Patent Document 2 adjusts the characteristics of the notch filter based on the frequency characteristics of the speed feedback loop. However, when determining the characteristics of the notch filter, it is necessary to determine a plurality of parameters such as a center frequency and a bandwidth of a band to be removed, and it is not easy to calculate their optimal values.

[0009] In the servo actuator of Patent Document 1, the means that superimposes the AC signal obtained by sweeping frequencies with the speed command value signal and the means that detects the amplitude of the torque command value signal obtained by the speed control unit as a result of the superimposition must be provided separately from a servo control unit, and a circuit arrangement becomes complex.In the servo actuator of Patent Document 2, the data collecting means that collects data indicative of the frequency characteristics of the speed feedback loop, the moving average means that calculates a moving average of the data collected by the data collecting means, and the comparing means that compares the data obtained by the moving average and the data obtained by the data collecting means to extract the resonance characteristics of the speed feedback loop must be provided separately from a servo control circuit, and a circuit arrangement becomes complex.

[0010] An object of the present invention is to provide a machine learning apparatus capable of facilitating setting parameters that determine the characteristics of a filter, detaching an external measuring instrument after machine learning is performed, reducing costs, and improving reliability, and to provide a control system incorporating the machine learning apparatus and a machine learning method.

[0011] (1) A machine learning device according to the present invention is a machine learning device (for example, a machine learning unit 130 described below) that performs machine learning of optimizing coefficients of a filter (for example, a filter 110 described below) provided in a motor control device (for example, a motor control device 100 described below) that controls the rotation of a motor (for example, a servo motor 127 described below), based on measurement data of an external measurement instrument (for example, an acceleration sensor 300 described below) provided outside the motor control device and a control command input to the motor control device.

[0012] (2) In the machine learning device according to (1), the measurement data of the external measuring instrument may include a position, a speed and / or an acceleration.

[0013] (3) In the machine learning apparatus according to (1) or (2), the motor control device may include a position feedback loop and / or a speed feedback loop, and the filter may be provided outside the position feedback loop or the speed feedback loop.

[0014] (4) In the machine learning device according to (1) or (2), the motor control device may have a feedback loop, and the measurement data of the external measuring instrument may not be used for feedback control of the feedback loop.

[0015] (5) In the machine learning device according to any one of (1) to (4), the external measuring instrument can be detached after adjusting the filter by machine learning.

[0016] (6) The machine learning device according to any one of (1) to (5) may further include: a state information acquisition unit (for example, a state information acquisition unit 131 described below) that acquires state information including the measurement data, the control command, and the coefficients of the filter; an action information output unit (for example, an action information output unit 133 described below) that outputs action information including adjustment information of the coefficients included in the state information to the filter; a reward output unit (for example, a reward output unit 1321 described below) that outputs a reward value of reinforcement learning using a value function based on a difference between the measurement data and the control command;and a value function updating unit (for example, a value function updating unit 1322 described below) that updates an action value function based on the reward value output by the reward output unit, the state information, and the action information;

[0017] (7) The machine learning apparatus according to (6) may further include: an optimization action information output unit (for example, an optimization action information output unit 135 described below) that outputs adjustment information of the coefficients based on the value function updated by the value function updating unit.

[0018] (8) A control system according to the present invention is a control system including: a motor control device (for example, a motor control device 100 described below) that includes the machine learning device (for example, a machine learning unit 130 described below) according to any one of (1) to (7), a motor (for example, a servo motor 127 described below), and a filter (for example, a filter 110 described below), and controls the rotation of the motor; and an external measuring instrument (for example, an acceleration sensor 300 described below) provided outside the motor control device.

[0019] (9) A machine learning method according to the present invention is a machine learning method of a machine learning device that performs machine learning of optimizing coefficients of a filter provided in a motor control device that controls rotation of a motor, based on measurement data of an external measuring instrument provided outside the motor control device and a control command input to the motor control device.

[0020] According to the present invention, it is easy to adjust the coefficients (parameters) that determine the characteristics of a filter. Furthermore, since the external measuring instrument is located outside an engine control device, it is possible to remove the external measuring instrument after machine learning has been performed, thereby reducing costs and improving reliability. Short description of the drawings Fig. 1 is a block diagram illustrating a control system including a motor control device according to an embodiment of the present invention, a machine tool, and an acceleration sensor. Fig. 2 is a diagram for describing an operation of a motor when a moving path of a table is circular. Fig. 3 is a diagram for describing an operation of a motor when a moving path of a table is rectangular. Fig. 4 is a diagram for describing an operation of a motor when a moving path of a table is octagonal. Fig. 5 is a diagram for describing an operation of a motor when a moving path of a table has a shape in which the corners of an octagon are alternately replaced by arcs. Fig. 6 is a block diagram illustrating a machine learning unit according to an embodiment of the present invention. Fig. 7 is a flowchart for describing an operation of a machine learning unit according to an embodiment of the present invention. Fig. 8 is a flowchart for describing an operation of an optimization action information output unit of the machine learning unit according to an embodiment of the present invention. Fig. Fig. 9 is an explanatory diagram illustrating a state in which a scale is attached to a table of a machine body. Fig. 10 is a block diagram illustrating an example in which a filter is formed by directly connecting a plurality of filters. Fig. Figure 11 is a block diagram illustrating another control system design example. Detailed description of the invention

[0021] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

[0022] Fig. 1 is a block diagram illustrating a motor control device according to an embodiment of the present invention, a machine tool, and a control system including an acceleration sensor. As shown in Fig. 1, a control system 10 includes a motor control device 100, a machine tool 200 controlled by the motor control device 100, and an acceleration sensor 300 mounted on the machine tool 200. The acceleration sensor 300 is an external measuring instrument provided outside the motor control device 100, and a measured acceleration is measurement data. Although a machine tool is selected and described as the control target of the motor control device 100, the control target is not limited to a machine tool and may be, for example, a robot, an industrial machine, or the like. The motor control device 100 may be provided as part of a control target such as a machine tool, a robot, or an industrial machine.

[0023] The motor control device 100 includes a filter 110, a servo control unit 120, and a machine learning unit 130. Although the motor control device 100 in this example includes the servo control unit 120 that controls a servo motor, the motor control device 100 may include a control unit that controls a spindle motor and does not perform feedback control. The filter 110 is a filter of the machine tool 200 and uses, for example, a notch filter, a filter that sets an acceleration or deceleration time constant, or an inverse characteristic filter. A position command is input to the filter 110, and the filter 110 serves as a position command value shaping device that performs shaping of the input position command.The position command is generated by a host controller or an external input device according to a predetermined machining program to change a pulse frequency to change the speed of a servo motor 127. The position command serves as a control command. Although the filter 110 is provided outside the servo control unit 120 (that is, outside a position feedback loop and a speed feedback loop described below), the filter 110 may be provided within a position feedback loop or a speed feedback loop of the servo control unit 120. For example, the filter 110 may be connected to an output side of a speed control unit 126 (described below) or an output side of an adder 123 of the servo control unit 120.However, in order to suppress oscillation outside a control loop (a position feedback loop or a speed feedback loop) of the servo control unit 120, the filter is preferably provided outside the position feedback loop or the speed feedback loop. In . Fig. 1, the filter 110 is arranged upstream of a subtractor 121 (described below) that calculates a position error. Although the design of the filter 110 is not particularly limited, the filter is preferably a second-order or higher-order IIR filter.

[0024] Expression 1 (hereinafter referred to as Math. 1) gives a transfer function F(ρ,s) of a notch filter as filter 110. The parameter ρ specifies coefficients ω, ζ, and R. In Example 1, the coefficient R is an attenuation coefficient, the coefficient ω is a center angular frequency, and the coefficient ζ is a specific bandwidth. When the center frequency is fc and the bandwidth is fw, the coefficient ω is represented as ω = 2πfc, and the coefficient ζ is represented as ζ = fw / fc. F(ρ,s)=s2+2ζRωs+ω2s2+2ζωs+ω2

[0025] The servo control unit 120 includes a subtractor 121, a position control unit 122, an adder 123, a position feedforward control unit 124, a subtractor 125, a speed control unit 126, a servo motor 127, a rotary encoder 128 serving as a position detection unit associated with the servo motor 127, and an integrator 129. The subtractor 121, the position control unit 122, the adder 123, the subtractor 125, the speed control unit 126, the servo motor 127, the rotary encoder 128, and the integrator 129 form a position feedback loop. In addition, the subtractor 125, the speed control unit 126, the servo motor 127 and the rotary encoder 128 form a speed feedback loop.

[0026] The subtractor 121 calculates a difference between a position command after shaping an output from the filter 110 and a position feedback detection position, and outputs the difference as a position error to the position control unit 122 and the position feedforward control unit 124.

[0027] The position control unit 122 outputs a value obtained by multiplying the position error by a position gain Kp as a speed command value to the adder 123. The position feedforward control unit 124 performs a position feedforward process represented by a transfer function G(s) indicated by Expression 2 (hereinafter indicated by Math. 2) with respect to a value obtained by differentiating the position command value and multiplying it by a constant α, and outputs the processing result thereof as a position feedforward term to the adder 123. For the coefficients a i and b j (X≥i, j≥0, X is a natural number) in Expression 2 are the coefficients of the transfer function G(s). G(s)=b0+b1s+b2s2+⋯+bxsxa0+a1s+a2s2+⋯+axsx

[0028] The adder 123 adds the speed command value and an output value (a position feedforward term) of the position feedforward control unit 124 and outputs an addition result as a feedforward speed command value to the subtracter 125. The subtracter 125 calculates a difference between the output of the adder 123 and the feedback speed detection value and outputs the difference as a speed error to the speed control unit 126.

[0029] The speed control unit 126 adds a value obtained by multiplying and integrating the speed error by an integral gain K1v and a value obtained by multiplying the speed error by a proportional gain K2v, and outputs an addition result as a torque command to the servo motor 127.

[0030] A rotation angle position of the servo motor 127 is detected by the rotary encoder 128, and a speed detection value is input as speed feedback (a speed RK) to the subtractor 125. The speed detection value is integrated by the integrator 129 as a position detection value, and the position detection value is input as position feedback (a position RK) to the subtractor 121. The servo control unit 120 is configured in this manner.

[0031] Before describing the machine learning unit 130, the machine tool 200 and the acceleration sensor 300 mounted on the machine tool 200 will be described next. The machine tool 200 includes a ball screw 230 connected to a rotation axis of the servo motor 127, a nut 240 screwed into the ball screw 230, and a machine body 250 including a table 251 connected to the nut. With the rotation of the servo motor 127, the nut 240 screwed into the ball screw 230 moves in an axial direction of the ball screw 230.

[0032] In the machine tool 200, when the table 251 on which a workpiece (a work) is mounted is moved in an x-axis direction and a y-axis direction shown in Fig. 1 is provided in the x-axis direction and the y-axis direction, respectively. When the table is moved in the directions of three or more axes, the motor control device 100 is provided in the respective axial directions.

[0033] The acceleration sensor 300 is provided outside the servo control unit 120 and, in this example, is attached to the machine body 250. The acceleration sensor serves as an external measuring instrument. Single-axis, dual-axis, and triple-axis acceleration sensors are known as acceleration sensors, and these acceleration sensors can be selected as needed. For example, a dual-axis acceleration sensor is used when the table of the machine body 250 is moved in the x-axis and y-axis directions, and a triple-axis acceleration sensor is used when the table of the machine body 250 is moved in the x-axis, y-axis, and z-axis directions. The acceleration sensor 300 is preferably provided at a location near a machining point.The acceleration sensor 300 measures an acceleration of the machine body 250 and outputs the measured acceleration to the machine learning unit 130. If the acceleration sensor 300 is used only during machine learning, the machine learning may be performed before delivery to adjust the coefficients of the filter 110, and the acceleration sensor 300 may be detached from the machine body 250 after the filter 110 has been adjusted. If relearning is performed after delivery, the acceleration sensor may be detached after the relearning is performed. Although the acceleration output from the acceleration sensor 300 can be used for feedback control of the servo control unit 120, the acceleration sensor 300 may be detached unless the acceleration is used for feedback control.In this case, it is possible to reduce the cost of the machine tool 200 and improve reliability. <Einheit 130 für maschinelles Lernen>

[0034] The machine learning unit 130 executes a predetermined machining program (hereinafter also referred to as a "machining program during learning") and performs machine learning (hereinafter also referred to as learning) on ​​the coefficients ω, ζ, and R of the transfer function of the filter 110 using the position command and the acceleration measurement value from the acceleration sensor 300. The machine learning unit 130 serves as a machine learning device. Although the learning of the machine learning unit 130 is performed before delivery, relearning may be performed after delivery. Here, the motor control device 100 drives the servo motor 127 using the machining program during learning and moves the table 251 in a state where no workpiece (no work) is mounted.A trajectory of an arbitrary position of the table 251 moved in the x-axis direction and in the y-axis direction is circular, rectangular, octagonal, or has a shape in which the corners of an octagon are alternately replaced by arcs. Fig. 2 to 5 are diagrams for describing the operation of a motor when the motion path of a table is circular, rectangular, octagonal, or has a shape in which the corners of an octagon are alternately replaced by arcs. Fig. 2 to 5, it is assumed that the table 251 moves in a clockwise direction in the x-axis direction and in the y-axis direction.

[0035] If the movement path of the table 251 is circular, as in Fig. 2, the speed of the servo motor that moves the table in the y-axis direction decreases at the speed shown in Fig. 2, gradually decreases as it approaches position A1, and gradually increases after passing through position A1, with the rotation direction being reversed at position A1. The table then moves in a linearly reversed manner in the y-axis direction with position A1 in between. In contrast, the servo motor that moves the table in the x-axis direction rotates at position A1 at the same speed as the speed before and after position A1, and the table moves at the same speed as the speed before and after position A1 in the x-axis direction. Fig. At position A2 illustrated in Figure 2, the respective servo motors are controlled so that the operation of the servo motor that moves the table in the x-axis direction and the operation of the servo motor that moves the table in the y-axis direction are reversed.

[0036] If the movement path of the table 251 is rectangular, as in Fig. 3, the speed of the servo motor that moves the table in the x-axis direction is measured at the Fig. 3, the position B1 is abruptly reversed, and the table moves so as to abruptly reverse linearly in the x-axis direction with the position B1 in between. In contrast, the servo motor that moves the table in the y-axis direction rotates at the same speed as the speed before and after the position B1 at the position B1, and the table moves at the same speed as the speed before and after the position B1 in the y-axis direction. Fig. At position B2 illustrated in Figure 3, the servo motors are controlled so that the operation of the servo motor that moves the table in the x-axis direction and the operation of the servo motor that moves the table in the y-axis direction are reversed.

[0037] If, as in Fig. 4, the movement path of the table 251 is octagonal, as in Fig. 4, at corner position C1, the speed of the motor that moves the table in the y-axis direction decreases, and the speed of the motor that moves the table in the x-axis direction increases. At corner position C2, the rotation direction of the motor that moves the table in the y-axis direction is reversed, and the table moves so that it is linearly reversed in the y-axis direction. Moreover, the motor that moves the table in the x-axis direction rotates at a constant speed in the same rotation direction from position C1 to position C2 and from position C2 to position C3. At corner position C3, the speed of the motor that moves the table in the y-axis direction increases, and the speed of the motor that moves the table in the x-axis direction decreases.At corner position C4, the rotation direction of the motor that moves the table in the x-axis direction is reversed, and the table moves in a linearly reversed direction. Furthermore, the motor that moves the table in the y-axis direction rotates at a constant speed in the same direction from position C3 to position C4 and from position C4 to the next corner position.

[0038] If the movement path of the table 251 has a shape in which the corners of the octagon are alternately replaced by arcs, as in Fig. 5, the speed of the motor that moves the table in the y-axis direction decreases at the corner position D1, and the speed of the motor that moves the table in the x-axis direction increases. At the arc position D2, the direction of rotation of the motor that moves the table in the y-axis direction is reversed, and the table moves so that it is linearly reversed in the y-axis direction. In addition, the motor that moves the table in the x-axis direction rotates at a constant speed in the same direction of rotation from position D1 to position D3. Unlike the case where the Fig. 4 is octagonal, the rotational speed of the motor that moves the table in the y-axis direction gradually decreases as it approaches the position D2, so that a movement trajectory of a circular arc is formed in front of and behind the position D2, and the rotational speed gradually increases after passing the position D2, with the rotation being stopped at the position D2. At the corner position D3, the rotational speed of the motor that moves the table in the y-axis direction increases, and the rotational speed of the motor that moves the table in the x-axis direction decreases. At the position D4 of a circular arc, the rotational direction of the motor that moves the table in the x-axis direction is reversed, and the table moves so as to be linearly reversed in the x-axis direction.In addition, the motor that moves the table in the y-axis direction rotates at a constant speed in the same direction of rotation from position D3 to position D4 and from position D4 to the next corner position. The speed of the motor that moves the table in the x-axis direction gradually decreases as it approaches position D4, forming a circular arc trajectory in front of and behind position D4. After passing position D4, the speed gradually increases, stopping rotation at position D4.

[0039] In the present embodiment, vibration generated when the rotation direction of the x-axis direction or the y-axis direction is reversed at positions A1 and A2, positions B1 and B2, positions C2 and C4, and positions D2 and D4 of the trajectory determined by the machining program during learning can be measured using the acceleration sensor 300. Furthermore, vibration generated when the rotational speed is changed during linear control in which the rotation direction is not reversed at positions C1 and C3 and positions D1 and D3 can be measured using the acceleration sensor 300. As a result, it is possible to perform machine learning of the coefficients of the filter 110 so as to suppress vibration.

[0040] The following describes the machine learning unit 130 in more detail. Although the following description describes a case where the machine learning unit 130 performs reinforcement learning, the learning performed by the machine learning unit 130 is not particularly limited to reinforcement learning, and the present invention can also be applied to a case where the machine learning unit 130 performs, for example, supervised learning.

[0041] Before describing the respective functional blocks included in the machine learning unit 130, a basic mechanism of reinforcement learning will be described. An agent (which in the present embodiment corresponds to the machine learning unit 130) observes an environmental state and selects a specific action. The environment then changes based on the action. A specific reward is given in accordance with the environmental change, and the agent learns choices (decisions) of a better action. While supervised learning provides a perfectly correct answer, the reward in reinforcement learning often provides a fragmented value based on a change in a part of the environment. Therefore, the agent learns to select an action such that the total reward is maximized in the future.

[0042] In this way, reinforcement learning learns a method for learning an appropriate action based on the interaction of an action with the environment (i.e., an action to maximize the reward to be achieved in the future) by learning an action. This means that in the present embodiment, such an action that affects the future, for example, an action for selecting action information for suppressing vibration of a machine end, is achieved.

[0043] Although any learning method can be used as reinforcement learning, the following description uses Q-learning as an example. This method is a method for learning a value function Q(S,A) for selecting an action A at a given environmental state S. One goal of Q-learning is to select an action A with the highest value function Q(S,A) as the optimal action from among actions A that can be taken in a given environmental state S.

[0044] However, at an initial time when Q-learning begins, the correct value of Q(S,A) for a combination of state S and action A is not known at all. Therefore, the agent learns the correct value Q(S,A) by selecting different actions A in a given state S and making a better selection of actions based on the rewards received for the selected actions A.

[0045] Since the agent wants to maximize the total reward achieved in the future, Q-learning aims to ultimately achieve a relationship Q(S,A) = E[L(γ t )r t ] to achieve. Here, E[ ] denotes an expected value, t denotes a time, γ represents a parameter described below, called the discount factor, rt is a reward at time t, and Σ is the sum at time t. In this expression, the expected value is an expected value when the state has been changed according to an optimal action. However, since it is unclear which action would be optimal in the process of Q-learning, reinforcement learning is performed while searching for an optimal action by performing various actions. An update expression of such a value Q(S,A) can be represented by the following Expression 3 (hereinafter denoted by Math. 3). Q(St+1,At+1)←Q(St,At)+α(rt+1+γmaxAQ(St+1,A)−Q(St,At))

[0046] In expression 2, S t a state of the environment at time t, and A t specifies an action at time t. The action A t the state changes to S t+1 . r t+1 indicates a reward achieved by changing the state. Furthermore, a term with max is a multiplication of the value Q by y if an action A, which has the highest value Q known at that time, is in the state S t+1 is selected. Here, γ is a parameter 0<γ≤1 and is called the discount rate. Furthermore, α is a learning coefficient and lies in the range 0<α≤1.

[0047] Expression 2 gives a method for updating a value Q(S t , A t ) of an action A t in a state S t based on a reward r t+1 which was offered in return when Action A tThis update expression indicates that if the value max a Q(S t+1 ,A) the best action in the next state S t+1 , which corresponds to an action A t is greater than the value Q(St,At) of an action A t in the state S t is, Q(St,At) is increased and Q(St,At) is decreased if it is smaller. That is, the update expression approximates the value of a particular action in a particular state close to the value of the best action in the next state corresponding to the action. However, this difference varies depending on the discount rate γ and the reward r t+1 differs, the value of the best action in a given state is essentially propagated to the value of an action in a state prior to that state.

[0048] Here, a Q-learning method is known for constructing a table of value functions Q(S,A) for all state-action pairs (S,A) to perform learning. However, Q-learning can take quite a long time to converge because the number of states is too large to calculate the Q(S,A) values ​​of all state-action pairs.

[0049] Therefore, Q-learning can utilize an existing technique called Deep Q Network (DQN). Specifically, DQN calculates the value of Q(S,A) by constructing a value function Q using a suitable neural network and approximating the value function Q with the appropriate neural network by adjusting the parameters of the neural network. Using DQN, it is possible to shorten the time required for Q-learning to converge. The details of DQN are disclosed, for example, in the following non-patent document. <nichtpatentdokument>

[0050] MNIH, Volodymyr [ua]: Human-level control through deep reinforcement learning. In: Nature, Vol. 518, 2015, H. 7540, pp. 529-533. - ISSN 0028-0836 (P), 1476-4687 (E). DOI: 10.1038 / nature14236. URL: https: / / files.davidqiu.com / research / nature14236.pdf [accessed on June 28, 2025]

[0051] The machine learning unit 130 performs the Q-learning described above. Specifically, the machine learning unit 130 learns a value Q for selecting an action A for adjusting the values ​​of the coefficients ω, ζ, and R of the transfer function of the filter 110 associated with a state S. The state S includes the values ​​of the coefficients ω, ζ, and R of the transfer function of the filter 110, the measured acceleration from the acceleration sensor 300 obtained by executing the machining program during learning, and the position command.

[0052] The machine learning unit 130 observes the state information S, which includes the measured acceleration from the acceleration sensor 300 and the position command, by executing one or more processing programs during learning based on the coefficients ω, ζ, and R of the transfer function of the filter 110 to determine the action A. The machine learning unit 130 receives a reward each time the action A is performed. The machine learning unit 130 searches for the optimal action A through trial and error so that the total reward is maximized over the future.Thereby, the machine learning unit 130 can select an optimal action A (that is, the optimal coefficients ω, ζ and R of the transfer function of the filter 110) with respect to the state S including the measured acceleration from the acceleration sensor 300 obtained by executing the machining program during learning based on the coefficients ω, ζ and R of the transfer function of the filter 110 and the position command.

[0053] That is, the machine learning unit 130 can select such an action A (that is, the coefficients ω, ζ and R of the transfer function of the filter 110) that minimizes the vibration of a machine end generated when a machining program is executed during learning by selecting such an action A that maximizes the value Q from the actions A applied to the coefficients ω, ζ and R of the transfer function of the filter 110 associated with a certain state S based on the value function Q learned by the machine learning unit 130.

[0054] Fig. 6 is a block diagram illustrating the machine learning unit 130 according to an embodiment of the present invention. As shown in Fig. 6, the machine learning unit 130 for performing the reinforcement learning described above includes a state information acquisition unit 131, a learning unit 132, an action information output unit 133, a value function storage unit 134, and an optimization action information output unit 135. The learning unit 132 includes a reward output unit 1321, a value function update unit 1322, and an action information generation unit 1323.

[0055] The state information acquisition unit 131 acquires the state S, which includes the position command and the measured acceleration from the acceleration sensor 300, which is obtained by executing the processing program during learning based on the coefficients ω, ζ, and R of the transfer function of the filter 110. The state information S corresponds to a state S of the environment in Q-learning. The state information acquisition unit 131 outputs the acquired state information S to the learning unit 132.

[0056] The coefficients ω, ζ, and R of the transfer function of the filter 110 at the time when Q-learning first starts are generated in advance by a user. In the present embodiment, the machine learning unit 130 adjusts the initial setting values ​​of the coefficients ω, ζ, and R of the transfer function of the filter 110, which are created by the user, to optimal values ​​through reinforcement learning. When a machine tool is adjusted in advance by an operator, the adjusted values ​​of the coefficients ω, ζ, and R can be machine-learned as initial values.

[0057] The learning unit 132 is a unit that learns the value Q(S,A) when a certain action A is selected in a certain environmental state S.

[0058] The reward output unit 1321 is a unit that calculates a reward when action A is selected at a certain state S. Here, a measured acceleration, which is a state variable of the state S, is denoted by y(S), a position command, which is a state variable associated with the state information S, is denoted by r(S), a measured acceleration, which is a state variable associated with the state information S', which is changed due to the action information A (corrections of the coefficients ω, ζ, and R of the transfer function of the filter 110) from the state S, is denoted by y(S'), and a position command, which is a state variable associated with the state information S', is denoted by r(S').

[0059] For example, Expression 4 (hereafter referred to as Math. 4) can be used as a value function f. Expression 4 indicates that the value function f is a time integration of a square of an absolute value of a difference between a two-fold differentiation of the position command r and the measured acceleration y. f(r,y)=∫|d2rdt2−y|2dt

[0060] A time integration of an absolute value of an expression (d 2 r / dt 2 -y), a time integration of a time(t)-weighted absolute value of the expression (d 2 r / dt 2 -y) and a largest value of a set of absolute values ​​of the expression (d 2 r / dt 2 -y) can be used as a value function.

[0061] In this case, when a value function f(r(S'),y(S')) when operating the motor control device 100 based on the filter 110 after correction related to the state information S' corrected by the action information A is larger than a value function f(r(S),y(S)) when operating the motor control device 100 based on the filter 110 before correction related to the state information S before being corrected by the action information A, the reward output unit 1321 sets a reward value to a negative value.

[0062] On the other hand, when a value function f(r(S'),y(S')) when operating the motor control device 100 based on the filter 110 after correction associated with the state information S' corrected by the action information A is smaller than a value function f(r(S),y(S)) when operating the motor control device 100 based on the filter 110 before correction associated with the state information S before being corrected by the action information A, the reward output unit 1321 sets a reward value to a positive value.Moreover, when a value function f(r(S'),y(S')) when operating the motor control device 100 based on the filter 110 after correction related to the state information S' corrected by the action information A is the same as a value function f(r(S),y(S)) when operating the motor control device 100 based on the filter 110 before correction related to the state information S before being corrected by the action information A, the reward output unit 1321 sets a reward value to zero.

[0063] Furthermore, when the evaluation function f(r(S'),y(S')) of state S' after performing action A is larger than the evaluation function f(r(S),y(S)) of the previous state S, the negative value may increase proportionally. That is, the negative value may increase according to the degree of increase of the evaluation function f(r(S'),y(S')). Conversely, when the evaluation function f(r(S'),y(S')) of state S' after performing action A is smaller than the evaluation function f(r(S),y(S)) of the previous state S, the positive value may increase proportionally. That is, the positive value may increase according to the degree of decrease of the evaluation function f(r(S'),y(S')).

[0064] The value function updating unit 1322 updates the value function Q stored in the value function storage unit 134 by performing Q-learning based on the state S, the action A, the state S' when the action A was applied to the state S, and the reward value calculated in this way. Updating the value function Q can be performed through online learning, batch learning, or mini-batch learning. Online learning is a learning method for applying a specific action A to a current state S and updating the value function Q immediately each time the current state S transitions to a new state S'.Batch learning is a learning technique that applies a specific action A to a current state S and repeatedly achieves a transition from state S to a new state S', collects learning data, and updates the value function Q using all of the collected learning data. Mini-batch learning is a learning technique that lies between online learning and batch learning and involves updating the value function Q each time a certain amount of learning data has been collected.

[0065] The action information generation unit 1323 selects the action A in the Q-learning process with respect to the current state S. The action information generation unit 1323 generates action information A and outputs the generated action information A to the action information output unit 133 to perform a process (corresponding to the action A of the Q-learning) for correcting the coefficients ω, ζ, and R of the transfer function of the filter 110 in the Q-learning process. More specifically, the action information generation unit 1323 incrementally adds or subtracts the coefficients ω, ζ, and R of the transfer function of the filter 110 included in the action A with respect to the coefficients ω, ζ, and R of the transfer function of the filter 110 included in the state S, for example.

[0066] When the coefficients ω, ζ, and R of the transfer function of the filter 110 are increased or decreased, the state S transitions to the state S', and a plus reward (a positive reward) is offered in return, the action information generation unit 1323 may select a strategy for selecting such an action A' that further decreases the value of the value function f, such as incrementally increasing or decreasing the coefficients ω, ζ, and R of the transfer function of the filter 110, similarly to the previous action, as the next action A'.

[0067] On the other hand, when a minus reward (a negative reward) is offered in return, the action information generation unit 1323 may, for example, select a strategy for selecting such an action A' that decreases the value function f to be smaller than the previous value, for example, by incrementally decreasing or increasing the coefficients ω, ζ, and R of the transfer function of the filter 110 opposite to the previous action, as the next action A'.

[0068] The action information generation unit 3023 may select a strategy for selecting the action A' according to a known method such as a greedy method for selecting an action A' having the highest value function Q(S,A) from the values ​​of currently estimated actions A and an ε-greedy method for randomly selecting an action A' with a certain low probability ε and, in other cases, selecting an action A' having the highest value function Q(S,A).

[0069] The action information output unit 133 is a unit that sends the action information A output from the learning unit 132 to the filter 110. As described above, the filter 110 finely adjusts the current state S (i.e., the currently adjusted coefficients ω, ζ, and R) based on the action information, thereby transitioning to the next state S' (i.e., the corrected coefficient of the filter 110).

[0070] The value function storage unit 134 is a storage device that stores the value function Q. The value function Q can be stored as a table (hereinafter referred to as an action value table), for example, for each state S and each action A. The value function Q stored in the value function storage unit 134 is updated by the value function updating unit 1322. Furthermore, the value function Q stored in the value function storage unit 134 can be shared with other machine learning units 130. When the value function Q is shared by a plurality of machine learning units 130, since reinforcement learning can be performed so as to be distributed to the respective machine learning units 130, it is possible to improve the efficiency of reinforcement learning.

[0071] The optimization action information output unit 135 generates the action information A (hereinafter referred to as "optimization action information") to cause the filter 110 to perform an operation for maximizing the value function Q(S,A) based on the value function Q updated by the value function updating unit 1322 that performs Q-learning. More specifically, the optimization action information output unit 135 acquires the value function Q stored in the value function storage unit 134. As described above, the value function Q is updated by the value function updating unit 1322 that performs Q-learning. The optimization action information output unit 135 generates the action information based on the value function Q and outputs the generated action information to the filter 110.The optimization action information includes information that corrects the coefficients ω, ζ, and R of the transfer function of the filter 110 similarly to the action information that the action information output unit 133 outputs in the process of Q-learning.

[0072] In the filter 110, the coefficients ω, ζ, and R of the transfer function are corrected based on the action information. With the above-described operations, the machine learning unit 130 can optimize the coefficients ω, ζ, and R of the transfer function of the filter 110 and operate to suppress vibration of a machine end. As described above, it is possible to simplify the adjustment of the parameters of the filter 110 using the machine learning unit 130.

[0073] The functional blocks included in the motor control device 100 have been described above. To implement these functional blocks, the motor control device 100 includes an arithmetic processing unit such as a central processing unit (CPU). The motor control device 100 further includes an auxiliary storage device such as a hard disk drive (HDD) for storing various control programs such as application software or an operating system (OS), and a main storage device such as a random access memory (RAM) for storing data temporarily required when the arithmetic processing device executes a program.

[0074] In the engine control device 100, the arithmetic processing device reads an application software and an OS from the auxiliary storage device and develops the read application software or OS in the main storage device to perform arithmetic processing based on the read application software or OS. The arithmetic processing device also controls various types of hardware provided in each device based on the arithmetic result. In this way, the functional blocks of the present embodiment are implemented. That is, the present embodiment can be implemented through the cooperation of hardware and software.

[0075] Since the machine learning unit 130 incurs a large amount of computation related to machine learning, graphics processing units (GPUs) may be provided on a personal computer and used for arithmetic processing related to machine learning using a technique called general-purpose computing on graphics processing units (GPGPUs). In this way, high-speed processing can be performed. Furthermore, in order for the machine learning unit 130 to perform higher-speed processing, a computer network may be created using a plurality of computers equipped with such GPUs, and the plurality of computers included in the computer network may perform parallel processing.

[0076] Next, an operation of the machine learning unit 130 during Q-learning according to the present embodiment will be described with reference to the flowcharts in Fig. 7 described.

[0077] In step S11, the state information acquisition unit 131 acquires state information S from the motor control device 100. The acquired state information S is output to the value function update unit 1322 and the action information generation unit 1323. As described above, the state information S is information corresponding to the state of Q-learning and includes the coefficients ω, ζ, and R of the transfer function of the filter 110 at the time of step S11. In this way, a position command r(S) and a measured acceleration y(S) corresponding to a predetermined feed rate and the shape of a trajectory are acquired when the coefficients of the transfer function of the filter 110 are initial values.

[0078] The position command r(S0) in state S0 at a time when Q-learning first begins and the measured acceleration y(S0) from the acceleration sensor 300 are obtained by operating the motor control device 100 according to the machining program during learning. The position command input to the motor control device 100 is a position command corresponding to a predetermined trajectory determined by the machining program (for example, the trajectory shown in Fig. 4 and Fig. 5). The position command is input to the filter 110 and the machine learning unit 130. The initial values ​​of the coefficients ω, ζ, and R of the transfer function of the filter 110 are generated in advance by a user, and the initial values ​​of the coefficients ω, ζ, and R are sent to the machine learning unit 130. The acceleration sensor 300 outputs the measured acceleration y(S0) at the respective positions, such as positions C1 to C4 and positions D1 to D4, of the trajectory to the machine learning unit 130. The machine learning unit 130 can extract the position command r(S0) and the measured acceleration y(S0) at the respective positions, such as positions C1 to C4 and positions D1 to D4, of the trajectory.

[0079] In step S12, the action information generation unit 1323 generates new action information A and outputs the generated new action information A to the filter 110 via the action information output unit 133. The action information generation unit 1323 outputs the new action information A based on the strategy described above. The motor control device 100, which has received the action information A, drives a machine tool including the servo motor 127 according to the state S' obtained by correcting the coefficients ω, ζ, and R of the transfer function of the filter 110 in relation to the current state S based on the received action information. As described above, the action information corresponds to the action A in Q-learning.

[0080] In step S13, the state information acquisition unit 131 acquires the measured acceleration y(S') from the acceleration sensor 300 and the coefficients ω, ζ, and R of the transfer function of the filter 110 in the new state S'. In this way, the state information acquisition unit 131 acquires the position command r(S') and the measured acceleration y(S') corresponding to the octagonal trajectory (specifically, the positions such as positions C1 to C4 and positions D1 to D4 of the trajectory), and the coefficients ω, ζ, and R in the state S' from the filter 110. The acquired state information is output to the reward output unit 1321.

[0081] In step S14, the reward output unit 1321 determines a magnitude ratio between the evaluation function f(r(S'),y(S')) in the state S' and the evaluation function f(r(S),y(S)) in the state S, and sets the reward to a negative value in step S15 if f(r(S'),y(S')) > f(r(S),y(S)). If f(r(S'),y(S')) < f(r(S),y(S)), the reward output unit 1321 sets the reward to a positive value in step S16. If f(r(S'),y(S')) = f(r(S),y(S)), the reward output unit 1321 sets the reward to zero in step S17. The reward output unit 1321 may apply a weight to the negative and positive reward values. The state S transitions to the state S0 at a time when Q-learning begins.

[0082] When any one of steps S15, S16, and S17 ends, the value function updating unit 1322 updates the value function Q stored in the value function storage unit 134 in step S18 based on the reward value calculated in any one of these steps. Then, the flow returns to step S11 again, and the above-described process is repeated, whereby the value function Q assumes an appropriate value. The process may end under a condition that the above-described process is repeated for a predetermined period of time. Although online updating is exemplified in step S18, batch updating or mini-batch updating may be performed instead of online updating.

[0083] In the present embodiment, due to the method described with reference to Fig. 7, it is possible to obtain a suitable value function for adjusting the coefficients ω, ζ, and R of the transfer function of the filter 110 by using the machine learning unit 130, and to simplify optimization of the coefficients ω, ζ, and R of the transfer function of the filter 110. Next, a process during the generation of the optimization action information by the optimization action information output unit 135 will be described with reference to the flowchart in Fig. 8. First, in step S21, the optimization action information output unit 135 acquires the value function Q stored in the value function storage unit 134. As described above, the value function Q is updated by the value function updating unit 1322, which performs Q-learning.

[0084] In step S22, the optimization action information output unit 135 generates the optimization action information based on the value function Q, and outputs the generated optimization action information to the filter 110.

[0085] In the present embodiment, due to the arrangement described with reference to Fig. 8, it is possible to generate optimization action information based on the value function Q obtained by the machine learning unit 130, simplify the adjustment of the currently set coefficients ω, ζ, and R of the transfer function of the filter 110 based on the optimization action information, suppress vibration of a machine end, and improve the quality of a machined surface of a workpiece. Since the external measuring instrument is disposed outside the motor control device, it is possible to remove the external measuring instrument after machine learning is performed, thereby reducing costs and improving reliability.

[0086] The servo control unit of the motor control device described above and the components included in the machine learning unit can be implemented by hardware, software, or a combination of these. The servo control method performed by the cooperation of the components included in the motor control device described above can also be implemented by hardware, software, or a combination of these. Here, "implemented by software" means an implementation where a computer reads and executes a program.

[0087] The programs can be stored on any of various types of non-transitory, computer-readable media and delivered to a computer. Non-transitory, computer-readable media includes various types of physical storage media. Examples of non-transitory, computer-readable media include a magnetic recording medium (for example, a floppy disk, a magnetic tape, and a hard disk drive), a magneto-optical recording medium (for example, a magneto-optical disk), a CD-ROM (read-only memory), a CD-R, a CD-R / W, a semiconductor memory (for example, a mask ROM, a PROM (programmable ROM), an EPROM (erasable PROM), a flash ROM, or a RAM (random access memory)). In addition, the programs can be delivered to a computer via various types of transitory, computer-readable media.

[0088] The above-described embodiment is a preferred embodiment of the present invention. However, the scope of the present invention is not limited to this embodiment, and the present invention may be embodied in various modifications without departing from the spirit of the present invention.

[0089] In the above-described embodiment, a case was described where an acceleration sensor is used as the external measurement instrument and the measurement data is acceleration information. However, a position sensor or a speed sensor can be used as the external measurement instrument to obtain position information or speed information, which can be differentiated or doubly differentiated to obtain acceleration information.

[0090] Although the value function f is a function that represents a difference (i.e. an acceleration error) between a measured acceleration y and a value d 2 r / dt 2 In addition, if the value function obtained by twice differentiating the position command is used, a function using a position error or a speed error may be used. Specifically, when a position error is used as the value function, the machine learning unit 130 may acquire a position command and a measured position from a position sensor as an external measuring instrument as state information, and may use, as the value function, a time integration of an absolute value of a difference between the position command and the measured position, a time integration of a square of an absolute value of a position error, a time integration of a time(t)-weighted absolute value of the position error, and a largest value of a set of absolute values ​​of the position error.

[0091] Furthermore, when a speed error is used as a value function, the machine learning unit 130 may acquire a position command and a measured position from a speed sensor as an external measuring instrument as state information, and may use, as a value function, a time integration of an absolute value of a difference (a speed error) between a measured speed and a value obtained by differentiating the position command, a time integration of a square of an absolute value of a speed error, a time integration of a time (t)-weighted absolute value of the speed error, and a largest value of a set of absolute values ​​of the speed error.

[0092] An example of a value function that uses a position error, a velocity error and an acceleration error is, for example, a time integration of [c a × (position error) 2 + c b × (speed error) 2 + c c × (Acceleration error) 2 ]. For the coefficients c a , c b , c c These are weighting coefficients.

[0093] When a position sensor is used as an external measuring instrument, a scale (a linear scale) is attached to a table as an external measuring instrument. Fig. 9 is an explanatory diagram illustrating a state in which a scale is attached to the table 251 of the machine body 250. In this case, a scale 301 detects the position of the table 251 and outputs position information to the machine learning unit 130. Although the above-described embodiment described a case where the machine tool 200 has one resonance point, the machine tool 200 may have a plurality of resonance points. When the machine tool 200 has a plurality of resonance points, a plurality of filters may be provided to correspond to the respective resonance points and connected in series, whereby all resonances can be attenuated. Fig. 10 is a block diagram illustrating an example in which a plurality of filters are connected in series to form a filter. In Fig. 10, when m (m is a natural number of 2 or more) resonance points exist, the filter 110 is formed by connecting m filters 110-1 to 110-m in series. Optimal values ​​for damping a resonance are sequentially calculated by machine learning with respect to the coefficients ω, ζ, and R of the m filters 110-1 to 110-m.

[0094] In the servo control unit 120 of the Fig. In the motor control device 100 illustrated in Figure 1, although an example is illustrated in which the position feedforward control unit 124 is provided only as a feedforward control unit, a further speed feedforward control unit may be provided in addition to the position feedforward control unit 124. An adder is provided on the output side of the motor control device 100 illustrated in Figure 1. Fig. 1 is provided, and a speed feedforward unit is provided between the input side of the adder and the output side of the filter 110. The adder adds the output of the speed control unit 126 and the output of the speed feedforward unit and outputs an addition result to the servo motor 127. The speed feedforward unit performs a speed feedforward process represented by a transfer function H(s) indicated by Expression 5 (hereinafter indicated by Math. 5) with respect to a value obtained by twice differentiating a position command value and multiplying it by a constant β, and outputs the processing result to the adder as a speed feedforward term. At the coefficients c i and d j (X≤1, j≥0, and X is a natural number) in Expression 5 are the coefficients of the transfer function H(s). The natural number X can be the same as or different from the natural number X in Expression 2. H(s)=d0+d1s+d2s2+⋯+dxsxc0+c1s+c2s2+⋯+cxsx

[0095] The tax system may have the following different arrangements than those in Fig. 1 illustrated design. <Modifizierung, bei der die Vorrichtung für maschinelles Lernen außerhalb der Motorsteuervorrichtung bereitgestellt wird>

[0096] Fig. Figure 11 is a block diagram illustrating another design example for the control system. A difference between a Fig. 11 illustrated control system 10A and the one in Fig. The control system 10 illustrated in Figure 1 is that n (n is a natural number of 2 or more) motor control devices 100A-1 to 100A-n and n machine tools 200-1 to 200-n, to which acceleration sensors 300-1 to 300-n are attached, are connected to machine learning devices 103A-1 to 130A-n via a network 400. The motor control devices 100A-1 to 100A-n have the same configuration as those shown in Fig. 1, except that the motor control device does not include a machine learning unit. The machine learning devices 130A-1 to 130A-n have the same configuration as those shown in Fig. 6 illustrates unit 130 for machine learning.

[0097] Here, the motor control device 100A-1, the acceleration sensor 300-1, and the machine learning device 130A-1 are connected as a one-to-one pair for data transmission. The motor control devices 100A-2 to 100A-n, the acceleration sensors 300-2 to 300-n, and the machine learning devices 130A-2 to 130A-n are similarly connected to the motor control device 100A-1, the machine tool 200-1, and the machine learning device 130A-1. Although in Fig. 11 n pairs of the motor control devices 100A-1 to 100A-n, the acceleration sensors 300-1 to 300-n, and the machine learning devices 130A-1 to 130A-n are connected via the network 400, the n pairs of the motor control devices 100A-1 to 100A-n, the acceleration sensors 300-1 to 300-n, and the machine learning devices 130A-1 to 130A-n may be directly connected such that the motor control device, the machine tool, and the machine learning device of each pair are directly connected through a connection interface. For example, a plurality of n pairs of the motor control devices 100A-1 to 100A-n, the machine tools 200-1 to 200-n to which acceleration sensors 300-1 to 300-n are attached, and the machine learning devices 130A-1 to 130A-n may be provided in the same manufacturing plant, and may also be provided in different manufacturing plants.

[0098] Network 400 may be, for example, a local area network (LAN) established in a manufacturing facility, the Internet, a public switched telephone network, a direct connection via a connection interface, or a combination of these. The specific data transmission scheme of network 400, whether the network uses a wired connection or a wireless connection, and the like, are not particularly limited. <Spielraum bei der Systemgestaltung>

[0099] In the above-described embodiment, the motor control devices 100A-1 to 100A-n, the acceleration sensors 300-1 to 300-n, and the machine learning devices 130A-1 to 130A-n are connected as one-to-one pairs for data transmission. However, for example, a machine learning device may be connected to a plurality of motor control devices and a plurality of acceleration sensors via the data transmission network 400, and machine learning may be performed on the respective motor control devices and the respective machine tools. In this case, a distributed processing system may be adopted in which respective functions of a machine learning device are distributed among a plurality of servers, if necessary. The functions of a machine learning device can be implemented by employing a virtual server function or the like in the cloud.

[0100] When there are n machine learning devices 130A-1 to 130A-n corresponding to n motor control devices 100A-1 to 100A-n and n machine tools 200-1 to 200-n, each with the same type name, specification, or series, the machine learning devices 130A-1 to 130A-n can be designed to share the learning results of the machine learning devices 130A-1 to 130A-n. This allows for the creation of a more optimal model. Explanation of reference symbols 10, 10A control system 100, 100A-1 to 100A-n Motor control device 110 filters 120 servo control unit 121 Subtraction device 122 Position control unit 123 Adding device 124 Position pilot control unit 125 Subtraction device 126 Speed ​​control unit 127 Servo motor 128 rotary encoders 129 Integrator 130 Machine Learning Unit 130A-1 to 130A-n Machine Learning Device 131 Status information acquisition unit 132 learning units 133 Action information output unit 134 Value function storage unit 135 Optimization action information output unit 200, 200-1 to 200-n machine tool 300 Accelerometer 400 Network< / nichtpatentdokument>

Claims

[1] A machine learning device (130, 130A) that performs machine learning of optimizing coefficients of a filter (110) provided in a motor control device (100) that controls the rotation of a motor (127) based on measurement data of an external measuring instrument (300) provided outside the motor control device (100) and a control command input to the motor control device (100), the device (130, 130A) comprising: a state information acquisition unit (131) that acquires state information including the measurement data, the control command and the coefficients of the filter (110); an action information output unit (133) that outputs action information including adjustment information of the coefficients included in the state information to the filter (110); a reward output unit (1321) that outputs a reward value for reinforcement learning using a value function based on a difference between the measurement data and the control command; and a value function updating unit (1322) that updates an action value function based on the reward value output by the reward output unit (1321), the state information, and the action information. [2] The machine learning apparatus (130, 130A) of claim 1, wherein the measurement data of the external measuring instrument (300) includes a position, a velocity, and / or an acceleration. [3] The machine learning apparatus (130, 130A) according to claim 1 or 2, wherein the motor control apparatus (100) includes a position feedback loop and / or a velocity feedback loop, and the filter (110) is provided outside the position feedback loop or the velocity feedback loop. [4] The machine learning device (130, 130A) according to claim 1 or 2, wherein the motor control device (100) has a feedback loop and the measurement data of the external measuring instrument (300) is not used for feedback control of the feedback loop. [5] The machine learning apparatus (130, 130A) according to any one of claims 1 to 4, wherein the external measuring instrument (300) is removed after the filter (110) is adjusted by machine learning. [6] The machine learning apparatus (130, 130A) of claim 1, further comprising: an optimization action information output unit (135) that outputs adjustment information of the coefficients based on the value function updated by the value function updating unit (1322). [7] Tax system that includes: a motor control device (100) including the machine learning device (130, 130A) according to any one of claims 1 to 6, a motor and a filter (110), and controlling the rotation of the motor; and an external measuring instrument (300) provided outside the engine control device (100). [8] A machine learning method of a machine learning device (130, 130A) that performs machine learning of optimizing coefficients of a filter (110) provided in a motor control device (100) that controls the rotation of a motor (127) based on measurement data of an external measuring instrument (300) provided outside the motor control device (100) and a control command input to the motor control device (100), the machine learning method comprising: capturing the state information including the measurement data, the control command and the coefficients of the filter (110); outputting the action information including adjustment information of the coefficients included in the state information to the filter (110); outputting a reward value for reinforcement learning using a value function based on a difference between the measurement data and the control command; and update an action value function based on the reward value, the state information, and the action information.

Images