Machine learning system, control device and machine learning method for optimizing filter coefficients

Abstract

The present invention provides a machine learning system, a control device, and a machine learning method for optimizing filter coefficients. The machine learning system has: an initial setting unit, which sets the initial value of the filter coefficient so that at least one specific frequency component is attenuated; a frequency characteristic calculation unit, which calculates at least one of the input-output gain and the input-output phase delay of the servo control device based on the input signal and the output signal of the frequency change in the servo control device; a filter characteristic removal unit, which removes the filter characteristic of the initial filter from at least one of the input-output gain and the input-output phase delay obtained based on the output signal and the input signal at the beginning of machine learning, wherein the output signal is obtained using the initial filter with the initial value set for the filter, and starts the machine learning of the filter coefficient so that at least the input-output gain is reduced or the phase delay is reduced after the filter characteristic is removed.

Description

Technical Field

[0001] The present invention relates to a machine learning system for optimizing coefficients of a filter in a servo control device for controlling a motor of a machine tool, a robot, or industrial machinery, a control device including the machine learning system, and a machine learning method. Background Art

[0002] There is known a device that automatically adjusts filter characteristics (for example, refer to Patent Document 1 and Patent Document 2).

[0003] Patent document 1 records: "A servo actuator having a speed feedback loop for controlling the speed of a motor, a notch filter unit is inserted into the speed feedback loop to remove machine resonance, the servo actuator having: a data collection unit, which obtains data representing the frequency response characteristics of the speed feedback loop; a moving average unit, which performs moving average processing on the data obtained by the data collection unit; a comparison unit, which compares the data obtained by the moving average unit with the data obtained by the data collection unit to extract the resonance characteristics of the speed feedback loop; and a notch filter setting unit, which sets the frequency and Q value of the notch filter unit according to the resonance characteristics extracted by the comparison unit."

[0004] Patent document 2 records "a servo actuator which, in a tuning mode, superimposes an AC signal obtained by frequency scanning with a speed command value signal, detects the amplitude of a torque command value signal obtained from a speed control unit as a result of the superposition, and sets the frequency of the torque command value signal when the rate of change of the amplitude changes from positive to negative to the center frequency of a notch filter."

[0005] Patent document 3 records "a motor control device, comprising: a notch filter, which is capable of changing the parameters of the notch filter including the notch frequency and the notch width; and a vibration frequency estimation unit for estimating the vibration frequency. It also comprises: a notch filter parameter setting unit, which sets the frequency between the notch frequency of the notch filter and the estimated vibration frequency as a new notch frequency of the notch filter, and changes the notch width so that the original notch frequency component and the estimated frequency component are attenuated."

[0006] Prior art literature

[0007] Patent Document 1: Japanese Patent Application Publication No. 2009-104439

[0008] Patent Document 2: Japanese Patent Application Laid-Open No. 5-19858

[0009] Patent Document 3: Japanese Patent Application Publication No. 2008-312339 Summary of the invention

[0010] When determining the characteristics of a filter, such as a notch filter, it is necessary to determine multiple parameters such as the attenuation coefficient, the center frequency of the frequency band to be removed, and the bandwidth, and it is desired to adjust these parameters to obtain the optimal values. In addition, when these parameters are not set appropriately, sometimes the resonance cannot be fully suppressed, or the phase delay of the servo control unit increases, resulting in deterioration of the servo control performance.

[0011] (1) One embodiment of the present disclosure provides a machine learning system that performs machine learning to optimize coefficients of at least one filter (e.g., filter 130 described below) provided in a servo control device (e.g., servo control device 100 described below) that controls a motor (e.g., motor 150 described below), wherein:

[0012] The machine learning system has:

[0013] an initial setting unit (e.g., the initial setting unit 500 described later) that sets initial values ​​of coefficients of the filter so as to attenuate at least one specific frequency component;

[0014] a frequency characteristic calculation unit (e.g., the frequency characteristic calculation unit 300 described below) for calculating at least one of an input-output gain and an input-output phase delay of the servo control device based on an input signal and an output signal of the servo control device whose frequency varies; and

[0015] a filter characteristic removing unit (for example, filter characteristic removing unit 4011 described later) which removes the filter characteristic of the initial filter from at least one of the input-output gain and the input-output phase delay obtained based on the output signal and the input signal when the machine learning starts, wherein the output signal is obtained using the initial filter in which the initial setting unit sets the initial value for the filter,

[0016] The machine learning of the filter coefficients is started so that at least the input / output gain is reduced or the phase delay is reduced after the filter characteristic of the initial filter is removed by the filter characteristic removal unit.

[0017] (2) In the machine learning system of (1) above, the frequency-varying input signal may be a frequency-varying sine wave, which is generated by a frequency generating unit (e.g., the frequency generating unit 200 described later), and the frequency generating unit is disposed inside or outside the servo control device.

[0018] (3) In the machine learning system of (1) or (2) above, the machine learning system may include a machine learning unit (e.g., the machine learning unit 400 described below),

[0019] The machine learning department has:

[0020] a state information acquisition unit (for example, state information acquisition unit 401 described later) that acquires state information, wherein the state information includes input and output gains and input and output phase delays of the servo control device output from the frequency characteristic calculation unit, and coefficients of the initial filter or coefficients of the filter after the machine learning starts;

[0021] a behavior information output unit (e.g., the behavior information output unit 403 described later), which outputs the behavior information included in the state information, the behavior information including adjustment information of the coefficients of the initial filter or the coefficients of the filter after the machine learning starts;

[0022] a reward output unit (e.g., reward output unit 4021 described later) that outputs a reward value in reinforcement learning based on the input-output gain and the input-output phase delay output from the state information acquisition unit; and

[0023] A value function updating unit (for example, the value function updating unit 4022 described later) updates the behavior value function based on the reward value output by the reward output unit, the state information, and the behavior information.

[0024] (4) In the machine learning system of (3) above, the frequency characteristic calculation unit may output the input-output gain and the input-output phase delay,

[0025] When the input-output gain of the servo control device is equal to or smaller than the input-output gain of a standard model of the input-output gain calculated based on the characteristics of the servo control device, the feedback output unit calculates a feedback of the input-output phase delay.

[0026] (5) In the machine learning system of (4) above, the input and output gains of the standard model may be fixed values ​​above a specified frequency.

[0027] (6) In the machine learning system of (4) or (5) above, the reward output unit may calculate the reward so as to reduce the phase delay between the input and output.

[0028] (7) In the machine learning system of any one of (3) to (6) above, the machine learning system may have: an optimization behavior information output unit (for example, the optimization behavior information output unit 405 described later), which outputs adjustment information of the coefficient based on the value function updated by the value function updating unit.

[0029] (8) Another aspect of the present disclosure provides a control device (for example, the control device 10 described below) having:

[0030] The machine learning system of any one of (1) to (7) above; and

[0031] A servo control device that controls a motor (for example, a motor 150 described below) includes at least one filter (for example, a filter 130 described below) that attenuates a specific frequency component.

[0032] (9) Another embodiment of the present disclosure provides a machine learning method of a machine learning system, the machine learning system performing machine learning to optimize coefficients of at least one filter (e.g., filter 130 described below) provided in a servo control device (e.g., servo control device 100 described below), the servo control device controlling a motor (e.g., motor 150 described below), the filter attenuating at least one specific frequency component, wherein:

[0033] setting initial values ​​of the coefficients of the filter so as to attenuate at least one specific frequency component,

[0034] Calculating at least one of an input-output gain and an input-output phase delay of the servo control device according to an input signal and an output signal of the servo control device with frequency variations,

[0035] At the start of machine learning, the filter characteristic of the initial filter is removed from at least one of an input-output gain and an input-output phase delay obtained based on an output signal of the servo control device and an input signal of the servo control device, wherein the output signal is obtained using the initial filter in which the initial value is set for the filter,

[0036] The machine learning of the filter coefficients is started so that at least the input-output gain is reduced or the phase delay is reduced after the filter characteristics are removed.

[0037] Effects of the Invention

[0038] By one approach, optimal parameters of a filter of a servo control device can be set. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a block diagram showing a control device of a machine learning system including one embodiment.

[0040] Figure 2 It is a diagram showing a speed command which is an input signal and a detected speed which is an output signal.

[0041] Figure 3It is a diagram showing the frequency characteristics of the amplitude ratio between the input signal and the output signal, and the phase delay.

[0042] Figure 4 This is a frequency characteristic diagram of the filter with initial settings.

[0043] Figure 5 This is a conceptual diagram for explaining the operation of the initial setting unit and the filter.

[0044] Figure 6 This is a block diagram showing a machine learning unit according to one embodiment.

[0045] Figure 7 It is a frequency characteristic diagram showing an example of the frequency characteristic with the filter characteristic added and the frequency characteristic without the filter characteristic.

[0046] Figure 8 This is a block diagram of a model that becomes a standard model for calculating input-output gain.

[0047] Fig. 9 It is a characteristic diagram showing the frequency characteristics of the input and output gains of the servo control unit of the standard model, the servo control unit before learning, and the servo control unit after learning.

[0048] Fig.10 This is a characteristic diagram showing the relationship between the bandwidth, gain, and phase of the filter.

[0049] Fig.11 This is a characteristic diagram showing the relationship between the filter's attenuation coefficient, gain, and phase.

[0050] Fig.12 : is a flowchart showing the operation of the machine learning unit during Q learning in this embodiment.

[0051] Fig.13 This is a flowchart illustrating the operation of the optimization behavior information output unit of the machine learning unit according to one embodiment.

[0052] Fig.14 This is a block diagram showing an example of configuring a filter by directly connecting a plurality of filters.

[0053] Fig.15 This is a block diagram showing another configuration example of the control device.

[0054] Explanation of symbols

[0055] 10, 10A control device

[0056] 100, 100-1~100-n Servo control unit

[0057] 110 Subtractor

[0058] 120 Speed ​​control unit

[0059] 130 Filter

[0060] 140 Current control unit

[0061] 150 Servo motor

[0062] 200 Frequency generation unit

[0063] 300 Frequency characteristics calculation unit

[0064] 400 Machine Learning Department

[0065] 400A-1~400A-n Machine Learning Department

[0066] 401 Status Information Acquisition Unit

[0067] 402 Learning Department

[0068] 403 Behavior Information Output Unit

[0069] 404 Value Function Storage

[0070] 405 Optimize behavior information output

[0071] 500 Initial setting section

[0072] 600 Network DETAILED DESCRIPTION

[0073] Hereinafter, embodiments of the present disclosure will be described in detail using the drawings.

[0074] Figure 1 1 is a block diagram showing a control device including a machine learning system according to an embodiment. The control object of the control device 10 is, for example, a machine tool, a robot, or an industrial machine. The control device 10 may also be provided as a part of the control object such as a machine tool, a robot, or an industrial machine.

[0075] The control device 10 includes a servo control unit 100 serving as a servo control device, a frequency generation unit 200 , a frequency characteristic calculation unit 300 , a machine learning unit 400 , and an initial setting unit 500 .

[0076] In addition, one or more of the frequency generation unit 200, the frequency characteristic calculation unit 300, the machine learning unit 400, and the initial setting unit 500 may be arranged in the servo control unit 100. In addition, one or both of the initial setting unit 500 and the frequency characteristic calculation unit 300 may be arranged in the machine learning unit 400. The frequency characteristic calculation unit 300, the machine learning unit 400, and the initial setting unit 500 constitute a machine learning system. The frequency characteristic calculation unit 300, the machine learning unit 400, and the initial setting unit 500 may be respectively set as independent devices, or may be set as one device. In addition, two devices may be constituted by one of the frequency characteristic calculation unit 300, the machine learning unit 400, and the initial setting unit 500 and the remaining two. Either case can constitute a machine learning system.

[0077] The servo control unit 100 includes a subtractor 110, a speed control unit 120, a filter 130, a current control unit 140, and a servo motor 150. The subtractor 110, the speed control unit 120, the filter 130, the current control unit 140, and the servo motor 150 constitute a speed feedback loop. The servo motor 150, which is a motor, is described below as a motor that performs rotational motion, but may also be a linear motor that performs linear motion.

[0078] The subtractor 110 obtains a difference between an input speed command and a detected speed of speed feedback, and outputs the difference as a position deviation to the speed control unit 120 .

[0079] The speed control unit 120 adds a value obtained by integrating the speed deviation by the integral gain K1v and a value obtained by multiplying the speed deviation by the proportional gain K2v, and outputs the result to the filter 130 as a torque command.

[0080] The filter 130 is a filter that attenuates a specific frequency component, for example, a notch filter or a low-pass filter is used. A machine such as a machine tool driven by an electric motor has a resonance point, and sometimes the resonance is increased by the servo control unit 100. In this case, for example, the resonance can be reduced by using a notch filter. The output of the filter 130 is output to the current control unit 140 as a torque command.

[0081] Mathematical formula 1 (hereinafter referred to as mathematical formula 1) represents the transfer function F(s) of the filter 130. The optimal values ​​of coefficients ai, bj (0≦i≦m, 0≦j≦n, m, n are natural numbers) of mathematical formula 1 are machine-learned by the machine learning unit 400.

[0082]

Mathematical formula 1

[0083]

[0084] The current control unit 140 generates a current command for driving the servo motor 150 based on the torque command, and outputs the current command to the servo motor 150 .

[0085] The rotation angle position of the servo motor 150 is detected by a rotary encoder (not shown) provided in the servo motor 150 , and the speed detection value is input to the subtractor 110 as speed feedback.

[0086] The servo control unit 100 is configured as described above. However, in order to perform machine learning on the optimal parameters of the filter, the control device 10 further includes a frequency generation unit 200 , a frequency characteristic calculation unit 300 , a machine learning unit 400 , and an initial setting unit 500 .

[0087] The frequency generating unit 200 outputs the sine wave signal as a speed command to the subtracter 110 and the frequency characteristic calculating unit 300 of the servo control unit 100 while changing the frequency. The speed command becomes an input signal to the servo control unit 100.

[0088] The frequency characteristic calculation unit 300 uses the speed command (sine wave) and the detected speed (sine wave) which is the output signal output from the rotary encoder (not shown) to obtain the amplitude ratio (input-output gain) and phase delay between the input signal and the output signal at each frequency specified by the speed command. Figure 2 It is a diagram showing a speed command which is an input signal and a detected speed which is an output signal. Figure 3 This is a graph showing the frequency characteristics of the amplitude ratio and phase delay between the input signal and the output signal.

[0089] like Figure 2 As shown, the speed command output from the frequency generating unit 200 changes the frequency to obtain Figure 3 The frequency characteristics of input-output gain (amplitude ratio) and phase delay are shown.

[0090] The machine learning unit 400 performs machine learning (hereinafter referred to as learning) on ​​the coefficients ai and bj of the transfer function F(s) of the filter 130 using the input-output gain (amplitude ratio) and phase delay output from the frequency characteristic calculation unit 300. The learning performed by the machine learning unit 400 may be performed before shipment, or re-learning may be performed after shipment.

[0091] When or before the learning by the machine learning unit 400 starts, the initial setting unit 500 sets the initial value of the coefficient of the transfer function F(s) of the filter 130. The filter 130 for which the initial value is set is called an initial filter. The coefficient of the initial value is set to suppress the resonance frequency. When the initial value of the coefficient of the filter 130 is set to suppress the resonance frequency, it is possible that when there is a resonance point of the machine, a large vibration is generated due to superimposed vibration, and data such as the detection speed cannot be measured.

[0092] As a method of setting the initial value of the filter coefficient, for example, the filter adjustment method disclosed in the servo motor control device described in Japanese Patent Application Laid-Open No. 2016-034224 and the servo control device described in Japanese Patent Application Laid-Open No. 2017-022855 can be used.

[0093] Regarding the servo motor control device described in Japanese Patent Application Publication No. 2016-034224, a sine wave generating unit generates a sine wave interference value, and a frequency response calculating unit calculates a frequency response when the sine wave interference value is input to a speed control loop. Furthermore, a resonance frequency detecting unit detects a resonance frequency at which a gain of the frequency response is maximum, and a resonance frequency comparing unit measures the rigidity of a machine tool based on the resonance frequency, and adjusts a filter based on the resonance frequency.

[0094] In the servo control device described in Japanese Patent Publication No. 2017-022855, a sine wave interference input unit performs a sine wave scan on a speed control loop, and a frequency characteristic calculation unit estimates the gain and phase of the input and output signals of the speed control loop based on the output of the speed control loop. Furthermore, the frequency characteristic calculation unit calculates the frequency characteristic by calculating the amplitude and phase of the fundamental wave component of the Fourier series by setting the interference input frequency from the sine wave interference input unit as the fundamental frequency and consisting of an arbitrary number of terms to represent the output of the speed control loop. Thereafter, the resonance frequency detection unit detects the resonance frequency based on the frequency characteristic, and the filter adjustment unit adjusts the filter based on the detected resonance frequency.

[0095] The frequency characteristic of the filter 130 with the coefficients set to the initial values ​​is as follows when there is one resonance point: Figure 4 The frequency characteristics are shown.

[0096] The frequency characteristic calculation unit 300 calculates the frequency characteristics of the input-output gain and the phase delay using the detection speed and the speed command obtained by the servo control unit 11, wherein the servo control unit 11 uses the filter 130 whose coefficients are set by the initial setting unit 500 at the start of learning or before the start. However, this frequency characteristic is not the frequency characteristic when the filter is used when the initial setting is not performed by the initial setting unit 500. Therefore, the machine learning unit 400 obtains the frequency characteristic when the filter characteristics of the filter 130 are set by the initial setting unit 500 at the start of learning, and learns the coefficients of the filter 130 based on the obtained frequency characteristics. After the start of learning, the detection speed and the speed command are obtained using the filter 130 whose coefficients are set by the machine learning unit 400, and the frequency characteristic calculation unit 300 calculates the frequency characteristics using the obtained detection speed and the speed command.

[0097] The method of removing the filter characteristics of the filter 130 (initial filter) whose coefficients are set by the initial setting unit 500 is performed, for example, as follows.

[0098] The servo control unit 100 performs servo control using the filter 130 whose coefficients are set by the initial setting unit 500, thereby obtaining the detection speed and the speed command. The frequency characteristic calculation unit 300 generates a mechanical model (coefficient c ) of the transfer function L1(s) of the servo control unit 100 represented by Mathematical Formula 2 (hereinafter referred to as Mathematical Formula 2) using the detection speed (output) and the speed command (input) obtained from the servo control unit 100. i , d j , 0≦i≦m, 0≦j≦n, m, n natural numbers) are generated.

[0099] Mathematical formula 2

[0100]

[0101] The frequency characteristic calculation unit 300 substitutes s=j·ω (j is a complex function) when the frequency ω[rad]=2πf[Hz] to obtain

[0102] Gain = |L1(jω)|

[0103] Phase = ∠L1(jω)

[0104] , and draw a graph according to frequency to obtain the frequency characteristics.

[0105] On the other hand, the machine learning unit 400 stores the transfer function of the filter 130 (initial filter) set by the initial setting unit 500. This transfer function is, for example, the transfer function H(s) represented by Mathematical Formula 3 (hereinafter referred to as Mathematical Formula 3). The transfer function H(s) is a transfer function of a notch filter.

[0106] In mathematical formula 3, coefficient k is the attenuation coefficient, and coefficient ω is c is the center angular frequency, and the coefficient τ is the specific bandwidth. Let the center frequency be fc and the bandwidth be fw, then by ω c =2πfc to represent the coefficient ω c , the coefficient τ is expressed by τ = fw / fc. In the coefficients ai, bj of the transfer function F(s) in mathematical formula 1, let a0, b0 = ω c 2 、a1=2τω c , b1=2kτω c 、a2=1、b2=1、a3~a m =0, b3~b n =0, from which the transfer function H(s) of mathematical formula 3 can be obtained.

[0107] Mathematical formula 3

[0108]

[0109] The machine learning unit 400 uses mathematical formula 3 to calculate the gain and phase of each frequency. In addition to obtaining these frequency characteristics using the mechanical model of the transfer function L1(s), the machine learning unit 400 also obtains the frequency characteristics by removing the filter characteristics of the filter 130 (initial filter) set by the initial setting unit 500.

[0110] Here, assuming that the transfer function of the servo control unit 100 without the filter characteristics of the filter 130 set by the initial setting unit 500 is L0(s), L0(s)+H(s)=L1(s), and therefore, L0(s)=L1(s)-H(s).

[0111] When the frequency ω[rad]=2πf[Hz], substitute s=j·ω (j is a complex function) and find

[0112] Gain = |L1(jω)-H(jω)|

[0113] Phase = ∠(L1(jω)-H(jω))

[0114] , and plot the frequency characteristics by frequency.

[0115] Below, use Figure 5 The operations of the initial setting unit 500 and the filter 130 will be described.

[0116] Figure 5 This is a conceptual diagram for explaining the operation of the initial setting unit 500 and the filter 130 .

[0117] Figure 5The initial filter 503 shown corresponds to the filter 130 whose coefficients are set to initial values, and the filter 502 corresponds to the filter 130 being learned. The coefficients of the initial filter 503 are set to the coefficients of the transfer function H(s) of Mathematical Formula 3, for example, so as to suppress the resonance frequency. When the learning by the machine learning unit 400 starts, the initial filter 503 is connected between the speed control unit 120 and the current control unit 140 through the switching units 501 and 504 to perform servo control. The frequency characteristic calculation unit 300 uses the detection speed and speed command obtained by the servo control using the initial filter 503 to calculate the frequency characteristics of the input-output gain and the phase delay through the transfer function L1(s) of Mathematical Formula 2. Thereafter, the machine learning unit 400 obtains the frequency characteristics when the filter characteristics are removed from the transfer function L0(s) obtained from L0(s)=L1(s)-H(s) to perform learning. After the learning starts, the filter 502 is connected between the speed control unit 120 and the current control unit 140 by the switching units 501 and 504 to perform servo control.

[0118] Thus, at the beginning of learning, an initial filter 503 with coefficients set is used to suppress the resonance frequency. Figure 1 In the control device 10 shown, the initial filter 503 corresponds to the initial setting unit 500 setting the coefficient for suppressing the resonance frequency as the initial value of the filter 103 .

[0119] In addition, in actual use Figure 5 When compared to the structure of Figure 1 The structure increases the number of filters by one and inserts two switching parts, which makes installation complicated and increases the cost. However, if the complexity and cost of installation are acceptable, it is also possible to replace Figure 1 The initial setting unit and filter structure shown in FIG. Figure 5 structure.

[0120] The following further describes the structure and operation of the machine learning unit 400 in detail.

[0121] <Machine Learning Department 400>

[0122] In the following description, a case where the machine learning unit 400 performs reinforcement learning is described, but the learning performed by the machine learning unit 400 is not particularly limited to reinforcement learning. For example, the present invention can also be applied to a case where supervised learning is performed.

[0123] Before describing each functional block included in the machine learning unit 400, the basic structure of reinforcement learning is described first. The agent (equivalent to the machine learning unit 400 in this embodiment) observes the state of the environment, selects a certain behavior, and the environment changes according to the behavior. As the environment changes, a certain reward is provided, and the agent learns better behavior choices (decisions).

[0124] Supervised learning represents a completely correct answer, while the rewards in reinforcement learning are mostly episodic values ​​based on partial changes in the environment. Therefore, the agent learns to choose actions that maximize the total reward in the future.

[0125] In this way, in reinforcement learning, appropriate behavior is learned based on the interaction between behavior and environment, that is, the method to be learned is learned to maximize the reward to be obtained in the future. This means that in this embodiment, behavior information that affects the future, such as selecting behavior information for suppressing vibration at the machine end, can be obtained.

[0126] Here, any learning method can be used as reinforcement learning. In the following description, the case of using Q-learning in a certain environmental state S is used as an example. The Q-learning is a method of learning the value Q(S, A) of selecting behavior A.

[0127] The purpose of Q learning is to select the behavior A with the highest value Q(S, A) as the best behavior from the available behaviors A in a certain state S.

[0128] However, at the time when Q learning first begins, the correct value of the value Q(S, A) for the combination of state S and behavior A is completely unknown. Therefore, the agent selects various behaviors A in a certain state S, and for the behavior A at that time, it selects a better behavior based on the reward given, thereby continuing to learn the correct value Q(S, A).

[0129] In addition, we want to maximize the total of future rewards, so the goal is to eventually achieve Q(S, A) = E[Σ(γ t )r t Here, E[] represents the expected value, t represents the time, γ represents a parameter called the discount rate described later, and r t represents the reward at time t, and Σ is the total at time t. The expected value in this mathematical formula is the expected value when the optimal behavior state changes. However, in the process of Q learning, since the optimal behavior is unknown, reinforcement learning is performed while searching by performing various behaviors. The update formula of such value Q(S, A) can be expressed by the following mathematical formula 4 (hereinafter referred to as mathematical formula 4), for example.

[0130]

Mathematical formula 4

[0131]

[0132] In the above mathematical formula 4, S t represents the environmental state at time t, A t Represents the behavior at time t. Through behavior A t , the state changes to S t+1 . r t+1 Indicates the reward obtained by changing the state. In addition, the item with max is: t+1 In the following example, γ is multiplied by the Q value when the behavior A with the highest Q value known at that time is selected. Here, γ is a parameter of 0<γ≤1, which is called the discount rate. In addition, α is a learning coefficient, and the range of α is 0<α≤1.

[0133] The above mathematical formula 4 represents the following method: According to the experiment A t The result is the feedback r t+1 , update the state S t The following behavior A t The value of Q(S t , A t ).

[0134] This update expression shows that if behavior A t The next state S t+1 The value of the best behavior under a Q(S t+1 , A) than state S t The following behavior A t The value of Q(S t , A t ) is large, then Q(S t , A t ), if it is small, then reduce Q(S t , A t ). That is, the value of a certain behavior in a certain state is close to the value of the best behavior in the next state caused by the behavior. Although the difference is due to the discount rate γ and the reward r t+1 The structure varies depending on the form of existence, but is basically a structure in which the best behavior value in a certain state is propagated to the behavior value in its previous state.

[0135] Here, there is a method of Q learning: a table of Q(S, A) for all state-action pairs (S, A) is created to perform learning. However, in order to find the value of Q(S, A) for all state-action pairs, there are sometimes too many states, making it take a long time for Q learning to converge.

[0136] Therefore, a well-known technology called DQN (Deep Q-Network) can be used. Specifically, a suitable neural network can be used to construct the value function Q, and the parameters of the neural network can be adjusted to approximate the value function Q through a suitable neural network to calculate the value of the value Q (S, A). By using DQN, the time required for Q learning to converge can be shortened. In addition, DQN is described in detail in, for example, the following non-patent literature.

[0137] <Non-patent literature>

[0138] "Human-level control through deep reinforcement learning", Volodymyr Mnih [online], [retrieved on January 17, 2011], Internet 〈URL: http: / / files.davidqiu.com / research / nature14236.pdf〉

[0139] The machine learning unit 400 performs the Q learning described above. Specifically, the machine learning unit 400 learns the following value Q: i 、b j The value of the input / output gain (amplitude ratio) and the phase delay output from the frequency characteristic calculation unit 300 are taken as the state S, and the coefficients a of the transfer function of the filter 130 related to the state S are taken as i 、b j The value of is adjusted, and behavior A is selected. At the beginning of learning, the initial setting unit 500 sets each coefficient a of the transfer function F(s) of the filter 130 i 、b j The machine learning unit 400 performs learning by setting each coefficient of the transfer function H(s) by removing the filter characteristic of the filter 130 set by the initial setting unit 500. Therefore, the input / output gain (amplitude ratio) and phase delay become the state S when the filter characteristic is removed.

[0140] After the learning starts, the servo control unit 100 calculates the coefficients a of the transfer function F(s) of the filter 130 set by the machine learning unit 400. i 、b j, using the sine wave of the frequency change, i.e., the speed command, to perform servo control. The machine learning unit 400 determines the behavior A by observing the state information S, which includes the input-output gain (amplitude ratio) and phase delay of each frequency obtained from the frequency characteristic calculation unit 300 by driving the servo control unit 100. The machine learning unit 400 returns a reward each time the behavior A is performed. The machine learning unit 400 searches for the best behavior A by trial and error, for example, so that the total reward in the future is maximized. In this way, the machine learning unit 400 determines the behavior A based on the coefficients a of the transfer function F(s) of the filter 130. i 、b j The servo control unit 100 is driven by a sine wave with a variable frequency, i.e., a speed command. Furthermore, the machine learning unit 400 can select the best behavior A (i.e., the best coefficient a of the transfer function F(s) of the filter 130) for the state S. i 、b j ), wherein the state S includes the input-output gain (amplitude ratio) and phase delay of each frequency obtained from the frequency characteristic calculation unit 300.

[0141] That is, each coefficient a of the transfer function of the filter 130 applied to a certain state S is selected based on the cost function Q learned by the machine learning unit 400. i 、b j In this way, the machine learning unit 400 can select the behavior A (that is, each coefficient a of the transfer function of the filter 130) in which the vibration of the mechanical end generated by executing the learning program is minimized. i 、b j ).

[0142] Figure 6 It is a block diagram showing the machine learning unit 400 according to one embodiment.

[0143] In order to carry out the above reinforcement learning, Figure 6 As shown, the machine learning unit 400 includes: a state information acquisition unit 401, a learning unit 402, a behavior information output unit 403, a value function storage unit 404, and an optimized behavior information output unit 405.

[0144] The state information acquisition unit 401 includes a filter characteristic removal unit 4011. The filter characteristic removal unit 4011 stores the transfer function H(s) of the filter 130 set by the initial setting unit 500.

[0145] When or before the start of learning, the initial setting unit 500 sets the transfer function of the filter 130 to the transfer function H(s). Then, the frequency characteristic calculation unit 300 calculates the frequency characteristics of the input-output gain (amplitude ratio) and the phase delay through the transfer function L1(s) of the servo control unit 100 using the detection speed and speed command obtained by the filter 130 using the transfer function H(s).

[0146] This frequency characteristic is not the frequency characteristic when the filter 130 is used without the initial setting by the initial setting unit 500. Therefore, when the learning starts, the state information acquisition unit 401 uses the filter characteristic removal unit 4011 to obtain the frequency characteristics of the input-output gain (amplitude ratio) and phase delay when the filter characteristic of the filter 130 is removed from the transfer function H(s) by using the transfer function L0(s) (L0(s)=L1(s)-H(s)). This frequency characteristic is the state S.

[0147] For example, Figure 7 An example of the frequency characteristics with the filter characteristics added and the frequency characteristics without the filter characteristics is shown. Figure 7 In FIG. 1 , a curve RC1 is a curve showing the frequency characteristics of the servo control unit 100 to which the filter characteristics are added, and a curve RC2 is a curve showing the frequency characteristics of the servo control unit 100 to which the filter characteristics are removed.

[0148] In addition, the filter characteristic removal unit 4011 is not included in the state information acquisition unit 401 of the machine learning unit 400 , but may be provided in the frequency characteristic calculation unit 300 .

[0149] The state information acquisition unit 401 acquires the state S from the frequency characteristic calculation unit 300, wherein the state S includes the coefficients a of the transfer function F(s) of the filter 130 set by the machine learning unit 400 in the learning process after the start of learning. i 、b j , the input-output gain (amplitude ratio) and phase delay obtained by driving the servo motor 150 using the speed command (sine wave). The input-output gain is the input-output gain calculated using the transfer function L(s) of the servo control unit 100 of the filter having the transfer function F(s) after the start of learning. The frequency characteristic calculation unit 300 generates a mechanical model of the transfer function L(s) of the servo control unit 100 using the detected speed (output) and speed command (input) obtained from the servo control unit 100. This state S is equivalent to the environmental state S in Q learning.

[0150] The state information acquisition unit 401 outputs the acquired state S to the learning unit 402 .

[0151] The learning unit 402 is a part that learns the value Q (S, A) when a certain behavior A is selected under a certain environmental state S. The learning unit 402 includes a reward output unit 4021 , a value function update unit 4022 , and a behavior information generation unit 4023 .

[0152] The reward output unit 4021 is a part that calculates a reward when the behavior A is selected in a certain state S.

[0153] The feedback output unit 4021 outputs the coefficients a of the transfer function F(s) of the modified filter 130. i 、b j The input-output gain G calculated in the case of is compared with the input-output gain Gb of each frequency of the preset standard model. In addition, the input-output gain G is the input-output gain calculated using the transfer function L(s) of the servo control unit 100.

[0154] The feedback output unit 4021 provides a negative feedback when the calculated input-output gain G is greater than the input-output gain Gb of the standard model. On the other hand, when the calculated input-output gain G is less than the input-output gain Gb of the standard model, the feedback output unit 4021 provides a positive feedback when the phase delay decreases, a negative feedback when the phase delay increases, and a zero feedback when the phase delay remains unchanged.

[0155] First, use Figure 8 and Fig. 9 The action of providing a negative feedback by the feedback output unit 4021 when the calculated input-output gain G is greater than the input-output gain Gb of the standard model is described.

[0156] The feedback output unit 4021 stores a standard model of input and output gain. The standard model is a model of a servo control unit having an ideal characteristic of non-resonance. The standard model can be based on, for example, Figure 8 The inertia Ja and torque constant K of the model shown t , proportional gain K p , integral gain K I , differential gain K D The inertia Ja is the sum of the motor inertia and the mechanical inertia.

[0157] Fig. 9 : is a characteristic diagram showing the frequency characteristics of the input and output gain of the servo control unit of the standard model and the servo control unit 100 before and after learning. Fig. 9 As shown in the characteristic diagram, the standard model has: region A and region B, wherein region A is a frequency region where the input-output gain is above a certain value, for example, an ideal input-output gain of -20 dB or above, and region B is a frequency region where the input-output gain is less than a certain value. Fig. 9 In region A, the ideal input-output gain of the standard model is represented by curve MC1 (bold line). Fig. 9 In area B, through the curve MC 11 The ideal virtual input-output gain of the standard model is represented by the thick dashed line, and the input-output gain of the standard model is set as a fixed value through the straight line MC 12 (bold line). Fig. 9 In regions A and B of , curves RC1 and RC2 represent curves of the input and output gain of the servo control unit before and after learning, respectively.

[0158] In region A, when the calculated input-output gain curve RC2 exceeds the ideal input-output gain curve MC1 of the standard model, the feedback output unit 4021 provides a first negative feedback.

[0159] In the region B where the input-output gain becomes sufficiently small, even if the input-output gain curve RC2 exceeds the ideal virtual input-output gain curve MC of the standard model, 11 Therefore, in region B, as mentioned above, the input-output gain of the standard model is not the ideal gain characteristic curve MC. 11 , but use a straight line MC with fixed input and output gain (e.g., -20dB) 12 However, the calculated input-output gain curve RC2 before learning exceeds the fixed value input-output gain straight line MC 12 It may be unstable at times, so a first negative value is given as a reward.

[0160] Next, the operation of the feedback output unit 4021 to determine the feedback based on the phase delay information when the calculated input-output gain G is less than or equal to the input-output gain Gb of the standard model will be described.

[0161] In the following description, D(S) represents the state variable related to the state information S, that is, the phase delay, and D(S') represents the coefficients a of the transfer function F(s) of the filter 130. i 、b j The state variable related to the state S' that has changed from state S is the phase delay.

[0162] The report output unit 4021 may determine a report based on the phase delay information in the following three methods, for example.

[0163] The first method is to determine the feedback by increasing, decreasing, or keeping the same the frequency of the phase delay of 180 degrees when the state S changes to the state S'. Here, the case of the phase delay of 180 degrees is cited, but it is not particularly limited to 180 degrees and may be other values.

[0164] For example, in Figure 3 The phase line diagram shown in the figure shows that when the phase is delayed, when the state changes from state S to state S', the curve changes so that the frequency at which the phase delay is 180 degrees becomes smaller (in Figure 3 On the other hand, when the state changes from state S to state S', the frequency at which the curve changes to a phase delay of 180 degrees increases (in Figure 3 X1 direction), the phase delay becomes smaller.

[0165] Therefore, when the state S changes to the state S', when the frequency at which the phase delay is 180 degrees decreases, it is defined as phase delay D(S) < phase delay D(S'), and the feedback output unit 4021 sets the feedback value to the second negative value. In addition, the absolute value of the second negative value is set smaller than the first negative value.

[0166] On the other hand, when the state changes to the state S′, if the frequency of the phase delay of 180 degrees increases, it is defined as phase delay D(S)>phase delay D(S′), and the feedback output unit 4021 sets the feedback value to a positive value.

[0167] When the state changes from state S to state S′, if the frequency at which the phase delay is 180 degrees does not change, it is defined as phase delay D(S)=phase delay D(S′), and the feedback output unit 4021 sets the feedback value to zero.

[0168] The second method is a method of determining a feedback based on whether the absolute value of the phase delay when the input-output gain crosses 0 dB becomes larger, smaller, or the same when the state changes from the state S to the state S'.

[0169] For example, in state S, Figure 3 The gain line graph shown in the figure shows the point where the input gain crosses 0dB (hereinafter referred to as the "zero crossing point"), Figure 3 The phase line diagram shown has a phase delay of -90 degrees.

[0170] When the state changes from the state S to the state S′, when the absolute value of the phase delay at the zero crossing point increases, it is defined as phase delay D(S)<phase delay D(S′), and the feedback output unit 4021 sets the feedback value to the second negative value.

[0171] On the other hand, when the state changes from S to S', when the absolute value of the phase delay at the zero crossing point becomes smaller, it is defined as phase delay D(S)>phase delay D(S'), and the feedback output unit 4021 sets the feedback value to a positive value.

[0172] When the state changes from state S to state S′, if the absolute value of the phase delay at the zero crossing point does not change, it is defined as phase delay D(S)=phase delay D(S′), and the feedback output unit 4021 sets the feedback value to zero.

[0173] The third method is to determine the return by whether the phase margin becomes larger, smaller, or the same when the state S changes to the state S'. The so-called phase margin is a quantity indicating how many degrees the phase is away from -180 degrees when the gain is 0 dB. For example, Figure 3 When the gain is 0dB, the phase is -90 degrees, so the phase margin is 90 degrees.

[0174] When the state changes from the state S to the state S′, when the phase margin becomes smaller, it is defined as phase delay D(S)<phase delay D(S′), and the feedback output unit 4021 sets the feedback value to the second negative value.

[0175] On the other hand, when the state changes from the state S to the state S′, when the phase margin increases, it is defined as phase delay D(S)>phase delay D(S′), and the feedback output unit 4021 sets the feedback value to a positive value.

[0176] Furthermore, when the state changes from the state S to the state S′, if the phase margin does not change, it is defined as phase delay D(S)=phase delay D(S′), and the feedback output unit 4021 sets the feedback value to zero.

[0177] In addition, as a negative value defined as when the phase delay D(S') of the state S' after the execution of behavior A is larger than the phase delay D(S) in the previous state S, the negative value can be set larger according to the ratio. For example, in the first method described above, the negative value can be set larger according to the degree of frequency reduction. Conversely, as a positive value defined as when the phase delay D(S') of the state S' after the execution of behavior A is smaller than the phase delay D(S) in the previous state S, the positive value can be set larger according to the ratio. For example, in the first method described above, the positive value can be set larger according to the degree of frequency increase.

[0178] The value function updating unit 4022 performs Q learning based on the state S, behavior A, the state S′ when behavior A is applied to the state S, and the reward value calculated as described above, thereby updating the value function Q stored in the value function storage unit 404 .

[0179] The update of the value function Q can be performed through online learning, batch learning, or mini-batch learning.

[0180] Online learning is a learning method that applies a certain behavior A to the current state S, and updates the value function Q every time the state S transitions to a new state S'. In addition, batch learning is a learning method that collects learning data by repeatedly applying a certain behavior A to the current state S, and transitions the state S to a new state S', and uses all the collected learning data to update the value function Q. Furthermore, mini-batch learning is a learning method between online learning and batch learning, and is a learning method that updates the value function Q every time a certain amount of learning data is accumulated.

[0181] The behavior information generation unit 4023 selects behavior A during Q learning for the current state S. In order to correct each coefficient a of the transfer function F(s) of the filter 130 during Q learning, the behavior information generation unit 4023 i 、b j action (equivalent to behavior A in Q learning), and generates behavior information A, and outputs the generated behavior information A to the behavior information output unit 403.

[0182] More specifically, the behavior information generation unit 4023 generates, for example, a value of each coefficient a of the transfer function F(s) of the filter 130 included in the state S. i 、b j , so that the coefficients a of the transfer function F(s) of the filter 130 included in behavior A i 、b j Plus or minus increment.

[0183] Furthermore, the following strategy may be adopted: the behavior information generating unit 4023 applies each coefficient a of the transfer function F(s) of the filter 130 to i 、b j When the state S' is transferred to the positive reward (positive reward) by the increase or decrease of the filter 130, the coefficients a of the transfer function F(s) of the filter 130 are selected as the next behavior A'. i 、b j The behavior A' is such that the calculated phase delay is smaller than the previous phase delay by increasing or decreasing the increment in the same manner as the previous operation.

[0184] In addition, the following strategy can be adopted conversely: when a negative reward (negative reward) is returned, the behavior information generation unit 4023 selects the coefficients a of the transfer function F(s) of the filter 130 as the next behavior A', for example i 、b jAction A' is to reduce or increase the increment opposite to the previous action, so that when the calculated input-output gain is larger than the input-output gain of the standard model, the difference in input gain is smaller than that of the previous time, or the calculated phase delay is smaller than that of the previous time.

[0185] In addition, each coefficient a i 、b j All or part of the coefficients may be corrected. For example, when the transfer function F(s) of the filter 130 is learned as the transfer function of the notch filter as shown in Mathematical Formula 3, the center frequency fc that generates resonance is easy to find and the center frequency fc is easy to determine. Therefore, the behavior information generation unit 4023 may temporarily fix the center frequency fc, correct the bandwidth fw and the attenuation coefficient k, that is, fix the coefficient ω c (=2πfc), the correction coefficient τ (=fw / fc) and the attenuation coefficient k. Specifically, the coefficient a of the transfer function F(s) in mathematical formula 1 is i 、b j In the above equation, let a0, b0 = ω c 2 、a1=2τω c , b1=2kτω c 、a2=1、b2=1、a3~a m =0, b3~b n = 0. In this case, a0 and b0 may be fixed, a1 and b1 may be modified to generate behavior information A, and the generated behavior information A may be output to the behavior information output unit 403.

[0186] In addition, the characteristics of the filter 130 are as follows Fig.10 As shown, the bandwidth fw of the filter 130 causes the gain and phase to change. Fig.10 In FIG. 1 , the dotted line indicates a case where the bandwidth fw is large, and the solid line indicates a case where the bandwidth fw is small. In addition, the characteristics of the filter 130 are as follows: Fig.11 As shown, the attenuation coefficient k of the filter 130 causes the gain and phase to change. Fig.11 In FIG. 1 , the dotted line indicates the case where the attenuation coefficient k is small, and the solid line indicates the case where the attenuation coefficient k is large.

[0187] In addition, the behavior information generation unit 4023 may also adopt the following strategy: select behavior A' by a greedy algorithm that selects behavior A' with the highest value Q(S, A) from the values ​​of the currently estimated behavior A, or randomly selects behavior A' with a smaller probability ε, and other well-known methods such as the ε greedy algorithm that selects behavior A' with the highest value Q(S, A).

[0188] The behavior information output unit 403 is a part that sends the behavior information A output from the learning unit 402 to the filter 130. As described above, the filter 130 adjusts the current state S, that is, the currently set coefficients a, based on the behavior information. i 、b j A slight correction is performed, and the state is transferred to the next state S' (ie, the coefficients of the filter 130 after correction).

[0189] The value function storage unit 404 is a storage device for storing the value function Q. The value function Q can be stored as a table (hereinafter referred to as a behavior value table) according to the state S and the behavior A, for example. The value function Q stored in the value function storage unit 404 is updated by the value function update unit 4022. In addition, the value function Q stored in the value function storage unit 404 can also be shared with other machine learning units 400. If the value function Q is shared among multiple machine learning units 400, reinforcement learning can be performed in a distributed manner through each machine learning unit 400, thereby improving the efficiency of reinforcement learning.

[0190] The optimized behavior information output unit 405 generates behavior information A (hereinafter referred to as “optimized behavior information”) for causing the filter 130 to perform an action that maximizes the value Q(S, A) based on the value function Q updated by the value function update unit 4022 through Q learning.

[0191] More specifically, the optimized behavior information output unit 405 obtains the value function Q stored in the value function storage unit 404. As described above, the value function Q is a function updated by the value function update unit 4022 through Q learning. In addition, the optimized behavior information output unit 405 generates behavior information based on the value function Q, and outputs the generated behavior information to the filter 130. The optimized behavior information is the same as the behavior information output by the behavior information output unit 403 during the Q learning process, and includes the coefficients a of the transfer function of the correction filter 130. i 、b j information.

[0192] In the filter 130, each coefficient a of the transfer function F(s) is corrected based on the behavior information. i 、b j .

[0193] The machine learning unit 400 can operate so as to calculate the coefficients a of the transfer function F(s) of the filter 130 through the above operation. i 、b j Optimization to suppress vibration at the mechanical end.

[0194] As described above, by using the machine learning unit 400 according to the present invention, parameter adjustment of the filter 130 can be simplified.

[0195] In the above, the functional blocks included in the control device 10 have been described.

[0196] In order to realize these functional blocks, the control device 10 has a CPU (Central Processing Unit) and other arithmetic processing devices. In addition, the control device 10 also has an auxiliary storage device such as a HDD (Hard Disk Drive) that stores various control programs such as application software or OS (Operating System), and a main storage device such as a RAM (Random Access Memory) that stores data temporarily required after the arithmetic processing device executes the program.

[0197] Furthermore, in the control device 10, the operation processing device reads the application software or OS from the auxiliary storage device, expands the read application software or OS on the main storage device, and performs operation processing based on the application software or OS. In addition, various hardwares possessed by each device are controlled according to the operation results. Thus, the functional blocks of this embodiment are realized. That is, this embodiment can be realized by the cooperation of hardware and software.

[0198] As for the machine learning unit 400, since the amount of computation accompanying machine learning increases, for example, a technology called GPGPU (General-Purpose computing on Graphics Processing Units) is used to mount GPUs (Graphics Processing Units) on personal computers, so that high-speed processing can be performed when GPUs are used for computational processing accompanying machine learning. In addition, in order to perform even higher-speed processing, a computer cluster can be constructed using multiple computers equipped with such GPUs, and parallel processing can be performed by multiple computers included in the computer cluster.

[0199] Next, refer to Fig.12 The process of FIG. 1 illustrates the operation of the machine learning unit 400 during Q learning in this embodiment.

[0200] In step S11 , the state information acquisition unit 401 acquires initial state information S from the servo control unit 100 and the frequency characteristics calculation unit 300 .

[0201] The servo control unit 100 uses a sine wave whose frequency changes, i.e., a speed command, to drive the transfer function of the filter 130, which is the transfer function H(s). As a result, the input-output gain (amplitude ratio) G(S0) and the phase delay D(S0) in the state S0 at the time point when Q learning is first started are obtained from the frequency characteristic calculation unit 300. The speed command and the detected speed are input to the frequency characteristic calculation unit 300, and the input-output gain (amplitude ratio) G(S0) and the phase delay D(S0) output from the frequency characteristic calculation unit 300 are input to the state information acquisition unit 401 as the initial state information. The frequency characteristic calculation unit 300 calculates the input-output gain (amplitude ratio) G(S0) and the phase delay D(S0) based on the transfer function L1(s) shown in Mathematical Formula 2 of the servo control unit 100 in the state S0. The coefficients ω of the transfer function H(s) of the filter 130 set by the initial setting unit 500 are c , τ, k are sent as initial state information to the state information acquisition unit 401. As already explained, by setting a0, b0 of the transfer function F(s) of mathematical formula 1 to be c 2 、a1=2τω c , b1=2kτω c 、a2=1、b2=1、a3~a m =0, b3~b m =0, the transfer function H(s) of mathematical formula 3 can be obtained.

[0202] In step S12, the state information acquisition unit 401 obtains the frequency characteristics of the input-output gain (amplitude ratio) and phase delay in the state S0 when the filter characteristics of the filter 130 set by the initial setting unit 500 are removed using the filter characteristics removal unit 4011. The obtained input-output gain (amplitude ratio) and phase delay are output to the feedback output unit 4021 and the behavior information generation unit 4023. As described above, the state information S is information corresponding to the state in Q learning.

[0203] In step S13, the behavior information generation unit 4023 generates new behavior information A, and outputs the generated new behavior information A to the filter 130 via the behavior information output unit 403. The behavior information generation unit 4023 outputs the new behavior information A according to the strategy. In addition, the servo control unit 100 that receives the behavior information A adjusts the coefficients a of the transfer function F(s) of the filter 130 related to the current state S according to the received behavior information. i 、b jIn the corrected state S', the servo motor 150 is driven using a sine wave with a changed frequency, i.e., a speed command. As described above, this behavior information corresponds to the behavior A in Q learning. In addition, in the state S0 at the start of learning, regarding the transfer function of the filter 130, each coefficient a of the transfer function F(s) is i 、b j The coefficients of the transfer function H(s) are set. In the next state S' after the state S0 at the start of learning, the behavior information generation unit 4023 adjusts the coefficients a of the transfer function F(s) according to the coefficients of the transfer function H(s). i 、b j Make corrections.

[0204] In step S14, the state information acquisition unit 401 acquires, as new state information, the input-output gain (amplitude ratio) G(S') and the phase delay D(S') in the new state S' and the coefficients a of the transfer function F(s) of the filter 130. i 、b j The new status information obtained is output to the report output unit 4021.

[0205] In step S15, the feedback output unit 4021 determines whether the input-output gain G(S') of each frequency in state S' is less than the input-output gain Gb of each frequency of the standard model. If the input-output gain G(S') of each frequency is greater than the input-output gain Gb of each frequency of the standard model (No in step S15), then in step S16, the feedback output unit 4021 sets the feedback to the first negative value and returns to step S13.

[0206] Regarding the feedback output unit 4021, if the input-output gain G(S') of each frequency in the state S' is less than the input-output gain Gb of each frequency of the standard model (step S15 is), a positive feedback is given when the phase delay D(S') is smaller than the phase delay D(S), a negative feedback is given when the phase delay D(S') is larger than the phase delay D(S), and a zero feedback is given when the phase delay D(S') does not change compared to the phase delay D(S). As described above, the method of determining the feedback so as to reduce the phase delay is listed in three methods, but the first method is listed in the following example for explanation. In addition, the state S is in state S0 at the beginning of learning.

[0207] In step S17, specifically, for example, Figure 3In the phase line diagram, when changing from state S to state S', when the frequency when the phase delay is 180 degrees becomes smaller, it is defined as phase delay D(S)<phase delay D(S'), and the report output unit 4021 sets the report value to the second negative value through step S18. In addition, the absolute value of the second negative value is set to be smaller than the first negative value. When changing from state S to state S', when the frequency when the phase delay is 180 degrees becomes larger, it is defined as phase delay D(S)>phase delay D(S'), and the report output unit 4021 sets the report value to a positive value through step S19. In addition, when changing from state S to state S', when the frequency when the phase delay is 180 degrees does not change, it is defined as phase delay D(S)=phase delay D(S'), and the report output unit 4021 sets the report value to zero through step S20.

[0208] When any one of step S18, step S19, and step S20 is completed, in step S21, the value function update unit 4022 updates the value function Q stored in the value function storage unit 404 based on the reward value calculated in the step. Then, the process returns to step S11 again, and the above process is repeated, and the value function Q converges to an appropriate value. In addition, the process can be terminated on the condition that the above process is repeated a predetermined number of times or the above process is repeated for a predetermined time.

[0209] In addition, although online updating is exemplified in step S21 , the online updating may be replaced by batch updating or small batch updating.

[0210] Above, by reference Fig.12 In the operation described above, in this embodiment, the machine learning unit 400 is used to obtain the coefficients a that can obtain the transfer function F(s) of the filter 130. i 、b j The appropriate cost function for adjusting the filter 130 can make the coefficients a of the transfer function F(s) of the filter 130 i 、b j The optimization simplifies such effects.

[0211] Next, refer to Fig.13 The process of generating the optimization behavior information by the optimization behavior information output unit 405 is described.

[0212] First, in step S21, the optimization behavior information output unit 405 obtains the value function Q stored in the value function storage unit 404. As described above, the value function Q is a function updated by the value function update unit 4022 performing Q learning.

[0213] In step S22 , the optimization behavior information output unit 405 generates optimization behavior information based on the cost function Q, and outputs the generated optimization behavior information to the filter 130 .

[0214] In addition, by referring to Fig.13 In the operation described above, in this embodiment, the optimization behavior information can be generated based on the cost function Q obtained by learning by the machine learning unit 400, and the coefficients a of the transfer function F(s) of the currently set filter 130 can be simplified based on the optimization behavior information. i 、b j It can adjust the machine end and suppress the vibration, which can improve the quality of the workpiece processing surface.

[0215] Each structural unit included in the above-mentioned control device can be implemented by hardware, software or a combination thereof. In addition, the servo control method performed by the cooperation of each structural unit included in the above-mentioned control device can also be implemented by hardware, software or a combination thereof. Here, the so-called implementation by software means that the computer reads in the program and executes it to implement it.

[0216] Various types of non-transitory computer readable recording media (non-transitorycomputer readable medium) can be used to store the program and provide the program to the computer. Non-transitory computer readable recording media include various types of tangible recording media (tangible storage medium). Examples of non-transitory computer readable recording media include: magnetic recording media (e.g., hard disk drives), optical-magnetic recording media (e.g., optical magnetic disks), CD-ROM (Read Only Memory), CD-R, CD-R / W, semiconductor memories (e.g., mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (random access memory). In addition, the program can also be provided to the computer via various types of transitory computer readable recording media (transitory computer readable medium).

[0217] The above-described embodiment is a preferred embodiment of the present invention. However, the scope of the present invention is not limited to the above-described embodiment, and the present invention can be implemented in various modified forms without departing from the spirit of the present invention.

[0218] In the above embodiment, the case where the machine driven by the servo motor 150 has one resonance point is described, but sometimes the machine has multiple resonance points. When the machine has multiple resonance points, multiple filters are provided in a manner corresponding to each resonance point and connected in series, thereby all resonances can be attenuated. Fig.14 is a block diagram showing an example of directly connecting a plurality of filters to form a filter. Fig.14 In the case where there are m (m is a natural number greater than or equal to 2) resonance points, the filter 130 is formed by connecting m filters 130-1 to 130-m in series. i 、b j , and then use machine learning to find the optimal value for attenuating the resonance point.

[0219] In addition, in the above-mentioned embodiment, the frequency characteristic calculation unit calculates both the input-output gain and the input-output phase delay of the servo control device based on the input signal and the output signal of the servo control device whose frequency changes, the filter characteristic removal unit removes the filter characteristic of the initial filter from both the input-output gain and the input-output phase delay, and the machine learning unit starts the machine learning of the filter coefficient so that both the input-output gain and the phase delay after the filter characteristic of the initial filter is removed are reduced. However, in the above-mentioned embodiment, the frequency characteristic calculation unit may calculate one of the input-output gain and the input-output phase delay of the servo control device based on the input signal and the output signal of the servo control device whose frequency changes, the filter characteristic removal unit removes the filter characteristic of the initial filter from one of the input-output gain and the input-output phase delay, and the machine learning unit starts the machine learning of the filter coefficient so that one of the input-output gain and the phase delay after the filter characteristic of the initial filter is removed is reduced.

[0220] In addition, the structure of the control device Figure 1 In addition to the structure, the following structure also exists.

[0221] <Modification Example in Which the Machine Learning Unit is Provided Outside the Servo Control Unit>

[0222] Fig.15 This is a block diagram showing another configuration example of the control device. Fig.15 The control device 10A shown is Figure 1The control device 10 shown in the figure is different in that n (n is a natural number greater than or equal to 2) servo control units 100A-1 to 100A-n are connected to n machine learning units 400A-1 to 400A-n via a network 600, and each of the machine learning units 400A-1 to 400A-n has a frequency generating unit 200 and a frequency characteristic calculating unit 300. Figure 6 The servo control units 100A-1 to 100A-n correspond to the servo control devices, respectively, and the machine learning units 400A-1 to 400A-n correspond to the machine learning devices, respectively. In addition, of course, one or more of the frequency generating unit 200, the frequency characteristic calculating unit 300, and the initial setting unit 500 may be arranged outside the servo control units 100A-1 to 100A-n.

[0223] Here, the servo control unit 100A-1 and the machine learning unit 400A-1 are connected in a one-to-one pair so as to be able to communicate. The servo control units 100A-2 to 100A-n and the machine learning units 400A-2 to 400A-n are also connected in the same manner as the servo control unit 100A-1 and the machine learning unit 400A-1. Fig.15 In the embodiment, n groups of servo control units 100A-1 to 100A-n and machine learning units 400A-1 to 400A-n are connected via a network 600, and the servo control units and machine learning units of each group can be directly connected via a connection interface. For example, multiple groups of these servo control units 100A-1 to 100A-n and machine learning units 400A-1 to 400A-n can be set in the same factory, or they can be set in different factories.

[0224] The network 600 is, for example, a LAN (Local Area Network) constructed in a factory, the Internet, a public telephone network, or a combination thereof. The specific communication method in the network 600 is not particularly limited to wired connection or wireless connection.

[0225] <Degrees of freedom of system structure>

[0226] In the above-mentioned embodiment, the servo control units 100A-1 to 100A-n and the machine learning units 400A-1 to 400A-n are respectively connected in a one-to-one group in a communicative manner, but for example, one machine learning unit can also be connected to multiple servo control units via the network 600 in a communicative manner to implement machine learning of each servo control unit.

[0227] In this case, each function of a machine learning unit can be implemented as a distributed processing system that is appropriately distributed across multiple servers. In addition, each function of a machine learning unit can also be implemented on the cloud using a virtual server function or the like.

[0228] Furthermore, when there are n machine learning units 400A-1 to 400A-n corresponding to n servo control units 100A-1 to 100A-n of the same model name, same specification, or same series, the learning results of the machine learning units 400A-1 to 400A-n can be shared, thereby building a more ideal model.

Claims

1. A machine learning system that performs machine learning to optimize coefficients of at least one filter provided in a servo control device that controls a motor, characterized in that: The machine learning system has: an initial setting unit that sets an initial value of a coefficient of the filter so as to attenuate at least one specific frequency component; a frequency characteristic calculation unit for calculating at least one of an input-output gain and an input-output phase delay of the servo control device based on an input signal and an output signal of the servo control device whose frequency varies; and a filter characteristic removing unit that removes the filter characteristic of the filter for which the initial value is set from at least one of the input-output gain and the input-output phase delay obtained based on the output signal and the input signal when the machine learning starts, wherein the output signal is obtained using the filter for which the initial value is set by the initial setting unit, Machine learning of the filter coefficients is started so that at least one of the input-output gain of the filter characteristic of the filter with the initial value set is reduced and the phase delay of the filter characteristic of the filter with the initial value set is reduced.

2. The machine learning system according to claim 1, characterized in that The frequency-varying input signal is a frequency-varying sine wave, The sine wave is generated by a frequency generating unit, and the frequency generating unit is provided inside or outside the servo control device.

3. The machine learning system according to claim 1, characterized in that The machine learning system comprises a machine learning unit, The machine learning department has: a state information acquisition unit that acquires state information, wherein the state information includes the input / output gain and the input / output phase delay of the servo control device output from the frequency characteristic calculation unit, and the coefficient of the filter to which the initial value is set or the coefficient of the filter after the machine learning starts; a behavior information output unit configured to output the behavior information included in the state information, the behavior information including adjustment information of the coefficient of the filter to which the initial value is set or the coefficient of the filter after the machine learning starts; a reward output unit that outputs a reward value in reinforcement learning based on the input-output gain and the input-output phase delay output from the state information acquisition unit; and A value function updating unit updates a behavior value function according to the reward value output by the reward output unit, the state information, and the behavior information.

4. The machine learning system according to claim 3, characterized in that The frequency characteristic calculation unit outputs the input-output gain and the input-output phase delay, When the input-output gain of the servo control device is equal to or smaller than the input-output gain of a standard model of the input-output gain calculated based on the characteristics of the servo control device, the feedback output unit calculates a feedback of the input-output phase delay.

5. The machine learning system according to claim 4, characterized in that The input and output gains of the standard model are fixed values ​​above a specified frequency.

6. The machine learning system according to claim 4, characterized in that The feedback output unit calculates a feedback so that a phase delay between the input and output becomes small.

7. The machine learning system according to claim 3, characterized in that The machine learning system includes an optimization behavior information output unit that outputs adjustment information of the coefficient based on the value function updated by the value function updating unit.

8. A control device, characterized in that: have: The machine learning system according to any one of claims 1 to 7; and A servo control device for controlling a motor includes at least one filter for attenuating a specific frequency component.

9. A machine learning method of a machine learning system, the machine learning system performing machine learning to optimize coefficients of at least one filter provided in a servo control device, the servo control device controlling a motor, the filter attenuating at least one specific frequency component, characterized in that: setting initial values ​​of the coefficients of the filter so as to attenuate at least one specific frequency component, Calculating at least one of an input-output gain and an input-output phase delay of the servo control device according to an input signal and an output signal of the servo control device with frequency variations, At the start of machine learning, the filter characteristic of the filter for which the initial value is set is removed from at least one of an input-output gain and an input-output phase delay obtained based on an output signal of the servo control device and an input signal of the servo control device, wherein the output signal is obtained using the filter for which the initial value is set, Machine learning of the filter coefficients is started so that at least one of the input-output gain of the filter characteristic of the filter with the initial value set is reduced and the phase delay of the filter characteristic of the filter with the initial value set is reduced.

Images