Evaluation method, evaluation device, and evaluation program

Abstract

When update of a learning model has been detected, this evaluation device: searches for, from among a plurality of operation data items registered in advance in a memory, an operation data item having an abnormality degree which has been changed before and after the update of the learning model; and displays, on a display, the searched operation data item and the abnormality degree corresponding to the operation data item.

Description

Evaluation method, evaluation device, and evaluation program

[0001] The present disclosure relates to techniques for evaluating learning models.

[0002] Patent Document 1 discloses an abnormality information estimation system that inputs operational data related to the operation of industrial equipment into a learning model, identifies multiple unit phenomena caused by the operation, and estimates abnormal phenomena that have occurred during the operation based on the identified multiple unit phenomena.

[0003] However, the technology of Patent Document 1 does not disclose how to evaluate a learning model, and therefore further improvements are needed to properly evaluate a learning model that is applied to a controlled device such as equipment.

[0004] JP 2023-067108 A

[0005] The present disclosure has been made to solve such problems, and aims to provide a technology for appropriately evaluating a learning model applied to a device to be controlled.

[0006] An evaluation method in one aspect of the present disclosure is a method for evaluating a learning model used to determine whether at least one of a load and a servo motor is normal or abnormal in a controlled device including a load, a servo amplifier, a servo motor that powers the load in accordance with the control of the servo amplifier, and a motion controller that outputs a command signal to the servo amplifier, wherein the learning model outputs an abnormality degree for at least one of the load and the servo motor from operation data of the servo motor, and when a computer detects an update to the learning model, it searches for operation data in which the abnormality degree has changed before and after the update of the learning model from multiple operation data pre-registered in memory, and displays the searched operation data and the abnormality degree corresponding to the operation data on a display.

[0007] This configuration allows the learning model applied to the controlled device to be appropriately evaluated.

[0008] FIG. 1 is a diagram illustrating an example of a configuration of a production system according to an embodiment of the present disclosure. FIG. 2 is a diagram illustrating an operation mode. FIG. 3 is a graph illustrating a relationship between a threshold and an abnormality degree. FIG. 4 is a flowchart illustrating an example of processing by an evaluation device in a storage phase in which operation data is stored. FIG. 5 is an explanatory diagram of processing in which operation data is stored in an operation data storage unit. FIG. 6 is a flowchart illustrating an example of processing for displaying operation data. FIG. 7 is a diagram illustrating an example of an evaluation screen.

[0009] (Findings underlying the present disclosure) In production facilities that produce products by operating loads such as industrial robots with servo motors, a learning model is used to determine abnormalities in the loads and servo motors. In order to confirm whether the learning model is being successfully applied to the production facility, it is necessary to evaluate the performance of the learning model. If the performance evaluation results show that the learning model is not being successfully applied to the site, it is necessary to retrain the learning model or change the threshold for abnormality determination.

[0010] Since production equipment operates normally in most cases, it is easy to collect operational data during normal operation (hereinafter referred to as normal data). On the other hand, since it is not known when an abnormality will occur in production equipment, it is difficult to obtain operational data during abnormal operation (hereinafter referred to as abnormal data). For this reason, the learning model applied to production equipment is machine-learned using normal data.

[0011] However, quantitative evaluation of learning models applied to production equipment also requires a large amount of abnormal data. This is because quantitative evaluation requires evaluation indicators for abnormal data, such as false negatives and true negatives. As mentioned above, production equipment operates normally in most cases, so it is difficult to obtain a large amount of abnormal data. Therefore, quantitative evaluation of learning models applied to production equipment is difficult.

[0012] Therefore, the inventor discovered that a learning model applied to a controlled device such as production equipment can be appropriately evaluated by evaluating the learning model qualitatively rather than quantitatively, and came up with the present disclosure.

[0013] (1) An evaluation method in one aspect of the present disclosure is a method for evaluating a learning model used to determine whether at least one of a load and a servo motor is normal or abnormal in a controlled device including a load, a servo amplifier, a servo motor that powers the load in accordance with the control of the servo amplifier, and a motion controller that outputs a command signal to the servo amplifier, wherein the learning model outputs an abnormality degree for at least one of the load and the servo motor from operation data of the servo motor, and when a computer detects an update to the learning model, the computer searches for operation data in which the abnormality degree has changed before and after the update of the learning model from multiple operation data pre-registered in memory, and displays the searched operation data and the abnormality degree corresponding to the operation data on a display.

[0014] According to this configuration, when an update of the learning model is detected, the display shows the operation data whose anomaly level has changed before and after the update of the learning model, and the anomaly level corresponding to that operation data. This allows the qualitative evaluation result of the learning model to be displayed on the display, allowing the machine learning model to be applied to the controlled device to be appropriately evaluated.

[0015] (2) In the evaluation method described in (1) above, the operation data displayed on the display may include a command signal for driving the servo motor and a measurement signal of the servo motor when the servo motor operates based on the command signal.

[0016] In this case, it is possible to present the command signal and the measurement signal when there is a change in the abnormality level.

[0017] (3) In the evaluation method described in (1) or (2) above, the display may include superimposing a first degree of abnormality output when the searched operation data is input into the learning model before the update and a second degree of abnormality output when the searched operation data is input into the learning model after the update.

[0018] In this case, the change in the degree of abnormality before and after the update can be presented in an easy-to-understand manner.

[0019] (4) In the evaluation method described in any one of (1) to (3) above, the operational data displayed on the display may include at least one of the operator's judgment of normality or abnormality, the measurement date and time of the searched operational data, a description by the operator of the cause of the abnormality, and a description of the work to restore the controlled device to normal operation.

[0020] In this case, further information can be provided for evaluating the validity of the performance of the updated learning model.

[0021] (5) In the evaluation method described in (2) above, the display may include displaying the retrieved motion data and the degree of abnormality corresponding to the retrieved motion data so that the areas where the degree of abnormality has changed before and after updating the learning model are highlighted.

[0022] In this case, it becomes easy to confirm which part of the searched operation data and the abnormality level corresponding to that operation data has changed in abnormality level.

[0023] (6) In the evaluation method described in any one of (1) to (5) above, the display on the display may include displaying the difference in the degree of anomaly before and after updating the learning model.

[0024] In this case, it becomes easy to check how much the abnormality level has changed before and after the update.

[0025] (7) In the evaluation method described in any one of (1) to (6) above, the servo motor may include a plurality of operating modes, and the display may include displaying the operating data and the abnormality level by distinguishing periods corresponding to the plurality of operating modes.

[0026] In this case, it becomes easy to check the operation mode in which the abnormality level has changed.

[0027] (8) In the evaluation method described in any one of (1) to (7) above, the computer may be any one of the servo amplifier, the motion controller, a personal computer, and a cloud server.

[0028] In this case, the operation data in which the degree of abnormality has been changed by any of the servo amplifier, the motion controller, the personal computer, and the cloud server is searched for.

[0029] (9) In the evaluation method described in any one of (1) to (8) above, the memory may be any one of the servo amplifier, the motion controller, a personal computer, and a cloud server.

[0030] In this case, the operation data is accumulated by any one of the servo amplifier, the motion controller, the personal computer, and the cloud server.

[0031] (10) In the evaluation method described in any one of (1) to (9) above, the motion data may be time-series data for a period from the start to the end of a certain motion pattern for the load.

[0032] In this case, the movement data for the period from the start to the end of one movement pattern can be presented in chronological order.

[0033] (11) In the evaluation method described in any one of (1) to (10) above, the learning model may be generated by unsupervised machine learning of operational data of the servo motor under normal conditions.

[0034] In this case, the learning model obtained by unsupervised machine learning of normal operation data can be appropriately evaluated.

[0035] (12) In the evaluation method described in any of (1) to (11) above, the search for the motion data may include: performing a process for each of the plurality of motion data to calculate the difference between a first degree of abnormality output when the motion data is input into a learning model before the update and a second degree of abnormality output when the motion data is input into a learning model after the update; and ranking the plurality of motion data so that the larger the difference, the higher the ranking.

[0036] In this case, the operational data can be ranked in descending order of the change in the degree of abnormality before and after the update.

[0037] (13) In the evaluation method described in any one of (1) to (12) above, the display on the display may include switching the display of the operation data and the abnormality degree in descending order of rank in response to a switching instruction from an operator.

[0038] In this case, the operation data and the abnormality levels corresponding to the operation data can be presented in descending order of the change in the abnormality level.

[0039] (14) In another aspect of the present disclosure, an evaluation device is an evaluation device that evaluates a learning model used to determine whether at least one of a load and a servo motor is normal or abnormal in a controlled device including a load, a servo amplifier, a servo motor that powers the load in accordance with the control of the servo amplifier, and a motion controller that outputs a command signal to the servo amplifier, wherein the learning model outputs an abnormality degree for at least one of the load and the servo motor from operation data of the servo motor, and when a processor of the evaluation device detects an update of the learning model, the evaluation device executes a process of searching for operation data in which the abnormality degree has changed before and after the update of the learning model from multiple operation data pre-registered in memory, and displaying the searched operation data and the abnormality degree corresponding to the operation data on a display.

[0040] This configuration makes it possible to provide an evaluation device that can appropriately evaluate a learning model that is applied to a device to be controlled.

[0041] (15) In another aspect of the present disclosure, an evaluation program causes a computer to execute an evaluation method for evaluating a learning model used to determine whether at least one of a load and a servo motor is normal or abnormal in a controlled device including a load, a servo amplifier, a servo motor that powers the load in accordance with control of the servo amplifier, and a motion controller that outputs a command signal to the servo amplifier, wherein the learning model outputs an abnormality degree for at least one of the load and the servo motor from operation data of the servo motor, and when an update of the learning model is detected, the computer executes a process of searching, from multiple operation data pre-registered in memory, for operation data whose abnormality degree has changed before and after the update of the learning model, and displaying the searched operation data and the abnormality degree corresponding to the operation data on a display.

[0042] This configuration makes it possible to provide an evaluation program that can appropriately evaluate a learning model that is applied to a device to be controlled.

[0043] The present disclosure can also be realized as a system operated by such an evaluation program. Needless to say, such a computer program can be distributed on a computer-readable non-transitory recording medium such as a CD-ROM or via a communication network such as the Internet.

[0044] Note that each of the embodiments described below represents a specific example of the present disclosure. The numerical values, shapes, components, steps, and step orders shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not described in the independent claims that represent the highest concept are described as optional components. Furthermore, in all of the embodiments, the respective contents can be combined.

[0045] 1 is a diagram illustrating an example of the configuration of a production system 1 according to an embodiment of the present disclosure. The production system 1 includes an evaluation device 10, a motion controller 20, a servo amplifier 30, a servo motor 40, and a load 50. The motion controller 20, the servo amplifier 30, the servo motor 40, and the load 50 constitute a production facility 60. The production facility 60 is an example of a device to be controlled.

[0046] The evaluation device 10 is a device that evaluates the performance of a learning model used to determine an abnormality in at least one of the servo motor 40 and the load 50. Hereinafter, at least one of the servo motor 40 and the load 50 will be referred to as a target device. The evaluation device 10 may be configured as a personal computer installed at the site where the motion controller 20, the servo amplifier 30, the servo motor 40, and the load 50 are installed, or may be configured as a cloud server. The motion controller 20, the servo amplifier 30, and the evaluation device 10 are connected via a network. Examples of the network NT are a local area network or the Internet. The motion controller 20 and the servo amplifier 30, and the servo amplifier 30 and the servo motor 40 are connected via a LAN cable or the like.

[0047] The load 50 is, for example, composed of a production device used to produce a product. An example of a production device is an industrial robot that performs, for example, equipment mounting, processing, machining, or transport. The load 50 includes a work arm that grips and processes parts. The work arm includes multiple arm elements and one or more joints that connect the multiple arm elements. The load 50 is installed, for example, on a factory production line. The load 50 has been described as being composed of an industrial robot, but this is just one example, and the load 50 may be composed of any device that is driven by a servo motor 40.

[0048] The servo motor 40 is a motor that precisely operates the load 50 in accordance with a drive signal output from the servo amplifier 30. For example, the servo motor 40 is provided at a joint of a working arm and rotates an arm element in the forward or reverse direction by a predetermined angle. The servo motor 40 includes a sensor (not shown) that detects the state of the servo motor 40, and inputs a measurement signal detected by the sensor to the servo amplifier 30. An example of a sensor is a torque sensor that detects the torque of the servo motor 40. Therefore, the measurement signal is a torque signal. However, this is just one example, and the measurement signal may be an acceleration signal or a velocity signal, and is not limited to a torque signal.

[0049] The servo amplifier 30 controls the servo motor 40 so that the servo motor 40 operates in accordance with the command signal output from the motion controller 20. The servo amplifier 30 generates a drive signal according to the command signal output from the motion controller 20 and inputs it to the servo motor 40. The servo amplifier 30 feedback-controls the servo motor 40 based on the measurement signal output from the servo motor 40 so that the servo motor 40 operates in accordance with the command signal.

[0050] The motion controller 20 generates a command signal for operating the servo motor 40 in a predetermined operation pattern, and inputs the generated command signal to the servo amplifier 30. The command signal is time-series data that specifies the position of the servo motor 40 from the start to the end of a certain operation pattern.

[0051] Note that the command signal has been described as a position command signal that specifies the position of the servo motor 40, but this is just one example, and the command signal may be a speed command signal, an acceleration command signal, or a torque command signal that specifies the speed, acceleration, or torque of the servo motor 40. Alternatively, the command signal may include at least two of a position command signal, a speed command signal, an acceleration command signal, and a torque command signal.

[0052] The evaluation device 10 includes a processor 11, a communication device 12, an input device 13, a display 14, and a memory 15. The processor 11 is configured as a central processing unit (CPU), and includes an acquisition unit 111, a search unit 112, a display control unit 113, an abnormality determination unit 114, and a learning unit 115. The acquisition unit 111 to the learning unit 115 may be configured as dedicated integrated circuits such as ASICs.

[0053] The acquiring unit 111 acquires the command signals and measurement signals transmitted from the servo amplifier 30 using the communication device 12. The acquiring unit 111 generates operation data based on the acquired command signals and measurement signals, and stores the generated operation data in the operation data storage unit 151.

[0054] The operation data includes a command signal for driving the servo motor 40, a measurement signal of the servo motor 40 when the servo motor 40 operates based on the command signal, and an identifier for identifying the operation data.

[0055] Furthermore, the operation data may include the operator's determination of normality or abnormality, the measurement date and time of the operation data, a description by the operator of the cause of the abnormality, and a description of the work required to restore the production equipment 60 to normal operation. The operator is a person who manages the production equipment 60. The operator's determination of normality or abnormality, the description by the operator of the cause of the abnormality, and the description of the work required to restore the production equipment 60 to normal operation are input by the operator on the evaluation screen 500 described below. The measurement date and time of the operation data is the year, month, day, and time when the measurement signal included in the operation data was measured. Furthermore, the operation data may include the determination of normality or abnormality made by the abnormality determination unit 114 based on the degree of abnormality output by the learning model.

[0056] The search unit 112 detects whether the learning model has been updated. Here, the search unit 112 may determine that the learning model has been updated when the input device 13 receives an operator's instruction indicating that the learning model has been updated. When the search unit 112 detects that the learning model has been updated, it searches for action data whose anomaly level has changed before and after the update of the learning model from among multiple action data pre-registered in the action data storage unit 151.

[0057] The learning model is generated by unsupervised machine learning of normal operation data of the target device (hereinafter referred to as normal data). The learning model is a model that receives command signals and measurement signals and outputs the degree of abnormality of the target device from the input command signals and measurement signals. Therefore, the normal command signals and measurement signals included in the operation data stored in the operation data storage unit 151 are used as learning data for the learning model.

[0058] As the learning model algorithm, for example, the k-nearest neighbor method, the k-means method, etc. can be adopted. The degree of anomaly is an index that represents the degree of deviation between the operation data and normal data, and the value increases as the degree of deviation from normal data increases, and decreases as the degree of deviation from normal data decreases.

[0059] An example of the processing of the learning model when the k-nearest neighbor method is adopted will be described below. The learning model calculates the distance between a command signal and a measurement signal to be judged and multiple command signals and measurement signals in normal states. Next, the learning model extracts the top k normal command signals and measurement signals from the multiple normal command signals and measurement signals in descending order of distance from the command signal and measurement signal to be judged. Next, the learning model calculates the average value of the distances between the extracted top k command signals and measurement signals and the command signal and measurement signal to be judged as the degree of abnormality. Note that the distance is the Euclidean distance between a vector defined by the values ​​of the command signal and measurement signal in normal states at a certain time point t and a vector defined by the values ​​of the command signal and measurement signal to be judged at time point t. As a result, the degree of abnormality is composed of time series data calculated according to time point t.

[0060] The search unit 112 performs a process for each of the multiple pieces of motion data to calculate the difference between the first degree of abnormality output when the motion data is input into the learning model before the update (hereinafter referred to as the old model) and the second degree of abnormality output when the motion data is input into the learning model after the update (hereinafter referred to as the new model), and ranks the multiple pieces of motion data so that the larger the difference, the higher the ranking.

[0061] The display control unit 113 displays the operation data searched by the search unit 112 and the abnormality level corresponding to the operation data on the display 14. In detail, the display control unit 113 displays the evaluation screen shown in FIG.

[0062] The display control unit 113 switches the display of the operation data and the abnormality level in descending order of rank in response to a switching instruction from the operator. The switching instruction is input by the operator using the input device 13. The evaluation screen 500 displays one operation data and the abnormality level corresponding to that operation data. Therefore, the display control unit 113 first displays the operation data and the abnormality level that are ranked first on the evaluation screen 500, and when a switching instruction is input, the display control unit 113 displays the operation data and the abnormality level that are ranked second on the evaluation screen 500, and so on, switching the display of the operation data and the abnormality level in descending order of rank.

[0063] The display control unit 113 superimposes and displays the first abnormality level output when a command signal and a measurement signal are input to the old model and the second abnormality level output when a command signal and a measurement signal are input to the new model.

[0064] The display control unit 113 may display the command signal and the measurement signal, as well as the abnormality levels corresponding to both signals, so that the areas where the abnormality level has changed before and after updating the learning model are highlighted.

[0065] The display control unit 113 may display the difference in the degree of anomaly before and after updating the learning model.

[0066] The display control unit 113 may distinguish periods corresponding to a plurality of operation modes of the servo motor 40 and display the command signal and the measurement signal, as well as the abnormality levels corresponding to both signals.

[0067] The abnormality determination unit 114 inputs the command signal and measurement signal to be determined, which are the command signal and measurement signal output from the servo amplifier 30 while the production equipment 60 is in operation, into a learning model to calculate the degree of abnormality, and compares the calculated degree of abnormality with a threshold value to determine whether the target device is normal. Here, the learning model used is a new model. If the degree of abnormality is equal to or greater than the threshold value, the abnormality determination unit 114 determines that the target device is abnormal, and if the degree of abnormality is less than the threshold value, it determines that the target device is normal.

[0068] The threshold value may be different for each of a plurality of operation modes, including an acceleration mode, a deceleration mode, and a constant speed mode, and the constant speed mode further includes a transient mode and a steady mode.

[0069] 2 is a diagram for explaining the operation modes, which shows, from top to bottom, the waveforms of a position command signal, a velocity command signal, an acceleration command signal, and a measurement signal.

[0070] The operating period of the servo motor 40 is divided into an acceleration period P1, a deceleration period P2, and a constant speed period P3. The constant speed period P3 is further divided into a transient period P31 including the initial period of the constant speed period P3, and a steady period P32 including the final period of the constant speed period P3. The transient period P31 is a period during which the value of the speed command signal is zero but the value of the measurement signal is greater than a predetermined value due to inertia. The steady period P32 is a period during which the value of the measurement signal is equal to or less than the predetermined value.

[0071] The abnormality determination unit 114 acquires a position command signal from the servo amplifier 30 and generates a speed command signal by differentiating the acquired position command signal. The abnormality determination unit 114 generates an acceleration command signal by differentiating the speed command signal. The abnormality determination unit 114 sets a period during which the absolute value of the acceleration command signal is equal to or greater than a certain value and has a positive sign as an acceleration period P1. The abnormality determination unit 114 sets a period during which the absolute value of the acceleration command signal is equal to or greater than a certain value and has a negative sign as a deceleration period P2. The abnormality determination unit 114 sets a period during which the absolute value of the acceleration command signal is less than a certain value as a constant speed period P3. The abnormality determination unit 114 sets a period from the start of the constant speed period P3 until a predetermined time has elapsed as a transient period P31. The abnormality determination unit 114 sets the remaining period from the end of the transient period P31 in the constant speed period P3 as a steady period P32.

[0072] The abnormality determination unit 114 determines that the operating mode of the servo motor 40 during the acceleration period P1 is the acceleration mode, the operating mode of the servo motor 40 during the deceleration period P2 is the deceleration mode, the operating mode of the servo motor 40 during the constant speed period P3 is the constant speed mode, the operating mode of the servo motor 40 during the transient period P31 is the transient mode, and the operating mode of the servo motor 40 during the steady period P32 is the steady mode.

[0073] FIG. 3 is a graph 300 showing the relationship between the threshold and the abnormality level. In FIG. 3, the vertical axis represents the abnormality level, and the horizontal axis represents time. The abnormality level 303 is divided into an acceleration period P1, a deceleration period P2, and a constant speed period P3. The constant speed period P3 is further divided into a transient period P31 and a steady period P32. A different threshold value 302 is set for each operating period, i.e., a different value for each operating mode. In this example, the threshold value 302 decreases in the order of acceleration period P1, deceleration period P2, transient period P31, and steady period P32. If the abnormality level 303 is equal to or greater than the threshold value 302, the abnormality determination unit 114 determines that the target device is abnormal. If the abnormality level 303 is less than the threshold value, the abnormality determination unit 114 determines that the target device is normal. Therefore, for example, the steady period P32 is more likely to be determined to be abnormal than the acceleration period P1 and the deceleration period P2. In this way, by setting different threshold values ​​depending on the operating mode, appropriate abnormality determination can be performed depending on the operating mode.

[0074] Returning to Fig. 1, the learning unit 115 updates the learning model stored in the learning model storage unit 152. The learning unit 115 may update the learning model when the input device 13 acquires an update instruction from an operator. The learning unit 115 may update the learning model using unlearned command signals and measurement signals acquired during the period from the last time the learning model was updated until the update instruction was input, or may update the learning model using command signals and measurement signals obtained by adding unlearned command signals and measurement signals to the command signals and measurement signals used to learn the old model.

[0075] The communication device 12 connects the evaluation device 10 to the network NT. The communication device 12 receives command signals and measurement signals corresponding to the command signals from the servo amplifier 30. The communication device 12 may also receive command signals and measurement signals corresponding to the command signals from the motion controller 20.

[0076] The input device 13 is configured with a mouse, a keyboard, a touch panel, or the like, and receives instructions from the user.

[0077] The display 14 is configured with a display device such as a liquid crystal display or an organic EL display, and displays an evaluation screen 500 shown in FIG.

[0078] The memory 15 is configured by a hard disk drive (HDD) or a solid state drive (SSD), and includes an operation data storage unit 151, a learning model storage unit 152, and a threshold storage unit 153.

[0079] The motion data storage unit 151 stores the motion data generated by the acquisition unit 111 .

[0080] The learning model storage unit 152 stores the new model and the old model. The learning model may be configured as a learning model that has been pre-machine-trained for each operation mode. For example, the learning model may be configured as a learning model corresponding to an acceleration mode, a deceleration mode, and a transient mode. For example, the learning model for the acceleration mode is generated by unsupervised machine learning of the command signal and measurement signal in a normal state in the acceleration mode. In this case, the abnormality determination unit 114 may calculate the degree of abnormality by inputting the command signal and measurement signal into the learning model for the corresponding operation mode, such as inputting the command signal and measurement signal for the acceleration mode into the learning model for the acceleration mode and inputting the command signal and measurement signal for the deceleration mode into the learning model for the deceleration mode.

[0081] The threshold storage unit 153 stores thresholds to be compared with the abnormality degrees calculated by the learning model. In this embodiment, the thresholds stored correspond to the acceleration mode, deceleration mode, transient mode, and steady mode, respectively.

[0082] 4 is a flowchart showing an example of processing by the evaluation device 10 in the storage phase in which operation data is stored. First, in step S1, the acquisition unit 111 acquires command signals and measurement signals transmitted from the servo amplifier 30 using the communication device 12. When the production facility 60 is in operation, the servo amplifier 30 transmits command signals and measurement signals at a predetermined sampling rate, and therefore the acquisition unit 111 acquires the command signals and measurement signals at the predetermined sampling rate.

[0083] Next, in step S2, the acquisition unit 111 generates operation data by assigning identifiers to the command signals and measurement signals acquired in step S1. Here, the operation data is managed in units of one operation pattern. Therefore, when the acquisition unit 111 acquires all the time-series data of the command signals and measurement signals from the start to the end of one operation pattern, the acquisition unit 111 assigns identifiers and the measurement dates and times of the operation data to the time-series data of the command signals and measurement signals, and generates operation data.

[0084] Next, in step S3, the acquisition unit 111 stores the operation data generated in step S2 in the operation data storage unit 151. The operator can input, from the evaluation screen 500, the result of the judgment of normality or abnormality, a description of the cause of the abnormality, and a description of the work required to restore the production equipment 60 to normal operation for the stored operation data. When this information is input, the acquisition unit 111 simply includes this information in the contents of the operation data. When the processing of step S3 is completed, the processing returns to step S1. That is, the processing of FIG. 4 is repeatedly executed while the production equipment 60 is in operation.

[0085] 5 is an explanatory diagram of the process by which operation data is stored in the operation data storage unit 151. When the acquisition unit 111 acquires a command signal and a measurement signal corresponding to one operation pattern, it inputs the acquired command signal and measurement signal to the abnormality determination unit 114. The abnormality determination unit 114 inputs the command signal and measurement signal to a new model to calculate the degree of abnormality. The abnormality determination unit 114 compares the degree of abnormality with a threshold value and determines whether the target device is abnormal or normal. When the abnormality determination unit 114 determines that the target device is abnormal, the acquisition unit 111 associates an abnormal label with the command signal and the measurement signal. On the other hand, when the abnormality determination unit 114 determines that the target device is normal, the acquisition unit 111 associates a normal label with the command signal and the measurement signal.

[0086] When an operator who has actually confirmed whether or not the target device is abnormal inputs an instruction indicating an abnormality using the input device 13, the acquisition unit 111 associates an abnormality label with the command signal and the measurement signal. On the other hand, when an operator inputs an instruction indicating a normality using the input device 13, the acquisition unit 111 associates a normality label with the command signal and the measurement signal.

[0087] When an operator inputs a comment using the input device 13, the acquisition unit 111 associates the comment with the command signal and the measurement signal. The comment includes the date and time when the abnormality occurred, a memo describing the confirmation content for the target device, the content of the recovery work, etc.

[0088] The acquiring unit 111 generates operation data by associating labels and comments with command signals and measurement signals, and stores the generated operation data in the operation data storage unit 151. At this time, the acquiring unit 111 stores operation data including a normal label in the normal data storage unit 161, and stores operation data including an abnormal label in the abnormal data storage unit 162.

[0089] In this way, the acquiring unit 111 distinguishes between the action data including the normal label and the action data including the abnormal label and stores the data in the action data storage unit 151 .

[0090] 6 is a flowchart showing an example of a process for displaying action data. In step S11, the search unit 112 detects that the learning model has been updated in accordance with an instruction from the operator.

[0091] Next, steps S12 and S13 are performed in parallel. In step S12, the search unit 112 inputs the plurality of pieces of motion data stored in the motion data storage unit 151 into the old model, and calculates a first abnormality degree corresponding to each of the plurality of pieces of motion data.

[0092] In step S13, the search unit 112 inputs the same multiple pieces of action data as the multiple pieces of action data input to the old model into the new model, and calculates a second abnormality degree corresponding to each of the multiple pieces of action data.

[0093] Next, in step S14, the search unit 112 calculates the difference between the first degree of abnormality and the second degree of abnormality for each of the plurality of pieces of motion data. This difference is the total value of the differences between the values ​​of the first degree of abnormality and the second degree of abnormality calculated for each of the plurality of sample points from the start point to the end point of the first degree of abnormality.

[0094] Next, in step S15, the search unit 112 ranks the plurality of pieces of action data so that the larger the difference, the higher the rank.

[0095] Next, in step S16, the display control unit 113 generates an evaluation screen 500 that displays the action data according to the ranking calculated in step S15, and displays the generated evaluation screen 500 on the display 14.

[0096] 7 is a diagram showing an example of the evaluation screen 500. A first display column 510, a second display column 520, and a third display column 530 are displayed in the right column of the evaluation screen 500. The first display column 510 displays a command signal 511. The second display column 520 displays a measurement signal 521 corresponding to the command signal 511. The third display column 530 displays a first abnormality level 531 and a second abnormality level 532 corresponding to the command signal 511 and the measurement signal 521 in a superimposed manner. In the first display column 510 to the third display column 530, the vertical axis represents the value of each signal, and the horizontal axis represents time. That is, the first display column 510 to the third display column 530 display the command signal 511, the measurement signal 521, the first abnormality level 531, and the second abnormality level 532 in chronological order.

[0097] In this example, the first display field 510 to the third display field 530 display the command signal 511, the measurement signal 521, the first abnormality degree 531, and the second abnormality degree 532 ranked nth (n is a natural number). The evaluation screen 500 includes a first switching button (not shown) for switching the display to the next rank and a second switching button (not shown) for switching the display to the next rank. When the input device 13 receives an operation to press the first switching button, the display control unit 113 displays the command signal 511, the measurement signal 521, the first abnormality degree 531, and the second abnormality degree 532 ranked n+1. On the other hand, when the input device 13 receives an operation to press the second switching button, the display control unit 113 displays the command signal 511, the measurement signal 521, the first abnormality degree 531, and the second abnormality degree 532 ranked n-1. By default, the first display field 510 to the third display field 530 display the first-ranked command signal 511, measurement signal 521, first abnormality degree 531, and second abnormality degree 532.

[0098] The first to third display columns 510 to 530 distinguish between the acceleration period, deceleration period, transient period, and steady period. For example, the first to third display columns 510 to 530 distinguish between the operation periods by displaying the background of each operation period in a different color, by displaying a border around each operation period, or by displaying text indicating the operation period in each operation period. This allows the first to third display columns 510 to 530 to distinguish between the multiple operation modes.

[0099] The first to third display columns 510 to 530 display highlight objects 590 indicating that there has been a change in the abnormality level. Here, the highlight objects 590 are displayed during periods in which the second abnormality level 532 is equal to or greater than the threshold. The highlight objects 590 are objects that display, in a semi-transparent color, rectangular areas indicating periods in which the second abnormality level is equal to or greater than the threshold.

[0100] The third display field 530 displays the first abnormality degree 531 and the second abnormality degree in a superimposed manner, so that it is possible to display the difference in the abnormality degree before and after updating the learning model.

[0101] The third display field 530 displays the first abnormality degree 531 and the second abnormality degree 532, allowing the operator to confirm that an event that was not determined to be abnormal in the old model was determined to be abnormal in the new model, thereby enabling a qualitative evaluation of the new model. Furthermore, the evaluation screen 500 displays the command signal 511, the measurement signal 521, the first abnormality degree 531, and the second abnormality degree 532. Therefore, the operator can compare the waveform of the command signal 511, the waveform of the measurement signal 521, the waveform of the first abnormality degree 531, and the waveform of the second abnormality degree to confirm the validity of the determination made by the new learning model and qualitatively evaluate the new learning model.

[0102] Additionally, an indicator 533 indicating the threshold for each operation period is displayed in the third display field 530. The operator can change the threshold by sliding the indicator 533 upward or downward using the input device 13. The changed threshold is stored in the threshold storage unit 153.

[0103] The left column of the evaluation screen 500 displays an event summary display column 540 , a label display column 550 , an abnormality cause display column 560 , and a recovery content display column 570 .

[0104] The event summary display field 540 displays events corresponding to the command signal 511, the measurement signal 521, the first abnormality level 531, and the second abnormality level 532 displayed in the right field of the evaluation screen 500. An event is information indicating an occurrence of an abnormality in the servo motor 40. In this example, an abnormality occurred during the acceleration period in operation pattern A, and a message indicating this is displayed as an event.

[0105] The label display field 550 is a field where an operator who has actually checked whether or not the target device has an abnormality enters the confirmation result. An operator who judges the target device to be normal enters a comment indicating that the device is normal in the label display field 550, and an operator who judges the target device to be abnormal enters a comment indicating that the device is abnormal in the label display field 550. For example, if the judgment result by the new learning model is normal but an abnormality actually occurs in the target device, the operator can enter a comment indicating this in the label display field 550. Note that the label display field 550 may display a normal label or an abnormal label generated based on the judgment result by the abnormality judgment unit 114.

[0106] The abnormality cause display field 560 is a field where the operator inputs the cause of the abnormality of the target device. In this example, slippage of the belt connected to the servo motor 40 is input as the abnormality cause.

[0107] The restoration content display field 570 is a field where the operator inputs the restoration work content for the target device where the abnormality occurred. In this example, belt cleaning is input as the restoration work content.

[0108] The acquisition unit 111 generates operation data by associating the information input in the event summary display field 540 to the recovery content display field 570 with the command signal 511 and the like displayed in the right column of the evaluation screen 500, and stores the generated operation data in the operation data storage unit 151. This allows the operator to store comments on the command signal 511 and the measurement signal 521 in the operation data storage unit 151 by associating them with the command signal 511 and the measurement signal 521.

[0109] As described above, according to this embodiment, when an update of the learning model is detected, the operation data whose anomaly degree has changed before and after the update of the learning model and the anomaly degree corresponding to that operation data are displayed on the display. This allows the qualitative evaluation result of the learning model to be displayed on the display, allowing the machine learning model to be appropriately evaluated when applied to production equipment.

[0110] The present disclosure can employ the following modifications.

[0111] (1) The evaluation device 10 may be implemented in the servo amplifier 30 or the motion controller 20, or may be implemented in a personal computer or a cloud server.

[0112] (2) The memory 15 may be implemented in the servo amplifier 30 or the motion controller 20, or may be implemented in a personal computer or a cloud server.

[0113] According to the present disclosure, a learning model can be appropriately evaluated, which is useful in the technical field of machine learning.

Claims

1. A method for evaluating a learning model used to determine whether at least one of a load and the servo motor is normal or abnormal in a controlled device including a load, a servo amplifier, a servo motor that powers the load in accordance with the control of the servo amplifier, and a motion controller that outputs a command signal to the servo amplifier, wherein the learning model outputs an abnormality degree for at least one of the load and the servo motor from operation data of the servo motor, and when a computer detects an update to the learning model, searches for operation data in which the abnormality degree has changed before and after the update of the learning model from among multiple operation data pre-registered in a memory, and displays the searched operation data and the abnormality degree corresponding to the operation data on a display.

2. The evaluation method according to claim 1, wherein the operational data displayed on the display includes a command signal for driving the servo motor and a measurement signal of the servo motor when the servo motor operates based on the command signal.

3. The evaluation method according to claim 1 or 2, wherein the display on the display includes superimposing a first degree of abnormality output when the searched motion data is input into the learning model before the update and a second degree of abnormality output when the searched motion data is input into the learning model after the update.

4. The evaluation method of claim 2, wherein the operational data displayed on the display includes at least one of the following: an operator's judgment of normality or abnormality, the measurement date and time of the searched operational data, a description by the operator of the cause of the abnormality, and a description of the work to restore the controlled device to normal operation.

5. The evaluation method according to claim 1 or 2, wherein the display on the display includes displaying the searched motion data and the degree of anomaly corresponding to the searched motion data so that the points where the degree of anomaly has changed before and after updating the learning model are highlighted.

6. The evaluation method according to claim 1 or 2, wherein the display on the display includes displaying a difference in the degree of anomaly before and after updating the learning model.

7. The evaluation method according to claim 1 or 2, wherein the servo motor includes a plurality of operating modes, and the display includes displaying the operating data and the degree of abnormality by distinguishing periods corresponding to the plurality of operating modes.

8. The evaluation method according to claim 1 or 2, wherein the computer is any one of the servo amplifier, the motion controller, a personal computer, and a cloud server.

9. The evaluation method according to claim 1 or 2, wherein the memory is one of the servo amplifier, the motion controller, a personal computer, and a cloud server.

10. The evaluation method according to claim 1 or 2, wherein the motion data is time-series data for a period from the start to the end of a certain motion pattern for the load.

11. The evaluation method according to claim 1 or 2, wherein the learning model is generated by unsupervised machine learning of operational data of the servo motor under normal conditions.

12. The evaluation method of claim 1 or 2, wherein the search for the motion data includes: executing, for each of the plurality of motion data, a process of calculating the difference between a first degree of abnormality output when the motion data is input into a learning model before the update and a second degree of abnormality output when the motion data is input into a learning model after the update; and ranking the plurality of motion data so that the larger the difference, the higher the ranking.

13. The evaluation method according to claim 12, wherein the display on the display includes switching the display of the operation data and the degree of abnormality in descending order of the rank in response to a switching instruction from an operator.

14. An evaluation device for evaluating a learning model used to determine whether at least one of a load and the servo motor is normal or abnormal in a controlled device including a load, a servo amplifier, a servo motor that powers the load in accordance with the control of the servo amplifier, and a motion controller that outputs a command signal to the servo amplifier, wherein the learning model outputs an abnormality degree for at least one of the load and the servo motor from operation data of the servo motor, and when a processor of the evaluation device detects an update to the learning model, the evaluation device executes a process of searching for operation data in which the abnormality degree has changed before and after the update of the learning model from among multiple operation data pre-registered in a memory, and displaying the searched operation data and the abnormality degree corresponding to the operation data on a display.

15. An evaluation program that causes a computer to execute an evaluation method for evaluating a learning model used to determine whether at least one of a load and the servo motor is normal or abnormal in a controlled device including a load, a servo amplifier, a servo motor that powers the load in accordance with the control of the servo amplifier, and a motion controller that outputs a command signal to the servo amplifier, wherein the learning model outputs an abnormality degree related to at least one of the load and the servo motor from operation data of the servo motor, and the evaluation program causes the computer to execute a process that, when an update of the learning model is detected, searches for operation data in which the abnormality degree has changed before and after the update of the learning model from among multiple operation data pre-registered in memory, and displays the searched operation data and the abnormality degree corresponding to the operation data on a display.

Images