Abnormal sign detection device, abnormal sign detection method, and computer program

Abstract

To detect a sign of an anomaly.SOLUTION: Time series data of a measured value of a target variable related to a situation of a manufacturing process of a product is obtained. A plurality of parameters included in a predictive model for predicting a value of the target variable is learned using at least a part of the time series data of the measurement value of the target variable. Using a learned model of the predictive model, a value of the target variable at a future point of time after the latest point of time of the time series data of the measurement value is predicted. If the predicted value of the target variable is outside a predetermined tolerance range, first data indicating that the predicted value of the target variable is outside the predetermined tolerance range is output.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present specification relates to a technology for detecting signs of abnormality in a product manufacturing process. [Background technology]

[0002] Conventionally, there have been processes for detecting abnormalities in the manufacturing process of products. For example, a technology has been proposed in which a normal model is created by a one-class support vector machine method using measurement data related to tools, such as measurement data from a vibration sensor that measures tool vibration, and then the measurement data measured during processing is diagnosed as normal or abnormal based on the normal model. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-24055 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, it is preferable to prevent abnormalities in the manufacturing process of a product before they occur. However, there is room for improvement in terms of detecting symptoms of abnormalities.

[0005] This specification discloses a technique for detecting a symptom of an abnormality. [Means for solving the problem]

[0006] The technology disclosed in this specification can be realized in the following application examples.

[0007] [Application Example 1] An abnormal sign detection device comprising: a data acquisition unit that acquires time series data of measurement values ​​of a target variable related to a status of a manufacturing process of a product; a first learning unit that learns a plurality of parameters included in a prediction model for predicting a value of the target variable using at least a part of the time series data of the measurement values ​​of the target variable; a first prediction unit that predicts a value of the target variable at a future time point after a latest time point of the time series data of the measurement values ​​using the learned model of the prediction model; and an output unit that outputs first data indicating that the predicted value of the target variable is outside the predetermined tolerance range when the predicted value of the target variable is outside the predetermined tolerance range.

[0008] According to this configuration, the first data indicating that the predicted value is outside the predetermined allowable range is output, and therefore the first data can be used as data indicating a symptom of an abnormality. [Application example 2] The abnormal sign detection device according to Application Example 1, The predictive model is configured to predict the value of the target variable using a plurality of components including a component that exhibits monotonic change over time. An abnormality detection device. [Application example 3] The abnormal sign detection device according to Application Example 1 or 2, further comprising: an anomaly score acquisition unit that acquires an anomaly score of partial time series data, which is data of a part of a time width of the time series data of the measurement values, by inputting the partial time series data into a calculation model configured to calculate an anomaly score indicating a difference between the input time series data and L pieces of reference time series data (L is an integer equal to or greater than 1); When the anomaly score is greater than a predetermined score threshold, the output unit outputs second data indicating that the anomaly score is greater than the predetermined score threshold. An abnormality detection device. [Application example 4] The abnormal sign detection device according to any one of Application Examples 1 to 3, The data acquisition unit further acquires p time series data of p measurement values ​​of p auxiliary variables (p is an integer equal to or greater than 1) that are measured in synchronization with the measurement of the target variable, the p auxiliary variables including a motion variable that indicates a motion of a worker; The abnormality symptom detection device further includes: a second learning unit that learns a plurality of parameters included in an estimation model for estimating p weights associated with the p auxiliary variables, using at least a portion of the time series data of the measurement values ​​of the target variable and at least a portion of the p time series data of the p measurement values ​​of the p auxiliary variables; an estimation unit that estimates the p weights using a trained model of the estimation model; Equipped with When the predicted value of the target variable is outside the predetermined allowable range, the output unit outputs, in addition to the first data, third data indicating one or both of m estimated values ​​(m is an integer not less than 1 and not more than p) of the p estimated values ​​of the p weights and m pieces of identification information identifying m auxiliary variables corresponding to the m estimated values. An abnormality detection device. [Application example 5] The abnormal sign detection device according to any one of Application Examples 1 to 4, The target variable indicates a position or orientation of a label sheet attached to the product relative to the product. An abnormality detection device. [Application Example 6] An abnormality symptom detection method, comprising: A data acquisition step of acquiring time series data of measured values ​​of a target variable related to the status of a manufacturing process of the product; a first learning step of learning a plurality of parameters included in a prediction model for predicting a value of the target variable using at least a part of the time series data of the measurement value of the target variable; A first prediction step of predicting a value of the target variable at a future time point after the latest time point of the time series data of the measurement value using a trained model of the prediction model; an output step of outputting, when the predicted value of the target variable is outside a predetermined tolerance range, first data indicating that the predicted value of the target variable is outside the predetermined tolerance range; An abnormality sign detection method comprising: [Application Example 7] A computer program comprising: A data acquisition function for acquiring time series data of measured values ​​of a target variable related to the status of the manufacturing process of the product; a first learning function that learns a plurality of parameters included in a prediction model for predicting a value of the target variable by using at least a portion of the time series data of the measurement value of the target variable; A first prediction function that predicts a value of the target variable at a future time point after the latest time point of the time series data of the measurement value by using a trained model of the prediction model; an output function for outputting, when the predicted value of the target variable is outside a predetermined tolerance range, first data indicating that the predicted value of the target variable is outside the predetermined tolerance range; A computer program that enables a computer to realize the above.

[0009] The technology disclosed in this specification can be realized in various forms, such as a method and device for detecting signs of abnormality, a computer program for realizing the functions of the method or device, a recording medium (e.g., a non-transitory recording medium) on which the computer program is recorded, and the like. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is an explanatory diagram illustrating a data processing device according to an embodiment; [Diagram 2] 13(A) and 13(B) show examples of images captured by the digital camera 110. [Diagram 3] 13 is a flowchart illustrating an example of a process for detecting a sign of abnormality. [Figure 4] 13 is a flowchart illustrating an example of a prediction process. [Diagram 5] 13(A)-(F) are explanatory diagrams of the state space model. [Figure 6] 13 is a flowchart illustrating an example of a filter process. [Figure 7] 13A-13C are graphs showing examples of time series data TSx, TSy, and TSa. [Figure 8] 13 is a flowchart illustrating an example of a score calculation process. [Figure 9] FIG. 13 is a schematic diagram of a score calculation model MBx. [Figure 10] FIG. 11 is an explanatory diagram illustrating a second embodiment of the data processing device. [Figure 11] 13(A) and 13(B) show examples of images captured by the second camera 120. [Figure 12] 13(A)-(E) are explanatory diagrams of the state space model. [Figure 13] 13A and 13B are a flowchart showing a second embodiment of the detection process. [Figure 14] Graphs (A) and (B) show examples of coefficients E1j and E2j. [Figure 15] FIG. 11 is a schematic diagram showing an example of third data. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] A. First Example: A1. Equipment configuration: FIG. 1 is an explanatory diagram showing a data processing device as an embodiment. In this embodiment, the data processing device 200 is, for example, a personal computer. The data processing device 200 executes various processes for inspecting an object (for example, a product such as a printer). The data processing device 200 includes a processor 210, a storage device 215, a display unit 240, an operation unit 250, and a communication interface 270. These elements are connected to each other via a bus. The storage device 215 includes a volatile storage device 220 and a non-volatile storage device 230.

[0012] The processor 210 is a device configured to perform data processing, and is, for example, a CPU. The volatile storage device 220 is, for example, a DRAM, and the non-volatile storage device 230 is, for example, a flash memory. The non-volatile storage device 230 stores a program 231, prediction models MAx, MAy, MAa, score calculation models MBx, MBy, MBa, and time series data TSx, TSy, TSa. The prediction models MAx, MAy, MAa are prediction models using a so-called Kalman filter, and are program modules in this embodiment. The score calculation models MBx, MBy, MBa are estimation models using a so-called k-nearest neighbor method, and are program modules in this embodiment. Details of the program 231, the prediction models MAx, MAy, MAa, the score calculation models MBx, MBy, MBa, and the time series data TSx, TSy, TSa will be described later.

[0013] The display unit 240 is a device configured to display images, such as a liquid crystal display or an organic EL display. The operation unit 250 is a device configured to receive operations by a user, such as a button, a lever, or a touch panel arranged overlaid on the display unit 240. A user can input various requests and instructions to the data processing device 200 by operating the operation unit 250. The communication interface 270 is an interface for communicating with other devices (for example, a USB interface, a wired LAN interface, or an IEEE802.11 wireless interface). The digital camera 110 is connected to the communication interface 270. The digital camera 110 captures an object 900 to be inspected, thereby generating image data of the captured image. Hereinafter, the object 900 is assumed to be a printer (the object 900 is also referred to as the printer 900).

[0014] A2.Photo: 2(A) and 2(B) show examples of images captured by the digital camera 110. The captured image IM is a rectangular image having two sides parallel to the D1 direction and two sides parallel to the D2 direction perpendicular to the D1 direction. The captured image IM is represented by the color values ​​(in this embodiment, R (red), G (green), and B (blue) component values) of a plurality of pixels arranged in a matrix along the D1 direction and the D2 direction. The captured image IM shows the outer surface 900s of the printer 900. In this embodiment, the shape of the edge of the outer surface 900s is approximately the same as a rectangle represented by four sides including a first side SD1 and a second side SD2. In addition, a label sheet 910 is attached to the outer surface 900s. In this embodiment, the shape of the label sheet 910 is approximately the same as a rectangle represented by four sides. The label sheet 910 is attached to the printer 900 by a machine or an operator. The process of attaching the label sheet 910 (called the attachment process) is part of the manufacturing process of the printer 900.

[0015] 2(A) and 2(B) show two axes Dx and Dy associated with the outer surface 900s. The first axis Dx is an axis indicated by a straight line indicating the first side SD1, and the second axis Dy is an axis indicated by a straight line indicating the second side SD2. In this embodiment, the second axis Dy is perpendicular to the first axis Dx. The origin O indicates the intersection of these axes Dx and Dy.

[0016] The axes Dx and Dy are used to determine the position and tilt of the label sheet 910 relative to the printer 900. Figure 2(A) shows the position P of the label sheet 910. The position P is expressed by a first coordinate Px on the first axis Dx and a second coordinate Py on the second axis Dy. Note that the position P of the label sheet 910 may be any of a variety of predetermined positions relative to the label sheet 910. In this embodiment, the position P is the position of the center of gravity of the image on the label sheet 910.

[0017] 2B shows an angle AG of the label sheet 910 relative to the printer 900. In this embodiment, the angle AG is the angle between side SDt of the label sheet 910 and the second axis Dy. Side SDt can be any of a variety of predetermined sides of the label sheet 910. In this embodiment, side SDt is the side closest to the second axis Dy and is approximately parallel to the second axis Dy.

[0018] In this embodiment, if the first coordinate Px is within a predetermined first coordinate tolerance range, the second coordinate Py is within a predetermined second coordinate tolerance range, and the angle AG is within a predetermined angle tolerance range, the inspection result of the printer 900 is pass. If one or more parameters of the first coordinate Px, the second coordinate Py, and the angle AG are outside the corresponding tolerance ranges, the inspection result of the printer 900 is fail. Each of the parameters Px, Py, and AG is an example of a target variable related to the status of the manufacturing process of the printer 900. Hereinafter, the parameters Px, Py, and AG are referred to as target variables Px, Py, and AG.

[0019] In this embodiment, in a production line for printers 900, label sheets 910 are affixed to multiple printers 900 one by one in sequence. Here, one or more of the parameters Px, Py, and AG may change little by little in the same direction. If such monotonic changes continue, the parameters may move outside of the allowable range. If a monotonic change in a parameter is detected, measures can be taken before the parameter moves outside of the allowable range (for example, the settings of a device that affixes label sheets 910 to printers 900 can be adjusted).

[0020] A3. Detection process: Fig. 3 is a flowchart showing an example of a process for detecting a sign of an abnormality. In this embodiment, when one or more of the parameters Px, Py, and AG are outside the corresponding allowable range, it is determined that the printer 900 has an abnormality. The processor 210 of the data processing device 200 (Fig. 1) detects a sign of an abnormality using a captured image of the printer 900. The processor 210 proceeds with the process of Fig. 3 in accordance with the program 231.

[0021] In S110, the processor 210 (FIG. 1) supplies a shooting instruction to the digital camera 110. The digital camera 110 takes a picture of the printer 900 and generates image data showing the captured image. The captured image shows the outer surface 900s of the printer 900, similar to the captured image IM of FIG. 2(A) and FIG. 2(B). The processor 210 acquires image data from the digital camera 110 and identifies parameters Px, Py, and AG using the image data. The method of identifying parameters Px, Py, and AG may be any method. For example, the processor 210 identifies a first axis Dx and a second axis Dy in the captured image by pattern matching using an image of the outer surface 900s prepared in advance. The processor 210 also identifies a position P and a side SDt in the captured image by pattern matching using an image of a label sheet 910 prepared in advance. The processor 210 calculates parameters Px, Py, and AG using the identified axes Dx, Dy, position P, and side SDt. The calculated values ​​of the parameters Px, Py, and AG are examples of measured values ​​of the target variables. The measured values ​​are also called observed values. The processor 210 may detect the outer surface 900s and the label sheet 910 in the captured image using a machine learning model called an object detection model (for example, YOLO (You only look once)).

[0022] As described later, the processor 210 executes the process of FIG. 3 for each of the multiple printers 900. The processor 210 executes the process of S110 multiple times to process the multiple printers 900. The processor 210 acquires the values ​​of the target variables Px, Py, and AG calculated by each of the repeated S110 as time series data. The processor 210 acquires a measurement value at a new time point t by executing S110 for a new printer 900. In this embodiment, the processor 210 uses the order of the detection process as the time point t. For example, the time point t is incremented by 1 each time a new printer 900 is processed. The time series data TSx, TSy, and TSa stored in the storage device 215 (non-volatile storage device 230 in this embodiment) in FIG. 1 are time series data of the three target variables Px, Py, and AG, respectively. In S110, the processor 210 adds data indicating the newly calculated values ​​of the target variables Px, Py, and AG to the time series data TSx, TSy, and TSa by storing the data in the non-volatile storage device 230. In this manner, the time series data TSx, TSy, and TSa are updated by S110 to include data at the new time point t.

[0023] In S120, the processor 210 predicts future values ​​of the target variables Px, Py, and AG, respectively, using the prediction models MAx, MAy, and MAa.

[0024] FIG. 4 is a flowchart showing an example of a prediction process of future values ​​using a prediction model. Each prediction model MAx, MAy, MAa (FIG. 1) is a prediction model that predicts values ​​using a state space model and a Kalman filter. In this embodiment, the three prediction models MAx, MAy, MAa correspond to three target variables Px, Py, AG, respectively. The prediction process in FIG. 4 shows a process for one target variable.

[0025] 5(A) to 5(F) are explanatory diagrams of state space models. In this embodiment, three target variables Px, Py, and AG are expressed using the state space models of FIG. 5(A) to FIG. 5(F).

[0026] FIG. 5(A) shows the components contained in the measured value. Yt is the measured value at time t, Tt is the trend component at time t, Rt is the autoregressive component at time t, and Wt is the noise at time t. The measured value Yt is expressed as the sum of Tt, Rt, and Wt. The noise Wt has a mean value of zero and a variance of s 2 It is assumed that the distribution follows a normal distribution.

[0027] FIG. 5B shows an example of a trend component Tt, where k is the order. The trend component Tt at time t is expressed as the sum of the total product of the coefficient Ck(i) and the trend component Tt-i at time ti, and the first noise Vat at time t (the subscript i ranges from 1 to k). The first noise Vat has a mean value of zero and a variance of sa 2 The graph specifically shows the coefficients Ck(i) for k=1, 2, and 3. As shown in the graph, Ck(i) is a predetermined constant. Such a trend component Tt represents a component that changes monotonically over time.

[0028] FIG. 5C shows an example of the autoregressive component Rt. The autoregressive component Rt at time t is expressed as the sum of the product of the coefficient φ(i) and the autoregressive component Rt-i at time ti, and the second noise Vbt at time t (the subscript i varies from 1 to q). The second noise Vbt has a mean value of zero and a variance of sb 2 It is assumed that the distribution follows a normal distribution.

[0029] Fig. 5(D) shows a system model. Xt is a state (specifically, a column vector) at time t, F is a first coefficient matrix, G is a second coefficient matrix, and Vt is noise (specifically, a column vector) at time t. The state Xt at time t is the sum of a column vector represented by the product of the state Xt-1 at time t-1 and the first coefficient matrix F, and a column vector represented by the product of the noise Vt at time t and the second coefficient matrix G.

[0030] 5E shows an observation model. H is a third coefficient matrix (specifically, a row vector). A measurement value Yt at time t is expressed as the sum of a value expressed as the product of the state Xt at time t and the third coefficient matrix H, and noise Wt.

[0031] Fig. 5(F) shows a concrete example of Xt, Vt, F, G, and H when k = 2 and q = 3. State Xt is a column vector having five elements Tt, Tt-1, Rt, Rt-1, and Rt-2 (the "T" at the top right of the row vector indicates transposition). Noise Vt is a column vector having two elements Vat and Vbt.

[0032] The first coefficient matrix F is a 5-row, 5-column matrix. The upper left 2-row, 2-column part Fa is used to calculate the trend component Tt (Figure 5(B)). The lower right 3-row, 3-column part Fb is used to calculate the autoregressive component Rt (Figure 5(C)). All other elements are zero (the 0 (zero) in the figure indicates a zero matrix).

[0033] The second coefficient matrix G is a 5-row, 2-column matrix. The upper 2-row, 2-column portion Ga is used to calculate the trend component Tt (Figure 5(B)). The lower 3-row, 2-column portion Gb is used to calculate the autoregressive component Rt (Figure 5(C)).

[0034] The third coefficient matrix H is a row vector with 1 row and 5 columns. The left two columns extract the trend component Tt from the state Xt, and the right three columns extract the autoregressive component Rt from the state Xt.

[0035] As described above, in this embodiment, the functions F and H are linear, and the noises Wt, Vat, and Vbt follow normal distribution. Such a state space model is also called a linear-Gaussian state space model. Hereinafter, k=2 and q=3 (FIG. 5(F)).

[0036] Next, the process of Fig. 4 will be described. As described above, Fig. 4 is a prediction process for one target variable. The processor 210 executes the process of Fig. 4 for each of the three target variables Px, Py, and AG. Below, the process of Fig. 4 will be described using the target variable Px and the prediction model MAx.

[0037] In S210, the processor 210 judges whether the adjustment condition is satisfied. The adjustment condition is a condition for adjusting the parameters of the state space model. The state space model of this embodiment (FIGS. 5(A)-5(F)) has a plurality of parameters (specifically, the variance s 2 , sa 2 , sb 2 and coefficient φ(i)). The prediction model MAx (FIG. 1) includes data indicating these parameters. In order to appropriately predict the target variable Px using the state space model, the processor 210 adjusts these parameters using the acquired time series data TSx.

[0038] The adjustment condition may be various conditions that can suppress inappropriate predictions caused by inappropriate parameters. In this embodiment, the adjustment condition is that the elapsed time from the previous adjustment exceeds a threshold value (e.g., one week, one month, etc.). That is, the adjustment of the parameters is performed periodically.

[0039] If the adjustment condition is not satisfied (S210: No), the processor 210 proceeds to S240.

[0040] If the adjustment condition is satisfied (S210: Yes), in S220, the processor 210 adjusts the parameters of the prediction model MAx (i.e., the parameters s 2 , sa 2 , sb 2 , φ(i)). In this embodiment, the processor 210 updates the data indicating the parameters included in the prediction model MAx to data indicating the adjusted parameters. Note that the prediction models MAx, MAy, and MAa include parameter data of the state space models of the corresponding target variables Px, Py, and AG.

[0041] The method of adjusting the parameters of the state space model may be any method of adjusting the parameters to values ​​that adequately explain the time series data. In this embodiment, the processor 210 performs so-called maximum likelihood estimation. In the maximum likelihood estimation, the parameters are determined so that the likelihood calculated from the parameters and the time series data is maximized. The likelihood indicates the plausibility of the parameters based on the time series data. The likelihood is high when the probability that the time series data is obtained based on the given parameters is high. As described above, in this embodiment, the state space model is a linear-Gaussian state space model. In this case, the processor 210 can perform maximum likelihood estimation of the parameters by various known algorithms. The algorithm may be, for example, an algorithm that calculates parameters such that the derivative of the log-likelihood function with respect to the parameters is zero. The entire acquired time series data may be used to adjust the parameters. Alternatively, a part of the acquired time series data may be used to adjust the parameters (for example, time series data of a predetermined time width including the latest time point t). Note that the parameters are adjusted based on the time series data. Therefore, the processor 210 uses the time series data to calculate the parameters of the prediction model MAx (here, the parameters s 2 , sa 2 , sb 2 , φ(i)) is said to be learned.

[0042] In S230, the processor 210 repeats the filter process from time 1 to time t-1 for the target variable Px. FIG. 6 is a flowchart showing an example of the filter process. In this embodiment, the filter process is a so-called Kalman filter. The process in FIG. 6 shows the filter process at time z. In this embodiment, the prediction models MAx, MAy, and MAa each include a program for the filter process of the target variables Px, Py, and AG. In addition, the prediction models MAx, MAy, and MAa each include data indicating a state Xz and a variance VSz of a prediction error, which will be described later, for the target variables Px, Py, and AG. The processor 210 executes the process in FIG. 6 according to the prediction model.

[0043] In S310, the processor 210 predicts the state Xz at time point z. This prediction is performed according to the system model (FIG. 5(D)). In this embodiment, the state Xz is calculated by multiplying the state Xz-1 at the previous time point z-1 by the first coefficient matrix F (since the average values ​​of the noises Vat and Vbt are zero, the addition of the noise Vt is omitted). Note that when the time point z is 1, the initial state X0 is used to estimate the state Xz. Each element of the initial state X0 may be set to various values. For example, each element may be set to zero or a random value.

[0044] In S320, the processor 210 obtains an actual measurement value Yz of the target variable Px at time z. Data indicative of the measurement value Yz is stored in the non-volatile storage device 230 in S110 (FIG. 3). The processor 210 obtains the data indicative of the measurement value Yz from the non-volatile storage device 230.

[0045] In S330, the processor 210 calculates the variance VSz of the prediction error of the state at time point z. This variance VSz is calculated in accordance with the system model (FIG. 5(D)). The variance VSz is expressed as the sum of the variance VSz-1 at the immediately previous time point z-1 and the variance of the process error. The variance of the process error indicates the variance of the noise component included in the system model. When the system model includes multiple noise components, the sum of multiple variances of the multiple noise components may be used as the variance of the process error. In this embodiment, the system model includes two noise components Vat and Vbt. The variance VSz is expressed as the sum of the variance VSz-1 at the immediately previous time point z-1 and the variance sa of the first noise Vat. 2 and the variance sb of the second noise Vbt 2 The initial variance VS0 may be set to various values.

[0046] In S340, the processor 210 predicts the measurement value at the time z. This prediction is performed according to the observation model (FIG. 5(E)). In this embodiment, the predicted value YPz is calculated by multiplying the state Xz predicted in S310 by the third coefficient matrix H (the addition of the noise Wt is omitted because the average value of the noise Wt is zero).

[0047] In S350, the processor 210 calculates the variance VMz of the prediction error of the measurement value at the time point z. This variance VMz is calculated according to the observation model (FIG. 5(E)). In this embodiment, the variance VMz is calculated by multiplying the variance VSz calculated in S330 by the variance s 2 It is the sum of and.

[0048] In S360, the processor 210 calculates the Kalman gain Kz at the time point z. In this embodiment, the Kalman gain Kz is calculated by VSz / VMz.

[0049] In S370, the processor 210 calculates a prediction residual dYz at the time point z. The prediction residual dYz is the difference obtained by subtracting the predicted value YPz from the measured value Yz.

[0050] In S380, the processor 210 corrects the state Xz at the time point z by using the Kalman gain Kz. In this embodiment, the corrected state Xz is the sum of the state Xz before correction and the product of the Kalman gain Kz and the prediction residual dYz. The processor 210 updates the data indicating the state Xz included in the prediction model MAx to data indicating the corrected state Xz.

[0051] In S390, the processor 210 corrects the variance VSz of the prediction error of the state at the time point z. In this embodiment, the corrected variance VSz is the product of "1-Kz" and the variance VSz before correction. The processor 210 updates the data indicating the variance VSz included in the prediction model MAx to data indicating the corrected variance VSz.

[0052] With the above, the processor 210 ends the filtering process at time point z, that is, the process of Fig. 6. The corrected state Xz and variance VSz are used in S310 and S330 of the next filtering process at time point z+1.

[0053] In S230 (FIG. 4), the processor 210 repeats the filter process of the target variable Px from time 1 to time t-1. As a result, the processor 210 acquires time series data of the corrected state X and time series data of the corrected variance VS for the target variable Px. The time series data of the state X indicates the state Xj at each time j from 1 to t-1. The time series data of the variance VS indicates the variance VSj at each time j from 1 to t-1. As described in S380 and S390 of FIG. 6, the processor 210 stores the data indicating the corrected state Xj and the data indicating the corrected variance VSj in the non-volatile storage device 230 as a part of the prediction model MAx. As a result, the prediction model MAx includes the time series data of the state Xj of the target variable Px and the time series data of the variance VSj (j is from 1 to t-1). After S230, the processor 210 proceeds to S240.

[0054] In S240, the processor 210 obtains an actual measurement value Yt of the target variable Px at the current time t. In this embodiment, data indicative of the measurement value Yt is stored in the non-volatile storage device 230 in S110 (FIG. 3). The processor 210 obtains the data indicative of the measurement value Yt from the non-volatile storage device 230.

[0055] In S250, the processor 210 executes a filter process (FIG. 6) of the target variable Px at the time t in accordance with the prediction model MAx. As a result, the state Xt and the variance VSt at the time t are corrected.

[0056] Next, the processor 210 predicts the measurement value N periods ahead (i.e., time point t+N) (hereinafter, N is also referred to as predicted time N). The predicted time N is an integer equal to or greater than 1 (for example, N=30).

[0057] In S260, the processor 210 initializes the counter h to 1. In S270, the processor 210 predicts the state Xt+h at time t+h. The prediction model MAx includes a program for the prediction in S270. The processor 210 executes the prediction in S270 according to the prediction model MAx. This prediction is the same as the prediction in S310 of FIG. 6. The processor 210 calculates the state Xt+h by multiplying the predicted state Xt+h-1 at the previous time t+h-1 by the first coefficient matrix F.

[0058] In S280, the processor 210 determines whether the counter h is equal to the prediction time N. If the counter h is different from the prediction time N, that is, if h < N (S280: No), the processor 210 adds 1 to the counter h in S285 and transfers to S270. Then, the processor 210 predicts the state Xt+h at the new time t+h.

[0059] If the counter h is the same as the prediction time N (S280: Yes), in S290, the processor 210 predicts the measured value at time t+N using the predicted state Xt+N at time t+N. The prediction model MAx includes a program for the prediction in S290. The processor 210 executes the prediction in S290 according to the prediction model MAx. This prediction is the same as the prediction in S340 of FIG. 6. The processor 210 calculates the predicted value YPt+N by multiplying the predicted state Xt+N at time t+N by the third coefficient matrix H. Then, the processor 210 ends the process of FIG. 4, that is, the process of S120 of FIG. 3.

[0060] The prediction process of the first coordinate Px has been described above. In S120, the processor 210 also executes the prediction process of FIG. 4 for other target variables Py and AG.

[0061] 7(A)-7(C) are graphs showing examples of time series data TSx, TSy, and TSa of the target variables Px, Py, and AG, respectively. The horizontal axis indicates time Tm. The vertical axis of FIG. 7(A)-7(C) indicates the target variables Px, Py, and AG, respectively. The range R1 of Px1 or more and Px2 or less in FIG. 7(A) indicates the first coordinate allowable range. The range R2 of Py1 or more and Py2 or less in FIG. 7(B) indicates the second coordinate allowable range. The range R3 of AG1 or more and AG2 or less in FIG. 7(C) indicates the angle allowable range. At the time Tm before the time t, the measured values ​​of the target variables Px, Py, and AG are shown. At the time Tm after the time t, the predicted values ​​of the target variables Px, Py, and AG are shown.

[0062] As described in FIG. 5(A) and FIG. 5(B), in the state space model of this embodiment, the measured value Yt includes a trend component Tt. Therefore, when the time series data of the actual measured value Yt includes a trend component that changes monotonically over time, the predicted value based on the state space model can reflect the trend component. For example, in the example of FIG. 7(A) to FIG. 7(C), in the time range TR close to the time point t, the actual measured value changes monotonically over time. As a result, in the time range from the time point t to the time point t+N, the predicted value changes monotonically over time. And, at the time point t+N, the predicted value may be outside the allowable ranges R1, R2, and R3.

[0063] In S130 of FIG. 3, the processor 210 judges whether or not the condition that "all three predicted values ​​of the three target variables Px, Py, and AG at the time t+N are within the corresponding allowable range" is satisfied. If the predicted values ​​of one or more target variables are outside the corresponding allowable range (S130: No), in S140, the processor 210 outputs first data indicating that the predicted values ​​are outside the allowable range. In this embodiment, the processor 210 outputs image data of a first image DP1 indicating that the predicted values ​​are outside the allowable range to the display unit 240 (FIG. 1). The display unit 240 displays the first image DP1 according to the received image data. The operator can know the signs of abnormality by observing the first image DP1. Then, the processor 210 proceeds to S180.

[0064] If the condition of S130 is satisfied (S130: Yes), in S150, the processor 210 calculates the anomaly scores of the target variables Px, Py, and AG, respectively, using the score calculation models MBx, MBy, and MBa.

[0065] FIG. 8 is a flowchart showing an example of a score calculation process using a calculation model. Each score calculation model MBx, MBy, MBa (FIG. 1) is a prediction model that predicts the score of time-series data using the so-called k-nearest neighbor method. In this embodiment, the three score calculation models MBx, MBy, MBa correspond to three target variables Px, Py, AG, respectively. The calculation process in FIG. 8 shows a process for one target variable. The processor 210 executes the calculation process in FIG. 8 for each of the three target variables Px, Py, AG. The process in FIG. 8 will be described below using the target variable Px and the score calculation model MBx.

[0066] In S410, the processor 210 extracts L pieces of reference partial time series data from the time series data of the target variable (L is a predetermined integer equal to or greater than 1, for example, L=1). The reference partial time series data is time series data of a predetermined time width WD (for example, WD=30), and indicates changes in the target variable within an allowable range. The time series data of the time width WD is composed of data at consecutive points in time WD+1.

[0067] FIG. 9 is a schematic diagram of the score calculation model MBx. In the figure, a graph showing the time series data TSx of the first coordinate Px is shown. The horizontal axis indicates time Tm, and the vertical axis indicates the first coordinate Px. In the figure, L pieces of reference partial time series data TD1-TDL are shown. The processor 210 randomly extracts L pieces of reference partial time series data from the range of time Tm in which the first coordinate Px changes within the allowable range R1 of the time series data TSx, for example. Alternatively, the operator may select L pieces of reference partial time series data. In addition, when L>2, the ranges of time Tm of the multiple reference partial time series data may overlap each other partially.

[0068] The processor 210 stores the L pieces of reference partial time series data as a part of the score calculation model MBx in the non-volatile storage device 230. The L pieces of reference partial time series data are adjusted based on the time series data. Therefore, it can be said that the processor 210 learns the parameters of the score calculation model MBx (here, the L pieces of reference partial time series data) using the time series data.

[0069] S410 (FIG. 8) may be performed only once for the detection process (FIG. 3) of multiple printers 900. For example, multiple learning printers 900 may be manufactured to obtain L pieces of reference partial time series data. In S410, the processor 210 may extract L pieces of reference partial time series data from the time series data of the target variable obtained by manufacturing multiple learning printers 900. The processor 210 may continue to use the same L pieces of reference partial time series data in the subsequent detection process of multiple printers 900. Alternatively, the processor 210 may execute S410 multiple times (e.g., periodically) for the detection process of multiple printers 900. In this case, it is preferable that the processor 210 extracts L pieces of reference partial time series data from the latest time series data.

[0070] In S420, the processor 210 acquires target partial time series data. The target partial time series data is time series data to be compared with L pieces of reference partial time series data to calculate the anomaly score. In this embodiment, the processor 210 acquires time series data of a time width WD including the current time point t from among the time series data TSx of the target variable Px as the target partial time series data. In the example of FIG. 9, the partial time series data TDt of the time width WD ending at the time point t is the target partial time series data.

[0071] In S430, the processor 210 calculates L distances between the target partial time series data and L reference partial time series data. In the example of FIG. 9, L distances DT1-DTL corresponding to L reference partial time series data TD1-TDL are calculated. The score calculation model MBx includes a program for the calculation of S430 (FIG. 8). The processor 210 executes the calculation of S430 according to the score calculation model MBx. The calculated distance may be any index value indicating the difference between two time series data. Such a distance indicates the dissimilarity between two partial time series data. Hereinafter, the distance is also referred to as dissimilarity. In this embodiment, the distance is a Euclidean distance. Here, one partial time series data is used as a vector of "WD+1" dimension.

[0072] In S440, the processor 210 selects the smallest M dissimilarities from among the L dissimilarities (M is a predetermined integer between 1 and L, for example, M=10). Then, the processor 210 calculates a representative value of the M dissimilarities as an anomaly score, and ends the process of FIG. 8. In the example of FIG. 9, an anomaly score SR is calculated. The score calculation model MBx includes a program for the calculation of S440. The processor 210 executes the calculation of S440 according to the score calculation model MBx. The calculated representative value may be various values ​​calculated according to a predetermined calculation formula using the M dissimilarities. In this embodiment, the representative value is an average value. Note that the representative value is not limited to an average value, and may be various values ​​determined based on the M dissimilarities, such as a minimum value, a maximum value, a median value, or an integrated value.

[0073] The anomaly score may be large when there is a high possibility that an anomaly will occur in the future. For example, in the pasting process, when the accuracy of the first coordinate Px of the label sheet 910 decreases, the variance of the target variable Px indicated by the target partial time series data increases. In such a case, the anomaly score of the target variable Px becomes large. Also, in the pasting process, when the first coordinate Px of the label sheet 910 changes monotonically, the anomaly score of the target variable Px becomes large. In general, the anomaly score becomes large when the way in which the target variable indicated by the target partial time series data changes is different from the way in which the target variable indicated by the reference partial time series data changes. A large anomaly score indicates that there is a high possibility that an anomaly will occur in the future. The same applies to the other target variables Py and AG.

[0074] The calculation process of the abnormality degree score of the first coordinate Px has been described above. In S150 of Fig. 3, the processor 210 executes the calculation process of Fig. 8 for the other target variables Py and AG.

[0075] In S160, the processor 210 judges whether or not the condition that "all three anomaly scores of the three target variables Px, Py, and AG are equal to or less than the corresponding score threshold" is satisfied. The anomaly score being greater than the score threshold indicates a sign of anomaly. The score threshold is experimentally determined in advance for each of the target variables Px, Py, and AG. If the anomaly score of one or more target variables is greater than the corresponding score threshold (S150: No), in S170, the processor 210 outputs second data indicating that the anomaly score is greater than the score threshold. In this embodiment, the processor 210 outputs image data of the second image DP2 indicating that the anomaly score is greater than the score threshold to the display unit 240 (FIG. 1). The display unit 240 displays the second image DP2 according to the received image data. The operator can know the sign of anomaly by observing the second image DP2. Then, the processor 210 proceeds to S180.

[0076] If the condition of S160 is satisfied (S160: Yes), the processor 210 proceeds to S180.

[0077] In S180, the processor 210 determines whether or not all printers 900 have been processed. If an unprocessed printer 900 remains (S180: No), the processor 210 proceeds to S110 and executes processing of a new printer 900. If all printers 900 have been processed (S180: Yes), the processor 210 ends the processing of FIG.

[0078] As described above, in this embodiment, the processor 210 of the data processing device 200 (FIG. 1) acquires the time series data TSx, TSy, and TSa of the measurement values ​​of the target variables Px, Py, and AG in S110 (FIG. 3). In S220 (FIG. 4), the processor 210 acquires the time series data TSx, TSy, and TSa of the measurement values ​​of the target variables Px, Py, and AG. 2 , sa 2 , sb 2, φ(i))) is learned using at least a part of the time series data TSx, TSy, TSa of the measurement values ​​of the target variables Px, Py, AG. In S250-S290 in FIG. 4, the processor 210 predicts the values ​​of the target variables Px, Py, AG at a future time point t+N after the latest time point t of the time series data TSx, TSy, TSa of the measurement values ​​using the learned model of the prediction model MAx, MAy, MAa. Then, when the predicted values ​​of the target variables Px, Py, AG are outside the predetermined allowable ranges R1, R2, R3 (FIG. 3: S130: No), in S140, the processor 210 outputs first data indicating that the predicted values ​​of the target variables Px, Py, AG are outside the predetermined allowable ranges R1, R2, R3. Such first data can be used as data indicating a sign of an abnormality. As described above, in this embodiment, the first data is image data of the first image DP1. The processor 210 outputs this first data to the display unit 240. The worker can know the signs of abnormality by observing the first image DP1. Then, the worker can review the settings of the attachment process before an abnormality occurs. Note that the data processing device 200 is an example of an abnormal sign detection device that detects signs of abnormality.

[0079] As described in Figures 5(A) and 5(B), the prediction models MAx, MAy, and MAa are configured to predict the values ​​of the target variables Px, Py, and AG using multiple components Tt and Rt, including a trend component Tt that indicates a monotonous change over time. Therefore, as described in Figures 7(A) to 7(C), the prediction models MAx, MAy, and MAa can appropriately predict the values ​​of the target variables Px, Py, and AG that may be outside the predetermined allowable ranges R1, R2, and R3 at a future time point when the target variables Px, Py, and AG change monotonically.

[0080] Also, in S150 of FIG. 3, specifically, in S430 and S440 of FIG. 8, the processor 210 inputs the target partial time series data of the corresponding target variables Px, Py, and AG to the score calculation models MBx, MBy, and MBa, thereby acquiring an anomaly score of the target partial time series data. As described in S410, S430, and S440, the score calculation models MBx, MBy, and MBa are configured to calculate an anomaly score indicating a difference between the input time series data and L pieces of reference time series data (L is an integer of 1 or more). Also, as shown in FIG. 9, the target partial time series data TDt input to the score calculation model (for example, the score calculation model MBx) in S430 is partial time series data that is a part of the time series data TSx of the measurement value having a time width WD. Then, in S440 (FIG. 8), the processor 210 acquires an anomaly score SR of the target partial time series data TDt (FIG. 9). Then, if the abnormality score is greater than the predetermined score threshold (FIG. 3: S160: No), in S170, the processor 210 outputs second data indicating that the abnormality score is greater than the predetermined score threshold. Such second data can be used as data indicating a sign of abnormality. As described above, in this embodiment, the second data is image data of the second image DP2. The processor 210 outputs this second data to the display unit 240. The worker can know the sign of abnormality by observing the second image DP2. Then, the worker can review the settings of the attachment process before an abnormality occurs.

[0081] 2(A), the object variables Px and Py indicate the position of the label sheet 910 affixed to the printer 900 relative to the printer 900. Therefore, the first data (FIG. 3: S140) output based on the predicted values ​​of the object variables Px and Py can be used as data that indicates an indication of an abnormality in the position of the label sheet 910. Furthermore, as shown in FIG. 2(B), the object variable AG indicates the orientation of the label sheet 910 affixed to the printer 900 relative to the printer 900. Therefore, the first data (FIG. 3: S140) output based on the predicted value of the object variable AG can be used as data that indicates an indication of an abnormality in the orientation of the label sheet 910.

[0082] B. Second Example: FIG. 10 is an explanatory diagram showing a second embodiment of the data processing device. There are four differences between the second embodiment and the first embodiment of FIG. 1. The first difference is that the program 231 is replaced with a program 231a. The second difference is that the non-volatile storage device 230 further stores the estimation models MCx, MCy, and MCa, the learned machine learning model MD, and the time series data TS1 and TS2. The estimation models MCx, MCy, and MCa and the machine learning model MD are all program modules. The third difference is that the digital camera 120 that photographs the worker 800 is connected to the communication interface 270. Hereinafter, the digital camera 110 that photographs the printer 900 is referred to as the first camera 110, and the digital camera 120 that photographs the worker 800 is referred to as the second camera 120. In this embodiment, the worker 800 performs the task of attaching a label sheet 910 to the printer 900. The fourth difference is that sensor 130, which measures parameter AP related to the operation of supply device 700, is connected to communications interface 270. In this embodiment, supply device 700 supplies label sheet 910 to worker 800. Supply device 700 includes a roller (not shown) that moves label sheet 910, and a motor 710 that rotates the roller. Sensor 130 measures the current of motor 710 (parameter AP is also referred to as current AP). The other parts of the hardware configuration of data processing device 200a are the same as the corresponding parts of the hardware configuration of data processing device 200 in FIG. 1. Below, a description of the parts in common with the first embodiment will be omitted.

[0083] 11A shows an example of an image captured by the second camera 120. The captured image IMa is a rectangular image having two sides parallel to the D1a direction and two sides parallel to the D2a direction perpendicular to the D1a direction (in this embodiment, the D1a direction is the horizontal direction on the captured image IMa, and the D2a direction is the vertical direction on the captured image IMa). The captured image IMa is represented by the color values ​​(in this embodiment, R, G, and B component values) of a plurality of pixels arranged in a matrix along the D1a and D2a directions.

[0084] The captured image IMa shows the printer 900 and the worker 800 viewed from above. The captured image IMa shows the worker 800 attaching a label sheet 910 to the outer surface 900s of the printer 900 with his or her hands. Specifically, the captured image IMa is an image of the label sheet 910 supported by the hand of the worker 800 as it comes into contact with the printer 900.

[0085] As described later, the image captured by the second camera 120 is processed by a machine learning model MD (FIG. 1). In this embodiment, the machine learning model MD (FIG. 1) is a human posture estimation model, and detects human joints (also called key points, for example, shoulders, elbows, wrists, etc.) from an image. In the example of FIG. 11(A), the machine learning model MD detects shoulders 810, elbows 820, and wrists 830. As the posture estimation model, various known models can be adopted. For example, a model selected from "OpenPose", "PoseNet", and "Cascaded Pyramid Network for Multi-Person Pose Estimation" may be used.

[0086] The processor 210 uses the detected joints to identify the angle of the arm relative to the printer 900. Here, in the captured image IMa, the outer surface 900s of the printer 900 is assumed to be parallel to the D1a direction. The first angle Aa1 is the angle between a line La1 parallel to the outer surface 900s and the upper arm 811. The upper arm 811 is represented by a straight line connecting the shoulder 810 and the elbow 820. The second angle Aa2 is the angle between a line La2 parallel to the outer surface 900s and the forearm 821. The forearm 821 is represented by a straight line connecting the elbow 820 and the wrist 830. The third angle Aa3 is the angle between the outer surface 900s and a straight line 9La approximating the label sheet 910 supported by the hand of the worker 800.

[0087] FIG. 11B shows another example of an image captured by the second camera 120. Unlike when capturing the captured image IMa in FIG. 11A, when capturing the captured image IMb in FIG. 11B, the second camera 120 captures the printer 900 and the worker 800 from the side. The D1a direction in the figure indicates the horizontal direction on the captured image IMb, similar to the D1a direction in FIG. 11A, and the D2a direction indicates the vertical direction on the captured image IMb, similar to the D2a direction in FIG. 11A. The captured image IMb is also an image when the label sheet 910 supported by the hand of the worker 800 contacts the printer 900, similar to the captured image IMa. The machine learning model MD detects the shoulder 810, elbow 820, and wrist 830 from the captured image IMb, similar to the case where the captured image IMa (FIG. 11A) is processed. Here, it is assumed that the outer surface 900s of the printer 900 in the captured image IMb is parallel to the D2a direction. The fourth angle Aa4 is the angle between a line La4 parallel to the outer surface 900s and the upper arm 811. The fifth angle Aa5 is the angle between a line La5 parallel to the outer surface 900s and the forearm 821. The sixth angle Aa6 is the angle between the outer surface 900s and a straight line 9Lb that approximates the label sheet 910 held by the hand of the worker 800.

[0088] The worker 800 repeats the same task of attaching the label sheet 910 to the printer 900. The feeder 700 repeats the same process of feeding the label sheet 910 to the worker 800. When appropriate object variables Px, Py, and Ag (FIGS. 2A and 2B) are maintained, the above angles Aa1-Aa6 and the current AP (FIG. 1) are maintained at approximately constant values. Changes in these parameters Aa1-Aa6 and AP can change any of the object variables Px, Py, and Ag. For example, a change in any of the angles Aa1-Aa6 can change the position or orientation of the label sheet 910 relative to the printer 900. As a result, any of the object variables Px, Py, and Ag can change. A change in the current AP can change the position or orientation of the label sheet 910 relative to the worker 800 when the worker 800 receives the label sheet 910 from the feeder 700. As a result, any of the target variables Px, Py, or Ag may change.

[0089] If the degree of contribution of each of the parameters Aa1-Aa6 and AP to the change of the target variables Px, Py, and Ag can be identified, the cause of the change of the target variables Px, Py, and Ag can be identified. The estimation models MCx, MCy, and MCa (FIG. 10) are configured to estimate the degree of contribution of each of the multiple parameters. In this embodiment, the estimation models MCx, MCy, and MCa are prediction models that predict the target variables Px, Py, and AG using a state space model and a Kalman filter, similar to the prediction models MAx, MAy, and MAa. The three estimation models MCx, MCy, and MCa correspond to the three target variables Px, Py, and AG, respectively. Note that in this embodiment, the estimation models MCx, MCy, and MCa are used to estimate the degree of contribution of each of the multiple parameters, rather than predicting the future values ​​of the target variables Px, Py, and AG.

[0090] 12(A) to 12(E) are explanatory diagrams of state space models for estimation models MCx, MCy, and MCa. In this embodiment, three target variables Px, Py, and AG are expressed using the state space models of FIG. 12(A) to 12(E).

[0091] Figure 12(A) shows the components contained in the measured values. The only difference from the measured value Yt in Figure 5(A) is that the sum of the products of the coefficient Eit and the auxiliary variable Uit is added (the subscript i varies from 1 to p). The trend component Tt, the autoregressive component Rt, and the noise Wt are the same as those in Figures 5(A) to 5(C).

[0092] The auxiliary variable Uit is the measurement value of the i-th variable at time t. As the p auxiliary variables Uit, p parameters pre-selected from the parameters Aa1-Aa6 and AP (FIGS. 10, 11(A), and 11(B)) are used. The coefficient Eit is a coefficient by which the i-th auxiliary variable Uit at time t is multiplied. The p coefficients Eit may change over time. As described later, in this embodiment, the p coefficients Eit are included in multiple elements of the state Xt.

[0093] FIG. 12B shows a system model of the i-th coefficient Eit. The coefficient Eit at time t is the sum of the coefficient Eit-1 at the previous time t-1 and the i-th noise Vit at time t. The noise Vit has a mean value of zero and a variance of si 2 It is assumed that the distribution follows a normal distribution.

[0094] Fig. 12(C) shows a system model. This system model is expressed in the same format as the system model in Fig. 5(D). That is, a state Xt at a time t is the sum of a column vector represented by the product of a state Xt-1 at a time t-1 and a first coefficient matrix F, and a column vector represented by the product of a noise Vt at a time t and a second coefficient matrix G.

[0095] Fig. 12(D) shows an observation model. This observation model is expressed in the same format as the observation model in Fig. 5(E). That is, a measurement value Yt at time t is expressed as the sum of a value expressed as the product of a state Xt at time t and a third coefficient matrix Ht, and noise Wt.

[0096] Figure 12(E) shows specific examples of Xt, Vt, F, G, and Ht when k = 2, q = 3, and p = 2. State Xt is obtained by adding two elements E1t and E2t (Figure 12(B)) to state Xt in Figure 5(F). Specifically, state Xt is a column vector having seven elements Tt, Tt-1, Rt, Rt-1, Rt-2, E1t, and E2t.

[0097] The noise Vt is obtained by adding two elements V1t and V2t (FIG. 12(B)) to the noise Vt in FIG. 5(F). Specifically, the noise Vt is a column vector having four elements Vat, Vbt, V1t, and V2t.

[0098] The first coefficient matrix F is obtained by expanding the first coefficient matrix F in FIG. 5(F) so as to include a portion Fc used in the calculation of elements E1t and E2t (FIG. 12(B)). Specifically, the first coefficient matrix F is a matrix with 7 rows and 7 columns. The portion Fa in the upper left corner with 2 rows and 2 columns and the portion Fb in the center with 3 rows and 3 columns are the same as the portions Fa and Fb in FIG. 5(F), respectively. The portion Fc is added to the lower right of the portion Fb. All elements of the other portions are zero.

[0099] The second coefficient matrix G is an extension of the second coefficient matrix G in FIG. 5(F) to include a portion Gcx used in the calculation of elements E1t and E2t (FIG. 12(B)). Specifically, the second coefficient matrix G is a matrix with 7 rows and 4 columns. The upper 2-row, 4-column portion Gax is used in the calculation of the trend component Tt (FIG. 5(B)). The central 3-row, 4-column portion Gbx is used in the calculation of the autoregressive component Rt (FIG. 5(C)). The lower 2-row, 4-column portion Gcx is used in the calculation of elements E1t and E2t (FIG. 12(B)).

[0100] The third coefficient matrix Ht is obtained by adding two elements U1t and U2t (FIG. 12(A)) to the third coefficient matrix H in FIG. 5(F). Specifically, the third coefficient matrix Ht is a row vector with 1 row and 7 columns. The left two columns extract the trend component Tt from the state Xt. The center three columns extract the autoregressive component Rt from the state Xt. The right two columns extract the sum of the products of the coefficient Eit and the auxiliary variable Uit from the state Xt.

[0101] As described above, the auxiliary variable Uit may change when the time t changes. Therefore, the third coefficient matrix Ht may also change when the time t changes. Furthermore, since the auxiliary variable Uit is a measured value, at the current or past time t, the auxiliary variable Uit is a fixed value that may vary depending on the time t. In this way, at the current or past time t, the third coefficient matrix Ht includes fixed elements that may vary depending on the time t.

[0102] As described above, in this state space model, the functions F and Ht are linear, and the noises Wt, Vat, Vbt, and Vit follow normal distributions. In other words, this state space model is also a linear-Gaussian state space model.

[0103] In the following, in this embodiment, k=2, q=3, and p=2 (FIG. 12(E)). The first auxiliary variable U1t is the current AP (FIG. 10), and the second auxiliary variable U2t is the first angle Aa1 (FIG. 11(A)). The second camera 120 generates data of a captured image of the printer 900 and the worker 800 viewed from above, such as the captured image IMa.

[0104] 13(A) and 13(B) are flow charts showing a second embodiment of the detection process. The only difference from the second embodiment of FIG. 3 is that S105 and S142-S147 are added. The other parts of the detection process are the same as the corresponding parts of FIG. 3 (illustrations and descriptions of the same parts are omitted). Processor 210 proceeds with the detection process according to program 231a.

[0105] S105 (FIG. 13(A)) is added before S110 in FIG. 3. In S105, the processor 210 (FIG. 1) obtains measurement data for each of p auxiliary variables (p=2 in this embodiment) using the measurement data generated by the sensor 130 and the image data generated by the second camera 120.

[0106] In this embodiment, the first auxiliary variable U1t is a measurement value of the current AP. In S105, the processor 210 acquires measurement data from the sensor 130. Then, the processor 210 adopts the value indicated by the measurement data as the measurement value of the current AP (i.e., the first auxiliary variable U1t) at the time point t.

[0107] In this embodiment, the second auxiliary variable U2t is the first angle Aa1 (FIG. 11(A)). The processor 210 (FIG. 1) uses an image captured by the second camera 120 to obtain a measurement value of the first angle Aa1 at the time t.

[0108] In this embodiment, the second camera 120 (FIG. 1) continues to capture images of the worker 800 and the printer 900 from above, and supplies the captured video data to the data processing device 200a. The video data shows a plurality of frame images arranged in chronological order. A frame image is a type of captured image. In S105, the processor 210 identifies the frame image when the label sheet 910 contacts the printer 900 by analyzing the video data. For example, the processor 210 detects the printer 900 and the label sheet 910 in the frame image by pattern matching using images of the printer 900 and the label sheet 910 that have been prepared in advance. The data processing device 200 then selects the frame image when the label sheet 910 contacts the printer 900 as the target image, which is the image to be processed. The processor 210 may detect the printer 900 and the label sheet 910 using an object detection model.

[0109] The processor 210 inputs image data of the target image into the machine learning model MD (FIG. 10) to detect the shoulder 810, elbow 820, and wrist 830 of the worker 800. The processor 210 uses the detected shoulder 810 and elbow 820 to calculate the first angle Aa1.

[0110] As in the first embodiment, the processor 210 executes the process of S105 multiple times to process multiple printers 900. The processor 210 acquires the values ​​of the current AP and the first angle Aa1 calculated by each of the repeated S105 as time series data. The processor 210 acquires the measurement value at a new time point t by executing S105 for a new printer 900. The time series data TS1 and TS2 stored in the storage device 215 (non-volatile storage device 230 in this embodiment) in FIG. 10 are time series data of two parameters AP and Aa1, respectively. In S105, the processor 210 adds data indicating the newly calculated values ​​of the parameters AP and Aa1 to the time series data TS1 and TS2 by storing them in the non-volatile storage device 230. In this way, the time series data TS1 and TS2 are updated by S105 to include data at the new time point t.

[0111] In this embodiment, the processor 210 executes the process of S105 at the same timing as the pasting step. Alternatively, the processor 210 may execute the process of S105 after the pasting step. In this case, the processor 210 stores the data from the second camera 120 and the sensor 130 at the time of performing the pasting step in the storage device 215 (for example, the non-volatile storage device 230). Then, in S105, the processor 210 uses the data stored in the storage device 215 to acquire the measurement value data of each of the p auxiliary variables.

[0112] Next, the process of Fig. 13(B) will be described. If the determination result of S130 (Fig. 3) is No, processor 210 executes the processes of S142-S147 in addition to the process of S140.

[0113] In S142, the processor 210 adjusts the parameters of the estimation models MCx, MCy, and MCa (i.e., the parameters of the state space model). In this embodiment, the processor 210 updates the data indicating the parameters included in the estimation models MCx, MCy, and MCa to data indicating the adjusted parameters. In S142, the parameters s 2 , sa 2 , sb 2 , φ(i) is adjusted. In S142, the variance s1 2 and the variance s2 of V2t 2 The processor 210 determines each parameter by maximum likelihood estimation, similarly to S220.

[0114] In the maximum likelihood estimation in S142, time series data of the auxiliary variables are used in addition to the time series data of the target variables. For example, the processor 210 performs maximum likelihood estimation for the estimation model MCx using time series data TSx of the first coordinate Px, time series data TS1 of the current AP, and time series data TS2 of the first angle Aa1. The time series data TS1 and TS2 of the auxiliary variables are used in common for the three estimation models MCx, MCy, and MCa. The processor 210 uses the time series data to calculate the parameters of the estimation models MCx, MCy, and MCa (here, parameters s 2 , sa 2 , sb 2 , φ(i), s1 2 , s2 2 ) can be said to be learning.

[0115] In S145, the processor 210 repeats the filter process from time 1 to time t for each of the estimation models MCx, MCy, and MCa. The process in FIG. 6 is used as the filter process. In this embodiment, the estimation models MCx, MCy, and MCa each include a program for the filter process of the target variables Px, Py, and AG. The processor 210 executes the process of S145 according to the estimation models MCx, MCy, and MCa.

[0116] By the process of S145, the estimation models MCx, MCy, and MCa each include data indicating the state Xj and the variance VSj at each time point j from 1 to t. The data indicating the state Xj includes data indicating the coefficients E1j and E2j at the time point j (FIG. 12(E)). The processor 210 estimates the state Xj at each time point j from 1 to t, i.e., the coefficients E1j and E2j, by the filter process from time point 1 to time point t.

[0117] 14(A) and 14(B) are graphs showing examples of estimated coefficients E1j and E2j at each time point j from 1 to t. The horizontal axis indicates time Tm. The vertical axes of FIG. 14(A) and FIG. 14(B) indicate coefficients E1j and E2j, respectively. As shown in the figures, coefficients E1j and E2j may change according to changes in time point j.

[0118] As shown in FIG. 12(A), the coefficients E1t and E2t indicate weights given to the auxiliary variables U1t and U2t. When the influence of the change in the first auxiliary variable U1t on the change in the measurement value Yt is small, a small coefficient E1t of the first auxiliary variable U1t is calculated in S145 (FIG. 13(B)). On the other hand, when the influence of the change in the first auxiliary variable U1t on the change in the measurement value Yt is large, a large coefficient E1t of the first auxiliary variable U1t is calculated in S145. The same is true for the coefficient E2t of the second auxiliary variable U2t. In this way, the coefficient Eit indicates the magnitude of the influence of the change in the auxiliary variable Uit on the change in the measurement value Yt (i.e., the degree of contribution).

[0119] In the example of FIG. 14(A), the coefficient E1j increases as the time j passes. This indicates that the influence of the auxiliary variable corresponding to the coefficient E1j, i.e., the current AP, increases. In the example of FIG. 14(B), the coefficient E2j decreases as the time j passes. This indicates that the influence of the auxiliary variable corresponding to the coefficient E2j, i.e., the first angle Aa1, decreases.

[0120] In S147 (FIG. 13), the processor 210 outputs the third data. Then, the processor 210 proceeds to S180 (FIG. 3). FIG. 15 is a schematic diagram showing an example of the third data. In this embodiment, the third data is image data of a third image DP3. The processor 210 outputs the image data of the third image DP3 to the display unit 240 (FIG. 1). The display unit 240 displays the third image DP3 according to the received image data.

[0121] The third image DP3 shows the identification information IA and the weight information IB. In this embodiment, the identification information IA includes the identification information of each of all the auxiliary variables. In the example of FIG. 15, the name "E1" indicates the current AP, and the name "E2" indicates the first angle Aa1. The weight information IB includes a bar graph showing the values ​​of all the coefficients. The third image DP3 shows the correspondence between the identification information of the auxiliary variables and the values ​​of the coefficients. By observing the third image DP3, the operator can easily know that the auxiliary variable associated with a large weight is likely to be the cause of the abnormality symptom.

[0122] If the determination result of S130 is No, the processor 210 outputs the first data (FIG. 3: S140) and the third data (S147). The order of these outputs may be any order. For example, S147 may be performed after S140. The processor 210 may also output one piece of data including the first data and the third data (for example, image data of an image including a first image DP1 and a second image DP2).

[0123] As described above, in this embodiment, the processor 210 of the data processing device 200a (FIG. 10) acquires p time series data of p measurement values ​​of p auxiliary variables (p is an integer equal to or greater than 1) measured in synchronization with the measurement of the target variables Px, Py, AG in S105 (FIG. 13(A)) (p=2 in this embodiment). The p auxiliary variables include a current AP (FIG. 10) and a first angle Aa1 (FIG. 11(A)). The first angle Aa1 indicates the movement of the operator 800. In S142 (FIG. 13(B)), the processor 210 acquires a plurality of parameters (here, parameters s 2 , sa 2 , sb 2 , φ(i), s1 2 , s2 2 ) is learned. In this learning, time series data TSx, TSy, TSa of the measured values ​​of the target variables Px, Py, AG and p time series data TS1, TS2 of the measured values ​​of the p auxiliary variables AP, Aa1 are used. In S145, the processor 210 estimates p weights using the learned models of the estimation models MCx, MCy, MCa. Then, when the predicted values ​​of the target variables Px, Py, AG are outside the predetermined allowable ranges R1, R2, R3 (FIG. 3: S130: No), the processor 210 outputs third data in S147 (FIG. 13(B)) indicating both the p estimated values ​​of the p weights (weight information IB (FIG. 15)) and the identification information (identification information IA (FIG. 15)) that identifies each of the p auxiliary variables. Such third data can be used to estimate the cause of the abnormality symptom. In this embodiment, the third data is image data of a third image DP3 (FIG. 15). The processor 210 outputs this third data to the display unit 240. The worker can estimate the cause of the abnormality sign by observing the third image DP3. For example, it can be estimated that an auxiliary variable associated with a large weight is the cause of the abnormality sign. Then, the worker can review the settings of the attachment process before an abnormality occurs.

[0124] C. Variations: (1) In S105 (FIG. 13(A)), the method for estimating the posture of the worker 800 in the image may be any other method instead of the method using the machine learning model MD. For example, predetermined marks may be attached to the shoulders, elbows, and wrists of the worker 800. The processor 210 may detect the shoulders, elbows, and wrists in the image by pattern matching using images of the marks prepared in advance.

[0125] (2) In S105 (FIG. 13(A)), the processor 210 may standardize p auxiliary variables (p is an integer equal to or greater than 1). For example, the processor 210 may convert the p auxiliary variables so that the average value and variance of the p auxiliary variables are approximately the same when no abnormality signs are present. This allows the p weights (coefficients Eit in FIG. 12(A)) corresponding to the p auxiliary variables to be directly compared. Note that the conversion equation for each auxiliary variable may be experimentally determined in advance.

[0126] (3) The process using the prediction model or the estimation model may be various other processes instead of the processes in Fig. 4, Fig. 6, Fig. 13(A), and Fig. 13(B). For example, in S220 (Fig. 4) or S142 (Fig. 13(B)), the processor 210 may use a part of the acquired time series data (for example, the latest time series data for a predetermined period).

[0127] (4) The first data output in S140 (FIG. 3) may be data in various other formats instead of image data. For example, the processor 210 may output the first data, which is sound data representing a warning sound, to a sound reproduction device (not shown). In this way, the processor 210 may notify the operator that the predicted value is outside the allowable range by outputting the first data. In addition, the processor 210 may output the first data indicating that the predicted value is outside the allowable range to a storage device (e.g., the non-volatile storage device 230 or an external storage device (not shown) connected to the data processing device 200) (i.e., the first data is stored in the storage device).

[0128] The same applies to the second data output in S170. The processor 210 may notify the operator that the anomaly score is greater than the score threshold by outputting the second data. The processor 210 may also output the second data indicating that the anomaly score is greater than the score threshold to a storage device (e.g., the non-volatile storage device 230 or an external storage device (not shown) connected to the data processing device 200) (i.e., the second data is stored in the storage device).

[0129] (5) The configuration of the prediction model may be any configuration capable of predicting the value of the target variable, instead of the configuration of the prediction models MAx, MAy, and MAa described in FIG. 5(A)-FIG. 5(F), etc. For example, the prediction model may be a model called a Recurrent Neural Network (RNN). In S120 of FIG. 3, the processor 210 performs learning of the RNN model and prediction using the learned RNN model. Specifically, the processor 210 learns a plurality of parameters (e.g., weights used in calculations in the intermediate layer, etc.) included in the RNN model using at least a part of the time series data of the measured value of the target variable. Then, the processor 210 predicts the value of the target variable at a future time point after the latest time point of the time series data of the measured value of the target variable using the learned RNN model. The method of learning the RNN model may be any method suitable for the RNN model. For example, a method called BPTT (Backpropagation Through Time) or a method called Truncated BPTT may be adopted as the learning method. The prediction model may be a model called Long Short Term Memory (LSTM) (LSTM is a type of RNN). The prediction model may be an image recognition model (e.g., a convolutional neural network) that predicts future values ​​from an image of a graph showing time-series data such as that shown in FIG. 7(A).

[0130] Similarly, the configuration of the estimation model may be any configuration capable of estimating p weights associated with p auxiliary variables instead of the configurations of the estimation models MCx, MCy, and MCa described in FIG. 12(A)-FIG. 12(E) and the like. For example, the estimation model may be an RNN model that predicts a future value of a target variable using a target variable and p auxiliary variables. The RNN model may be configured to predict p future values ​​of p auxiliary variables in addition to the future value of the target variable. In S142 of FIG. 13(B), the processor 210 learns a plurality of parameters included in the RNN model using at least a part of the time series data of the measured value of the target variable and at least a part of the p time series data of the measured values ​​of the p auxiliary variables. The method of learning the RNN model may be any method suitable for the RNN model (for example, BPTT or truncated BPTT). S145 of FIG. 13(B) is changed to a process of estimating each weight of p auxiliary variables using a trained RNN model. For example, the processor 210 estimates the importance (i.e., weight) of each of p auxiliary variables from the trained RNN model according to an algorithm called SHAP (SHapley Additive exPlanations). SHAP is an example of an algorithm that calculates the importance value for prediction for each feature variable of a machine learning model. SHAP is disclosed in, for example, the paper "Scott M. Lundberg, Su-In Lee, "A Unified Approach to Interpreting Model Predictions", 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA."In addition, as a method for calculating the importance of variables from an RNN model using SHAP, for example, the method disclosed in the preprint “Long V. Ho, Melissa Aczon, David Ledbetter, Randall Wetzel, “Interpreting a Recurrent Neural Network's Predictions of ICU Mortality Risk”, arXiv:1905.09865 (2019), https: / / arxiv.org / abs / 1905.09865” may be adopted.

[0131] The method of learning the parameters of the prediction model or the estimation model is not limited to maximum likelihood estimation, and may be any method that adjusts the parameters of the model to values ​​suitable for time-series data. Here, any method suitable for the model may be adopted as the learning method. For example, the least squares method, the brute force method, a method using the backpropagation method and the gradient descent method, etc. may be used.

[0132] (6) The process using the calculation model may be various other processes instead of the process in Fig. 8. For example, in S420, the processor 210 may acquire past partial time series data that does not include the current time point t as the target partial time series data.

[0133] Furthermore, the method of calculating the anomaly score using the calculation model may be any method capable of calculating an anomaly score indicating the difference between the input time series data and L reference time series data (L is an integer equal to or greater than 1), instead of the k-nearest neighbor method. For example, the calculation model may be an anomaly detection model called "Encoder / Decoder for Anomaly Detection". Note that the processes using the calculation model (for example, S150, S160, and S170 in FIG. 3) may be omitted.

[0134] (7) The third data output in S147 (FIG. 13(B)) may be data showing various information related to the p estimated values ​​of the p weights estimated by the estimation model. For example, the third data may show an image of a graph showing the change over time of the weights as shown in FIG. 14(A) and FIG. 14(B).

[0135] Moreover, the number m of the estimated values ​​indicated by the third data among the p estimated values ​​of the p weights may be the same as p or may be less than p. In general, m may be an integer not less than 1 and not more than p. For example, the processor 210 may output the third data indicating the largest m estimated values ​​among the p estimated values ​​of the p weights. Here, the third data preferably indicates, in addition to the m estimated values, m identification information identifying the m auxiliary variables associated with the m estimated values. By using such third data, the operator can estimate the cause of the abnormality sign from the m auxiliary variables. Note that the third data may indicate m identification information without indicating the m estimated values. In this case, too, the cause of the abnormality sign can be estimated from the m auxiliary variables. Also, the third data may indicate m estimated values ​​without indicating m identification information.

[0136] The third data may be data in various other formats instead of image data. For example, the third data may be sound data that reads out m pieces of identification information (e.g., names of auxiliary variables). The processor 210 may output the third data, which is sound data, to a sound playback device (not shown). In this way, the processor 210 may notify the operator of an auxiliary variable that may be the cause of a symptom of an abnormality by outputting the third data. The processor 210 may also output the third data indicating one or both of the m estimated values ​​and the m pieces of identification information to a storage device (e.g., the non-volatile storage device 230, or an external storage device (not shown) connected to the data processing device 200) (i.e., the third data is stored in the storage device).

[0137] (8) The object of the detection process is not limited to the printer 900, but may be any product such as a scanner, a multifunction device, a digital camera, a cutting machine, or a mobile terminal.

[0138] (9) The auxiliary variables acquired in Fig. 13(A) are not limited to the current AP and the first angle Aa1, and may be any variable related to the manufacturing process of the product. The p auxiliary variables preferably include a variable indicating the movement of the worker. The variable indicating the movement of the worker may be, for example, the time from when a new printer 900 is brought in front of the worker 800 to when the label sheet 910 is affixed to the printer 900. If this time is significantly different from the normal time, there is a high possibility that an abnormality will occur in the future.

[0139] Also, the robot hand may attach the label sheet 910 to the printer 900. In this case, the p auxiliary variables may include a current or a voltage of a motor that drives the robot hand. If the position of the robot hand changes to attach the label sheet 910, the p auxiliary variables may include an output value of an encoder indicating the position of the robot hand (the encoder is attached to a movable part of the robot hand). A vibration sensor may be attached to the robot hand. The auxiliary variables may include the amplitude or frequency of vibration of the robot hand measured by the vibration sensor. Also, the p auxiliary variables may include the operating time of the robot hand or the number of processes.

[0140] Also, a roller device including a roller may attach the label sheet 910 to the printer 900. In this case, the p auxiliary variables may include a current or voltage of a motor that drives the roller device. Also, the p auxiliary variables may include an output value of an encoder indicating a rotational position of the roller. Also, the p auxiliary variables may include an amplitude or frequency of vibration of the rotor. Also, the p auxiliary variables may include an operating time or number of processing times of the roller device.

[0141] (10) The target variables may be any variables related to the status of the manufacturing process of the product, instead of the first coordinate Px, the second coordinate Py, and the angle AG of the label sheet 910 (FIGS. 2(A) and 2(B)). For example, the target variables may indicate the positions of parts to be attached to the object. The total number of target variables may be any integer number equal to or greater than 1, instead of 3.

[0142] (11) The detection process may be shared among multiple devices (for example, data processing devices such as computers) that can communicate with each other via a network (a system including these devices corresponds to an abnormal sign detection device).

[0143] In each of the above embodiments, a part of the configuration realized by hardware may be replaced by software, and conversely, a part or all of the configuration realized by software may be replaced by hardware. For example, the functions of the prediction models MAx, MAy, and MAa in FIG. 1 may be realized by a dedicated hardware circuit.

[0144] Furthermore, when some or all of the functions of the present invention are realized by a computer program, the program can be provided in a form stored in a computer-readable recording medium (e.g., a non-transitory recording medium). The program can be used in a state stored in the same or a different recording medium (computer-readable recording medium) from when it was provided. "Computer-readable recording medium" is not limited to portable recording media such as memory cards and CD-ROMs, but can also include internal storage devices within a computer, such as various ROMs, and external storage devices connected to a computer, such as a hard disk drive.

[0145] The present invention has been described above based on examples and modifications, but the above-described embodiments of the invention are intended to facilitate understanding of the present invention and are not intended to limit the present invention. The present invention may be modified or improved without departing from the spirit of the invention, and equivalents thereof are included in the present invention. [Explanation of symbols]

[0146] 110...digital camera (first camera), 120...digital camera (second camera), 130...sensor, 200, 200a...data processing device, 210...processor, 215...storage device, 220...volatile storage device, 230...non-volatile storage device, 231, 231a...program, 240...display unit, 250...operation unit, 270...communication interface, 700...supply device, 710...motor, 800...worker, 810...shoulder, 811...upper arm, 820...elbow, 821...forearm, 830...wrist, 900...object (printer), 900s...outer surface, 910...label sheet, MAx, MAy, MAa...prediction model, MBx, MBy, MBa...score calculation model, MCx, MCy, MCa...estimation model, MD...machine learning model

Claims

1. An abnormality symptom detection device, A data acquisition unit that acquires time series data of measured values ​​of a target variable related to the status of a manufacturing process of a product; a first learning unit that learns a plurality of parameters included in a prediction model for predicting a value of the target variable by using at least a portion of the time-series data of the measurement value of the target variable; a first prediction unit that predicts a value of the target variable at a future time point after a latest time point of the time series data of the measurement value by using a trained model of the prediction model; an output unit configured to output, when the predicted value of the target variable is outside a predetermined tolerance range, first data indicating that the predicted value of the target variable is outside the predetermined tolerance range; an anomaly score acquisition unit that acquires an anomaly score of partial time series data, which is data of a portion of the time series data of the measurement values, by inputting the partial time series data, which is data of a portion of the time series data of the measurement values, into a calculation model configured to calculate an anomaly score indicating a difference between the input time series data and L pieces of reference time series data (L is an integer of 1 or more); Equipped with When the anomaly score is greater than a predetermined score threshold, the output unit outputs second data indicating that the anomaly score is greater than the predetermined score threshold. An abnormality detection device.

2. The abnormality symptom detection device according to claim 1, The predictive model is configured to predict the value of the target variable using a plurality of components including a component that exhibits monotonic change over time. An abnormality detection device.

3. The abnormality symptom detection device according to claim 1 or 2, The data acquisition unit further acquires p time series data of p measurement values ​​of p auxiliary variables (p is an integer of 1 or more) that are measured in synchronization with the measurement of the target variable, the p auxiliary variables including a motion variable that indicates a motion of a worker; The abnormality symptom detection device further includes: a second learning unit that learns a plurality of parameters included in an estimation model for estimating p weights associated with the p auxiliary variables, using at least a portion of the time series data of the measurement values ​​of the target variable and at least a portion of the p time series data of the p measurement values ​​of the p auxiliary variables; an estimation unit that estimates the p weights using a trained model of the estimation model; Equipped with When the predicted value of the target variable is outside the predetermined allowable range, the output unit outputs, in addition to the first data, third data indicating one or both of m estimated values ​​(m is an integer of 1 to p) of the p estimated values ​​of the p weights and m pieces of identification information identifying m auxiliary variables associated with the m estimated values. An abnormality detection device.

4. The abnormality symptom detection device according to any one of claims 1 to 3, The target variable indicates a position or orientation of a label sheet attached to the product relative to the product. An abnormality detection device.

5. An abnormality symptom detection method, comprising: A data acquisition step of acquiring time series data of measured values ​​of a target variable related to the status of a manufacturing process of the product; a first learning step of learning a plurality of parameters included in a prediction model for predicting a value of the target variable using at least a portion of the time-series data of the measurement value of the target variable; a first prediction step of predicting a value of the target variable at a future time point after the latest time point of the time series data of the measurement value by using a trained model of the prediction model; an output step of outputting first data indicating that the predicted value of the target variable is outside a predetermined acceptable range when the predicted value of the target variable is outside the predetermined acceptable range; an anomaly score acquisition step of inputting partial time series data, which is data of a part of a time width of the time series data of the measurement values, into a calculation model configured to calculate an anomaly score indicating a difference between the input time series data and L pieces of reference time series data (L is an integer of 1 or more), thereby acquiring an anomaly score of the partial time series data; outputting, when the abnormality score is greater than a predetermined score threshold, second data indicating that the abnormality score is greater than the predetermined score threshold; An abnormality sign detection method comprising:

6. A computer program comprising: A data acquisition function for acquiring time series data of measured values ​​of a target variable related to the status of the manufacturing process of the product; a first learning function that learns a plurality of parameters included in a prediction model for predicting a value of the target variable by using at least a portion of the time series data of the measurement value of the target variable; A first prediction function that predicts a value of the target variable at a future time point after the latest time point of the time series data of the measurement value by using a trained model of the prediction model; an output function for outputting first data indicating that the predicted value of the target variable is outside a predetermined tolerance range when the predicted value of the target variable is outside the predetermined tolerance range; an anomaly score acquisition function that acquires an anomaly score of partial time series data, which is data of a part of a time width of the time series data of the measurement values, by inputting the partial time series data into a calculation model configured to calculate an anomaly score indicating a difference between the input time series data and L reference time series data (L is an integer equal to or greater than 1); a function of outputting, when the anomaly score is greater than a predetermined score threshold, second data indicating that the anomaly score is greater than the predetermined score threshold; A computer program that enables a computer to realize the above.

Images