Image generation program, method, and device

Abstract

To generate images capable of detecting anomalies that appear as changes in the correlation between multiple dimensions from multidimensional time-series data.SOLUTION: An image generation device generates a multidimensional first image representing the frequency characteristics at each time point of each time-series data set based on each multidimensional time-series data set, and then generates a single second image by combining the multidimensional first images weighted using a random matrix with different values for each frequency.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The disclosed technology relates to an image generation program, an image generation method, and an image generation device. [Background technology]

[0002] For example, machine learning models are used in anomaly detection systems that monitor time-series data such as the operating status of equipment and human biometric information to detect abnormalities in the target at an early stage. These machine learning models must be trained for each individual anomaly detection case, and particularly when deep learning is used as the machine learning model, this requires an enormous amount of training time.

[0003] In addition, a method has been proposed for detecting anomalies in image data using a trained image classifier as a base model, without training a deep learning model for each case. Therefore, a method has been proposed for converting time-series data into images in order to apply the anomaly detection method using this base model to anomaly detection in time-series data.

[0004] For example, a method has been proposed for handling multidimensional time series data by imaging each dimension of time series data individually using wavelet transform, Gramian angle field, etc., and then arranging the resulting images for each dimension to create a single image.

[0005] Also proposed is a signal processing method that performs a wavelet transform of multiple signals to determine a desired parameter, such as a physiological parameter. The method includes receiving first and second signals, performing a continuous wavelet transform on the first and second signals, and generating first and second scalograms based on the first and second transformed signals. The method also includes generating a scalogram mask based on the first and second scalograms, filtering the first and second scalograms with the scalogram mask, and determining the physiological parameter based on the filtered scalograms. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] US Patent Application Publication No. 2010 / 0014723 [Non-patent literature]

[0007] [Non-Patent Document 1] Biegel, T., Helm, P., Jourdan, N. et al., "SSMSPC: self-supervised multivariate statistical in-process control in discrete manufacturing processes," Journal of Intelligent Manufacturing, 2003. [Non-patent document 2] Li L., Li Q., ​​Yong W., Zhang S., Yang M., Jiang P., "Intelligent Online Inspection of the Paste Quality of Prebaked Carbon Anodes Using an Anomaly Detection Algorithm. Systems," 2023; 11(9):484. Summary of the Invention [Problem to be solved by the invention]

[0008] In many cases, time-series data is multidimensional data acquired by multiple sensors. To achieve highly accurate anomaly detection, it is desirable to visualize the time-series data while taking into account the correlation between dimensions. However, images in which time-series data for each dimension are visualized individually and then arranged may not be able to detect anomalies that appear as changes in the correlation between multiple dimensions. Furthermore, the above-mentioned signal processing method mainly assumes two time-series data and cannot consider the correlation between any dimensions within the high-dimensional data.

[0009] In one aspect, the disclosed technology aims to generate an image capable of detecting an abnormality that appears as a change in correlation between multiple dimensions from multidimensional time-series data. [Means for solving the problem]

[0010] In one embodiment, the disclosed technology generates a multidimensional first image representing the frequency characteristics of each time point of each piece of multidimensional time-series data based on the respective pieces of multidimensional time-series data, and then generates a single second image by combining the multidimensional first images weighted using a random matrix in which different values ​​are assigned to each frequency. [Effects of the Invention]

[0011] One aspect is that it has the effect of being able to generate an image from multidimensional time-series data that can detect abnormalities that appear as changes in correlation between multiple dimensions. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 10 is a diagram for explaining an abnormality that appears as a change in correlation between multiple dimensions. [Figure 2] FIG. 10 is a diagram for explaining an example in which an abnormality cannot be detected when the first principal component is selected. [Figure 3] FIG. 2 is a diagram for explaining the image generation in the first embodiment. [Figure 4] FIG. 2 is a functional block diagram of the anomaly detection system according to the first and third embodiments. [Figure 5] FIG. 10 is a diagram for explaining generation of n scalogram images. [Figure 6] FIG. 10 is a diagram for explaining generation of a random matrix. [Figure 7] FIG. 10 is a diagram for explaining the effect of using a random matrix. [Figure 8] FIG. 10 is a diagram for explaining generation of a single RGB image. [Figure 9]FIG. 1 is a block diagram showing a schematic configuration of a computer that functions as an image generating apparatus. [Figure 10] 5 is a flowchart showing an example of image generation processing according to the first embodiment. [Figure 11] 10 is a flowchart illustrating an example of an abnormality detection process. [Figure 12] FIG. 10 is a functional block diagram of an anomaly detection system according to a second embodiment. [Figure 13] FIG. 2 is a detailed functional block diagram of the AE training unit. [Figure 14] 10 is a flowchart illustrating an example of an AE training process. [Figure 15] 10 is a flowchart showing an example of image generation processing according to the second embodiment. [Figure 16] 11 is a flowchart showing an example of image generation processing according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, an example of an embodiment of the disclosed technology will be described with reference to the drawings.

[0014] Before describing the details of each embodiment, the necessity of generating an image capable of detecting anomalies that appear as changes in correlation between multiple dimensions when generating an image from multidimensional time-series data will be explained.

[0015] FIG. 1A shows an example of time-series data f1 and f2 under normal conditions. As shown in FIG. 1B, assume that the time-series data f1 has a portion (shaded portion) that differs from normal conditions. In this case, conventional technology can detect the occurrence of an abnormality. However, as shown in FIG. 1C, although there is no significant difference between the time-series data f1 and f2 and normal conditions, when the correlation between the time-series data f1 and f2 is examined, it differs from the correlation under normal conditions, and this is a situation that should be detected as an abnormality. Such a situation cannot be detected with conventional technology.

[0016] Furthermore, there are technologies for compressing multidimensional time series data into one-dimensional time series data, such as principal component analysis (PCA) and autoencoder (AE). Therefore, it is possible to compress multidimensional time series data into one-dimensional time series data using PCA or AE, and then apply imaging using wavelet transform, Gramian angle fields, or the like. For example, time series data f1, f2, and f3 shown in Figure 2A are subjected to principal component analysis and converted into time series data PC1 of the first principal component, time series data PC2 of the second principal component, and time series data PC3 of the third principal component, as shown in Figure 2B. Then, suppose the time series data PC1 of the first principal component is visualized. In this case, even if a change indicating an abnormality occurs in the time series data f3, the change cannot be detected.

[0017] Therefore, in the following embodiments, when generating an image from multidimensional time-series data, an image capable of detecting an abnormality that appears as a change in correlation between multiple dimensions is generated. In a first embodiment, a wavelet transform is applied to n-dimensional time-series data to generate n scalogram images. Then, as shown in FIG. 3, a random matrix is ​​used to linearly combine the n scalogram images (three in the example of FIG. 3) to generate a combined scalogram image, which is a single image. Note that the random matrix shown in FIG. 3 is a simplified representation, and in reality, it is a matrix with the same pixel size as the scalogram image.

[0018] Since scalograms are sparse and have values ​​close to 0 in most regions (pixels), the original information is unlikely to be lost even when linearly combined using a random matrix. In addition, by using different weights in the frequency direction, it is possible to grasp the correlation between dimensions in various combinations. For example, the vertical stripes (dashed oval areas) seen in the combined scalogram image shown in Figure 3 represent the correlation between dimensions.

[0019] In addition, in the second embodiment, a machine learning model such as a neural network that compresses multiple images into one image is used instead of the random matrix used in the first embodiment. Each embodiment will be described in detail below.

[0020] First Embodiment 4, the anomaly detection system 100 according to the first embodiment includes an image generation device 10 and an anomaly detection device 30. The image generation device 10 and the anomaly detection device 30 are connected via a network.

[0021] N-dimensional time series data is input to the image generation device 10, which generates a single image from the n-dimensional time series data. n is an integer equal to or greater than 2. The n-dimensional time series data may be, for example, data detected at each time by n types of sensors. The anomaly detection device 30 performs anomaly detection using the image generated by the image generation device 10. Functionally, the image generation device 10 includes a first generation unit 12 and a second generation unit 14.

[0022] The first generator 12 generates a multidimensional first image representing the frequency characteristics of each piece of n-dimensional time series data input to the image generator 10 at each time point.

[0023] Specifically, in order to prevent information duplication when linearly combining n-dimensional scalogram images (described in detail later), the first generation unit 12 converts the n-dimensional time series data into time series data indicating feature quantities mapped onto n-dimensional principal component axes by principal component analysis. More specifically, the first generation unit 12 applies principal component analysis to the dimension directions of the n-dimensional time series data to calculate eigenvectors and eigenvalues ​​(contribution degrees). The first generation unit 12 then performs orthogonalization using the eigenvectors to convert the data into time series data (hereinafter referred to as "n-dimensional principal component time series data") having feature quantities for each time point as values ​​for each of the n-dimensional principal component axes. Note that the feature quantities mapped onto the n-dimensional principal component axes can also be considered as numerical values ​​obtained by linearly transforming the n-dimensional time series data using an eigenvector matrix.

[0024] The first generation unit 12 also performs a wavelet transform on each principal component time series data of the n-dimensional principal component time series data to calculate a scalogram (time / frequency characteristics). Note that the time-frequency analysis method is not limited to the wavelet transform as long as it is a transform method that can obtain a sparse image, such as performing a short-time Fourier transform on each principal component time series data to calculate a spectrum. The wavelet transform used may be a complex Morlet wavelet, a Ricker wavelet, or the like.

[0025] For example, the first generation unit 12 associates the column direction (horizontal direction) of the scalogram image with time and the row direction (vertical direction) with frequency. Then, the first generation unit 12 generates the scalogram intensity S k (i,j) is embedded in the pixel position (i,j) of the scalogram image. The value to be embedded in the scalogram image is calculated by the following formula (1): k The normalized value of (i,j) is S k ~ It can also be (i,j). S k ~ (i,j) =S k (i,j) / max(|S k,min |,|S k,max |) (1) S k,min is S k The minimum value of (i,j), S k,max is S k (i,j) is the maximum value. Depending on the type of wavelet transform used, S k Since (i,j) may be negative, the absolute value is taken.

[0026] As a result, n scalogram images are generated from n-dimensional (n=8 in the example of FIG. 5) time-series data as shown in Fig. 5. The scalogram image is an example of the "first image" of the disclosed technology.

[0027] Furthermore, the first generating unit 12 may generate two types of n scalogram images representing two different types of frequency characteristics for each time series data. The two types of frequency characteristics may be, for example, the real and imaginary parts of a complex Morlet wavelet transform, or the absolute value of a complex Morlet wavelet and a Ricker wavelet.

[0028] The second generation unit 14 generates a single image by combining n scalogram images weighted using a random matrix in which a different value is assigned to each frequency.

[0029] Specifically, the second generation unit 14 generates a random matrix for combining the scalogram images at each time based on the contribution obtained by the principal component analysis. For example, as shown in A of Fig. 6, the second generation unit 14 creates a sparse matrix in which a weight of 1 is assigned to m random principal components (m = 3 in the example of Fig. 6) for each frequency, and a weight of 0 is assigned to the remaining principal components. Furthermore, as shown in B of Fig. 6, the second generation unit 14 multiplies the sparse matrix by the contribution obtained by PCA for each principal component. Then, as shown in C of Fig. 6, the second generation unit 14 normalizes the weights so that the sum of the elements for each frequency is 1.

[0030] Note that the above-described random matrix generation method is merely an example, and a different generation method may be adopted as long as the random matrix has different weights in the frequency direction. The second generation unit 14 may use the same random matrix in the time direction, or may use a random matrix with different weights in the time direction as well, as shown in Fig. 3. In the latter case, the random matrix may be generated for each time point using, for example, the above-described method.

[0031] Second generation unit 14 linearly combines n scalogram images weighted by a random matrix to generate a single combined scalogram image. Note that if first generation unit 12 generates two types of n scalogram images, second generation unit 14 generates two types of combined scalogram images.

[0032] For example, suppose scalogram images are generated for each of the two-dimensional time series data shown in Figure 7A. Figure 7B shows a waveform illustrating the temporal change in scalogram intensity at frequencies corresponding to the solid and dashed lines in the scalogram image shown in Figure 7A. Figure 7C shows a waveform illustrating the fluctuation in the average value of the solid and dashed waveforms in Figure 7B. As shown in Figure 7B, during training (normal conditions), the two waveforms are nearly synchronized. During testing (abnormal conditions), the periods of the two waveforms change slightly, but portions of the individual scalogram images (waveforms) are almost identical to those in normal conditions, making it difficult to detect abnormalities from these waveforms. By combining the two scalogram images (waveforms), completely different waveforms are obtained for normal and abnormal conditions, as shown in Figure 7C, making it easy to detect abnormalities.

[0033] However, when dealing with m-dimensional time series data, all dimensions m For the C2 combinations, combining two scalogram images (waveforms) at all frequencies to detect anomalies is computationally difficult. Also, depending on the combination of dimensions, values ​​may cancel each other out, making it impossible to detect anomalies. It is also possible to select and combine any of the frequencies, as shown in Figure 7, but in that case, it may not be possible to detect anomalies depending on the selected frequency.

[0034] Therefore, as in this embodiment, by combining scalogram images using weighting with a random matrix in which a different value is assigned for each frequency, m dimensions are selected randomly for each frequency. This makes it possible to check various combinations of dimensions, increasing the possibility of detecting disturbances in changes in the correlation between dimensions. Furthermore, since the same two-dimensional combination appears multiple times in the frequency direction, the possibility of detecting an abnormality at some frequency increases.

[0035] Second generation unit 14 may output this combined scalogram image as an image used for anomaly detection, but in this embodiment, second generation unit 14 generates an RGB image (hereinafter referred to as a "single RGB image") as a single image. The combined scalogram image and the single RGB image are examples of the "second image" of the disclosed technology.

[0036] Specifically, second generation unit 14 embeds the value of the combined scalogram image into the first component of the R, G, and B components of an RGB image, in which the horizontal direction corresponds to the time direction and the vertical direction corresponds to the frequency direction. Furthermore, when two types of combined scalogram images have been generated, second generation unit 14 embeds the value of the other type of combined scalogram image into the second component. Furthermore, second generation unit 14 sets values ​​that vary in stages according to frequency, such as setting a larger value as the frequency increases, and embeds the set value into the third component.

[0037] For example, suppose two types of combined scalogram images have been generated: one for the absolute value of the complex Morlet wavelet and one for the Ricker wavelet. In this case, the second generation unit 14 embeds the value of the combined scalogram image for the absolute value of the complex Morlet wavelet in the G component, as shown in A of FIG. 8. The second generation unit 14 also embeds the value of the combined scalogram image for the Ricker wavelet in the R component, as shown in B of FIG. 8. The second generation unit 14 also embeds a value set according to frequency in the B component, as shown in C of FIG. 8. Then, the second generation unit 14 generates a single RGB image by combining the R component, G component, and B component with the embedded values, as shown in D of FIG. 8. Note that in FIG. 8, for convenience of illustration, the images of each component and the RGB image are shown in grayscale.

[0038] Furthermore, when the second generation unit 14 generates a combined scalogram image using a random matrix that also varies in the time direction, it embeds the value of the combined scalogram image in the first component of the R, G, and B components of the RGB image. The second generation unit 14 embeds values ​​that vary in stages depending on frequency in the second component. The second generation unit 14 also sets values ​​that vary in stages depending on the passage of time, for example, by setting a larger value as time passes, and embeds the set value in the third component. For example, when a combined scalogram image is generated for the absolute value of a complex Morlet wavelet, the second generation unit 14 embeds values ​​that vary in stages depending on the passage of time in the R component shown in B of FIG. 8. In this case, the image of the B component in which values ​​depending on frequency are embedded shown in C of FIG. 8 has a vertical gradation, while the image of the R component in which values ​​depending on the passage of time are embedded has a horizontal gradation. The second generation unit 14 outputs the generated single RGB image to the anomaly detection device 30.

[0039] The anomaly detection device 30 functionally includes a training unit 32, an anomaly detection model 34, and a detection unit 36.

[0040] The training unit 32 acquires a training data set to be input to the anomaly detection device 30. Each training data set included in the training data set is a single RGB image generated based on n-dimensional time-series data under normal conditions. The training unit 32 uses the acquired training data set to train the anomaly detection model 34. The anomaly detection model 34 may be, for example, a machine learning model configured using a deep neural network or the like.

[0041] The detection unit 36 ​​acquires a single RGB image generated by the image generation device 10 based on the n-dimensional time-series data that is the target of anomaly detection. The detection unit 36 ​​inputs the acquired single RGB image to the anomaly detection model 34, and acquires and outputs the anomaly detection result output from the anomaly detection model 34.

[0042] The image generating device 10 may be realized by, for example, a computer 40 shown in Fig. 9. The computer 40 includes a CPU (Central Processing Unit) 41, a GPU (Graphics Processing Unit) 42, a memory 43 as a temporary storage area, and a non-volatile storage device 44. The computer 40 also includes an input / output device 45 such as an input device and a display device, and an R / W (Read / Write) device 46 that controls reading and writing of data from and to a storage medium 49. The computer 40 also includes a communication I / F (Interface) 47 that is connected to a network such as the Internet. The CPU 41, GPU 42, memory 43, storage device 44, input / output device 45, R / W device 46, and communication I / F 47 are connected to one another via a bus 48.

[0043] The storage device 44 is, for example, a hard disk drive (HDD), a solid state drive (SSD), a flash memory, etc. The storage device 44 serving as a storage medium stores an image generation program 50 for causing the computer 40 to function as the image generation device 10. The image generation program 50 has first generation process control instructions 52 and second generation process control instructions 54.

[0044] The CPU 41 reads the image generation program 50 from the storage device 44, expands it in the memory 43, and sequentially executes the control instructions of the image generation program 50. The CPU 41 operates as the first generation unit 12 shown in FIG. 4 by executing the first generation process control instruction 52. The CPU 41 also operates as the second generation unit 14 shown in FIG. 4 by executing the second generation process control instruction 54. As a result, the computer 40 that has executed the image generation program 50 functions as the image generation device 10. The CPU 41 that executes the program is hardware. A part of the program may also be executed by the GPU 42.

[0045] The functions realized by the image generation program 50 may be realized by, for example, a semiconductor integrated circuit, more specifically, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), etc. The hardware configuration of the anomaly detection device 30 is generally similar to the hardware configuration of the image generation device 10 shown in Fig. 9 except for the program stored in the storage device 44, and therefore a detailed description thereof will be omitted.

[0046] Next, the operation of the anomaly detection system 100 according to the first embodiment will be described. When n-dimensional time series data is input to the image generation device 10 and an instruction to generate a single RGB image is issued, the image generation device 10 executes the image generation process shown in FIG. 10 . Note that the image generation process is an example of an image generation method of the disclosed technology. When normal n-dimensional time series data is input and the image generation process is executed, a single RGB image is generated as training data for training the anomaly detection model 34. Once multiple single RGB images that serve as training data are generated, they are input to the anomaly detection device 30 as a training data set. The anomaly detection device 30 then trains the anomaly detection model 34 using the training data set. When a single RGB image generated based on n-dimensional time series data that is the target of anomaly detection is input to the anomaly detection device 30 with the trained anomaly detection model 34 stored in the anomaly detection device 30, the anomaly detection device 30 executes the anomaly detection process shown in FIG. 11 . The image generation process and the anomaly detection process are each described in detail below.

[0047] First, the image generation process shown in FIG. 10 will be described.

[0048] In step S10, the first generation unit 12 acquires the n-dimensional time series data input to the image generation device 10. Next, in step S12, the first generation unit 12 converts the n-dimensional time series data into n-dimensional principal component time series data by principal component analysis.

[0049] Next, in step S14, the first generation unit 12 performs a wavelet transform on each principal component time series data of the n-dimensional principal component time series data to calculate a scalogram. The first generation unit 12 calculates the intensity S of the scalogram at frequency j at time i calculated from the k-th dimension principal component time series data. k (i,j) is the k-th dimension of S k The normalized value S based on the maximum and minimum values ​​of (i,j) k ~ Then, the first generation unit 12 calculates S at the pixel position (i, j) of the image in which the horizontal direction corresponds to the time direction and the vertical direction corresponds to the frequency direction. k ~ By embedding (i, j), a scalogram image for the kth dimension (k=1, 2, . . . , n) is generated, resulting in n scalogram images.

[0050] Next, in step S16, second generation unit 14 generates a random matrix to which a different value is assigned for each frequency based on the contribution obtained by the principal component analysis. Next, in step S18, second generation unit 14 linearly combines the n scalogram images weighted by the random matrix to generate a single combined scalogram image.

[0051] Next, in step S20, the second generation unit 14 embeds the values ​​of the combined scalogram image into the first components of the R, G, and B components of the RGB image, in which the horizontal direction corresponds to the time direction and the vertical direction corresponds to the frequency direction, and embeds values ​​that gradually change according to frequency into the second components. Furthermore, if two types of combined scalogram images have been generated for the remaining components, the second generation unit 14 embeds the values ​​of another type of combined scalogram image. Alternatively, the second generation unit 14 embeds values ​​that gradually change according to the passage of time into the remaining components. The second generation unit 14 then combines the R, G, and B components with the embedded values ​​to generate and output a single RGB image, and the image generation process ends.

[0052] Next, the abnormality detection process shown in FIG. 11 will be described.

[0053] In step S30, the detection unit 36 ​​acquires a single RGB image of the target of anomaly detection from the image generation device 10. Next, in step S32, the detection unit 36 ​​inputs the acquired single RGB image to the anomaly detection model 34. Next, in step S34, the detection unit 36 ​​acquires and outputs the anomaly detection result output from the anomaly detection model 34, and the anomaly detection process ends.

[0054] As described above, according to the anomaly detection system of the first embodiment, the image generation device generates a multidimensional first image representing the frequency characteristics of each piece of multidimensional time-series data at each time point, based on each piece of multidimensional time-series data. The image generation device then generates a single second image by combining the multidimensional first images weighted using a random matrix in which a different value is assigned to each frequency. This makes it possible to generate an image from the multidimensional time-series data that can detect anomalies that appear as changes in correlations between multiple dimensions. As a result, by applying the image classifier, which is a base model, it is possible to quickly create an anomaly detection model for any multidimensional time-series data.

[0055] Second Embodiment Next, a second embodiment will be described. In the anomaly detection system according to the second embodiment, the same components as those in the anomaly detection system 100 according to the first embodiment are denoted by the same reference numerals, and components having some common functions are denoted by reference numerals having the same last two digits, and detailed descriptions thereof will be omitted.

[0056] 12, an anomaly detection system 200 according to the second embodiment includes an image generation device 210 and an anomaly detection device 30. The image generation device 210 and the anomaly detection device 30 are connected via a network.

[0057] The image generating device 210 functionally includes an AE training unit 20, a first generating unit 212, and a second generating unit 214. In addition, an encoder 26 is stored in a predetermined storage area of ​​the image generating device 210.

[0058] More specifically, the AE training unit 20 includes, as functional units, a conversion unit 21, an encoding unit 22, a decoding unit 23, an inverse conversion unit 24, and a loss calculation unit 25, as shown in FIG.

[0059] The transform unit 21 acquires n-dimensional time series data for training input to the image generating device 210. The transform unit 21 performs wavelet transform on each piece of time series data in the n-dimensional time series data to generate a scalogram image. The method for generating the scalogram image is similar to the process in which the first generating unit 12 in the first embodiment generates n scalogram images from n-dimensional principal component time series data. Note that instead of wavelet transform, other transform methods capable of inverse transform, such as Fourier transform, may be used.

[0060] Encoding unit 22 is an encoder configured with a neural network, and decoding unit 23 is a decoder configured with a neural network, and encoding unit 22 and decoding unit 23 together form an autoencoder (AE). Encoding unit 22 converts the n scalogram images generated by conversion unit 21 into a single RGB image. Decoding unit 23 restores the single RGB image converted by encoding unit 22 into n restored scalogram images.

[0061] The inverse transform unit 24 performs an inverse wavelet transform on each of the n restored scalogram images to convert it into n-dimensional restored time-series data.

[0062] The loss calculation unit 25 calculates the loss between the input n-dimensional time series data and the n-dimensional restored time series data converted by the inverse conversion unit 24. For example, the loss calculation unit 25 calculates the error between each piece of time series data and the restored time series data corresponding to that time series data as the loss. The error may be, for example, the squared error between a waveform representing the time series data and a waveform representing the restored time series data.

[0063] The loss calculation unit 25 updates the parameters of the encoder and decoder so as to minimize the calculated loss. The loss calculation unit 25 repeatedly executes the processes of the conversion unit 21, the encoding unit 22, the decoding unit 23, and the inverse conversion unit 24 until the loss is minimized. The loss calculation unit 25 may determine that the loss has been minimized, for example, when the number of times the process has been repeated exceeds a predetermined number, when the loss is equal to or less than a predetermined value, or when the difference between the loss calculated last time and the loss calculated this time is equal to or less than a predetermined value. The loss calculation unit 25 extracts the encoder 26 when the loss is minimized and stores it in a predetermined storage area.

[0064] That is, the AE training unit 20 generates an encoder for converting n scalogram images into a single RGB image by training a machine learning model configured such that the AE is sandwiched between the conversion unit 21 and the inverse conversion unit 24.

[0065] The first generation unit 212 acquires n-dimensional time series data for which an image is to be generated, which is input to the image generation device 210. The first generation unit 212 is similar to the first generation unit 212 in the first embodiment, except that the first generation unit 212 generates n scalogram images without converting the n-dimensional time series data into n-dimensional principal component time series data.

[0066] The second generation unit 214 generates a single RGB image by inputting the n scalogram images generated by the first generation unit 212 to the trained encoder 26.

[0067] The image generation device 210 may be realized by, for example, a computer 40 shown in Fig. 9. An image generation program 250 for causing the computer 40 to function as the image generation device 210 is stored in a storage device 44 of the computer 40. The image generation program 250 includes an AE training process control instruction 60, a first generation process control instruction 252, and a second generation process control instruction 254. The storage device 44 also includes an information storage area 66 in which information configuring the encoder 26 is stored.

[0068] The CPU 41 reads the image generation program 250 from the storage device 44, loads it in the memory 43, and sequentially executes the control instructions of the image generation program 250. The CPU 41 operates as the AE training unit 20 shown in FIG. 12 by executing the AE training process control instruction 60. The CPU 41 also operates as the first generation unit 212 shown in FIG. 12 by executing the first generation process control instruction 252. The CPU 41 also operates as the second generation unit 214 shown in FIG. 12 by executing the second generation process control instruction 254. The CPU 41 also reads information from the information storage area 66 and loads the encoder 26 in the memory 43. As a result, the computer 40 that has executed the image generation program 250 functions as the image generation device 210. The CPU 41 that executes the program is hardware. A part of the program may be executed by the GPU 42.

[0069] The functions realized by the image generation program 250 may be realized by, for example, a semiconductor integrated circuit, more specifically, an ASIC, an FPGA, or the like.

[0070] Next, the operation of the anomaly detection system 200 according to the second embodiment will be described. When n-dimensional time series data for training is input to the image generation device 210 and an instruction to train the encoder is given, the image generation device 210 executes the AE training process shown in Fig. 14. When n-dimensional time series data for image generation is input to the image generation device 210 with the trained encoder 26 stored in the image generation device 210 and an instruction to generate a single RGB image is given, the image generation device 210 executes the image generation process shown in Fig. 15. The operation of the anomaly detection device 30 is the same as in the first embodiment. The AE training process and the image generation process will be described in detail below.

[0071] First, the AE training process shown in FIG. 14 will be described.

[0072] In step S40, transform unit 21 acquires n-dimensional time series data for training input to image generating device 210. Next, in step S42, transform unit 21 performs wavelet transform on each piece of time series data in the n-dimensional time series data to generate a scalogram image. Next, in step S44, encoding unit 22 converts the n scalogram images into a single RGB image using an encoder.

[0073] Next, in step S46, decoding unit 23 restores the single RGB image into n restored scalogram images using a decoder. Next, in step S48, inverse transform unit 24 performs an inverse wavelet transform on each of the n restored scalogram images to convert them into n-dimensional restored time-series data.

[0074] Next, in step S50, the loss calculation unit 25 calculates the loss between the n-dimensional time series data acquired in step S40 and the n-dimensional restored time series data converted in step S48. Next, in step S52, the loss calculation unit 25 determines whether the loss has been minimized. If the loss has been minimized, the process proceeds to step S56; if not, the process proceeds to step S54. In step S54, the loss calculation unit 25 updates the parameters of the encoder and decoder so as to minimize the loss, and the process returns to step S40. In step S56, the loss calculation unit 25 extracts the encoder 26 when the loss has been minimized and stores it in a predetermined storage area, and the AE training process ends.

[0075] Next, the image generation process in the second embodiment will be described with reference to FIG.

[0076] In step S60, first generation unit 212 acquires n-dimensional time-series data for image generation that is input to image generation device 210. Next, in step S62, first generation unit 212 generates n scalogram images from the n-dimensional time-series data. Next, in step S64, second generation unit 214 inputs the n scalogram images to trained encoder 26, thereby generating and outputting a single RGB image, and the image generation process ends.

[0077] As described above, according to the anomaly detection system of the second embodiment, the image generation device generates multidimensional first images representing frequency characteristics at each time of each piece of multidimensional time-series data, based on each piece of multidimensional time-series data. The image generation device also generates an encoder by training an autoencoder to convert the multidimensional first images into a single image using the multidimensional first images as training data. The image generation device then inputs the generated multidimensional first images into the encoder to generate a single second image by combining the multidimensional first images. This makes it possible to generate an image capable of detecting anomalies that appear as changes in correlation between multiple dimensions from the multidimensional time-series data. As a result, an anomaly detection model for any multidimensional time-series data can be created in a short time by applying the image classifier, which is a base model.

[0078] In the second embodiment, similarly to the first embodiment, a scalogram image may be generated after converting n-dimensional time series data into n-dimensional principal component time series data. In this case, an orthogonalization unit that converts n-dimensional time series data into n-dimensional principal component time series data may be inserted before the conversion unit 21, and a restoration unit that converts n-dimensional principal component time series data into n-dimensional time series data may be inserted after the inverse conversion unit 24.

[0079] In the first and second embodiments, a single RGB image is generated from n-dimensional time-series data as an image to be input to an anomaly detection model, but this is not limiting. Even if a single-component grayscale image representing a combined scalogram image is used, the same effects as those of the above embodiments can be obtained.

[0080] <Third embodiment> Next, a third embodiment will be described. In the anomaly detection system according to the third embodiment, the same components as those in the anomaly detection system 100 according to the first embodiment are denoted by the same reference numerals, and components having some common functions are denoted by reference numerals having the same last two digits, and detailed descriptions thereof will be omitted.

[0081] 4, an anomaly detection system 300 according to the third embodiment includes an image generation device 310 and an anomaly detection device 30. The image generation device 310 and the anomaly detection device 30 are connected via a network.

[0082] The image generating device 310 functionally includes a first generating unit 12 and a second generating unit 314. The first generating unit 12 is similar to that in the first embodiment, but in the third embodiment, it is necessary to generate multiple types of scalogram images representing different frequency characteristics. The first generating unit 12 generates, for example, scalogram images for the real part and the imaginary part of a complex Morlet wavelet transform. The first generating unit 12 may also generate, for example, scalogram images for the absolute value of the complex Morlet wavelet and the Ricker wavelet. The first generating unit 12 may also generate three types of scalogram images.

[0083] Second generation unit 314 generates a single RGB image by combining multiple types of n scalogram images. Specifically, second generation unit 314 embeds values ​​of each type of combined scalogram image into the R component, G component, and B component of an RGB image in which the horizontal direction corresponds to the time direction and the vertical direction corresponds to the frequency direction. When there are two types of scalogram images, second generation unit 314 may embed values ​​that gradually change depending on the frequency or the passage of time into the remaining components.

[0084] The second generation unit 314 synthesizes the R component, G component, and B component with the embedded values ​​to generate a single RGB image, and outputs the generated single RGB image to the anomaly detection device 30.

[0085] The image generation device 310 may be realized, for example, by a computer 40 shown in Fig. 9. An image generation program 350 for causing the computer 40 to function as the image generation device 310 is stored in the storage device 44 of the computer 40. The image generation program 350 has a first generation process control instruction 52 and a second generation process control instruction 354.

[0086] The CPU 41 reads the image generation program 350 from the storage device 44, loads it into the memory 43, and sequentially executes the control instructions of the image generation program 350. The CPU 41 operates as the first generation unit 12 shown in FIG. 4 by executing the first generation process control instruction 52. The CPU 41 also operates as the second generation unit 314 shown in FIG. 4 by executing the second generation process control instruction 354. As a result, the computer 40 that has executed the image generation program 350 functions as the image generation device 310. The CPU 41 that executes the program is hardware. A part of the program may also be executed by the GPU 42.

[0087] The functions realized by the image generation program 350 may be realized by, for example, a semiconductor integrated circuit, more specifically, an ASIC, an FPGA, or the like.

[0088] Next, the operation of the anomaly detection system 300 according to the third embodiment will be described. When n-dimensional time series data is input to the image generation device 310 and an instruction to generate a single RGB image is given, the image generation device 310 executes the image generation process shown in Fig. 16. The operation of the anomaly detection device 30 is the same as in the first embodiment. The image generation process in the third embodiment will be described in detail below.

[0089] In step S70, the first generation unit 12 acquires the n-dimensional time series data input to the image generation device 10. Next, in step S72, the first generation unit 12 converts the n-dimensional time series data into n-dimensional principal component time series data by principal component analysis.

[0090] Next, in step S74, the first generation unit 12 performs wavelet transform on each principal component time series data of the n-dimensional principal component time series data to generate a scalogram image. At this time, the first generation unit 12 generates two types of scalogram images representing two different types of frequency characteristics.

[0091] Next, in step S76, the second generation unit 314 embeds the values ​​of the first type of combined scalogram image into the first components of the R, G, and B components of the RGB image, in which the horizontal direction corresponds to the time direction and the vertical direction corresponds to the frequency direction. The second generation unit 314 also embeds the values ​​of the second type of combined scalogram image into the second components. The second generation unit 314 also embeds values ​​that vary stepwise depending on the frequency or the passage of time into the third component. The second generation unit 314 then combines the R, G, and B components with the embedded values ​​to generate and output a single RGB image, completing the image generation process.

[0092] As described above, according to the anomaly detection system of the third embodiment, the image generation device generates, based on each of the multidimensional time series data, multiple types of multidimensional first images representing multiple different types of frequency characteristics at each time of each of the time series data. The image generation device then generates a single second image by combining the multiple types of multidimensional first images. This makes it possible to generate an image that can improve the accuracy of anomaly detection compared to an image simply arranging images generated from each of the time series data, or a case where multidimensional time series data is compressed into one-dimensional time series data using PCA, AE, or the like and then imaged. Furthermore, by applying the image classifier, which is a base model, an anomaly detection model for any multidimensional time series data can be created in a short time.

[0093] In the first and third embodiments, the process of converting n-dimensional time series data into n-dimensional principal component time series data is not essential.

[0094] In addition, in each of the above embodiments, the image generation program is stored (installed) in advance in a storage device, but this is not limiting. The program according to the disclosed technology may be provided in a form stored in a storage medium such as a CD-ROM, a DVD-ROM, or a USB memory.

[0095] The following additional notes are provided regarding the above-described embodiments.

[0096] (Appendix 1) generating a multidimensional first image representing a frequency characteristic at each time of each piece of multidimensional time series data based on the pieces of multidimensional time series data; generating a single second image by combining the multi-dimensional first images weighted using a random matrix in which different values ​​are assigned to different frequencies; An image generation program for causing a computer to execute a process including the above.

[0097] (Appendix 2) 2. The image generation program according to claim 1, wherein the process of generating the multidimensional first image includes converting the multidimensional time series data into time series data indicating feature quantities mapped to multidimensional principal component axes by principal component analysis, and generating the multidimensional first image based on the converted time series data.

[0098] (Appendix 3) The image generation program according to claim 2, wherein the random matrix is ​​a matrix whose elements are values ​​obtained by multiplying a sparse matrix in which a weight of 1 is assigned to a predetermined number of first images randomly selected from the multidimensional first images and a weight of 0 is assigned to the other first images at each frequency by the contribution obtained by the principal component analysis, and normalizing the multiplied values ​​for each frequency.

[0099] (Appendix 4) The image generating program according to any one of Supplementary Note 1 to Supplementary Note 3, wherein the process of generating the second image includes weighting the multidimensional first image using a different random matrix at each time.

[0100] (Appendix 5) The image generation program according to claim 4, wherein the process of generating the second image includes embedding a weighted value for the first image into a first component among the R, G, and B components of the RGB image, embedding values ​​that gradually change depending on frequency into a second component, and embedding values ​​that gradually change depending on the passage of time into a third component.

[0101] (Appendix 6) the process of generating the multidimensional first image includes generating, for each time series data, two types of the multidimensional first image representing two different types of frequency characteristics; the process of generating the second image includes embedding a weighted value for the first type of the multidimensional first image into a first component among the R component, the G component, and the B component of the RGB image, embedding a weighted value for the second type of the multidimensional first image into a second component, and embedding values ​​that vary in stages according to frequency into a third component; The image generating program according to any one of Supplementary Note 1 to Supplementary Note 3, which includes:

[0102] (Appendix 7) 7. The image generating program according to claim 6, wherein the two types of frequency characteristics are the real part and imaginary part of a complex Morlet wavelet transform, or the absolute value of a complex Morlet wavelet and a Ricker wavelet.

[0103] (Appendix 8) generating a multidimensional first image representing a frequency characteristic at each time of each piece of multidimensional time series data based on the pieces of multidimensional time series data; generating a single second image by combining the multidimensional first images by inputting the generated multidimensional first images into an encoder obtained by training an autoencoder to convert the multidimensional first images into a single image using the multidimensional first images as training data; An image generation program for causing a computer to execute a process including the above.

[0104] (Appendix 9) The image generation program according to Appendix 8, wherein the encoder converts a multidimensional reconstructed image of the multidimensional first image obtained by inputting the multidimensional first image converted from the multidimensional time series data into multidimensional reconstructed time series data into the autoencoder including an encoder and a decoder, and the autoencoder is trained to minimize an error between the multidimensional time series data and the multidimensional reconstructed time series data.

[0105] (Appendix 10) generating a plurality of types of multidimensional first images representing different types of frequency characteristics at each time of each of the multidimensional time series data based on the respective multidimensional time series data; generating a single second image by combining the plurality of types of multidimensional first images; An image generation program for causing a computer to execute a process including the above.

[0106] (Appendix 11) generating a multidimensional first image representing a frequency characteristic at each time of each piece of multidimensional time series data based on the pieces of multidimensional time series data; generating a single second image by combining the multi-dimensional first images weighted using a random matrix in which different values ​​are assigned to different frequencies; An image generating method for causing a computer to execute a process including the steps of:

[0107] (Appendix 12) 12. The image generation method according to claim 11, wherein the process of generating the multidimensional first image includes converting the multidimensional time series data into time series data indicating feature quantities mapped onto multidimensional principal component axes by principal component analysis, and generating the multidimensional first image based on the converted time series data.

[0108] (Appendix 13) The image generation method according to claim 12, wherein the random matrix is ​​a matrix whose elements are values ​​obtained by multiplying a sparse matrix in which a weight of 1 is assigned to a predetermined number of first images randomly selected from the multidimensional first images and a weight of 0 is assigned to the other first images at each frequency by the contribution obtained by the principal component analysis, and normalizing the multiplied values ​​for each frequency.

[0109] (Appendix 14) The image generation method according to any one of Supplementary Note 11 to Supplementary Note 13, wherein the process of generating the second image includes weighting the multidimensional first image using a random matrix that differs at each time.

[0110] (Appendix 15) The image generating method according to claim 14, wherein the process of generating the second image includes embedding a weighted value for the first image into a first component among the R, G, and B components of the RGB image, embedding values ​​that vary in stages depending on frequency into a second component, and embedding values ​​that vary in stages depending on the passage of time into a third component.

[0111] (Appendix 16) the process of generating the multidimensional first image includes generating, for each time series data, two types of the multidimensional first image representing two different types of frequency characteristics; the process of generating the second image includes embedding a weighted value for the first type of the multidimensional first image into a first component among the R component, the G component, and the B component of the RGB image, embedding a weighted value for the second type of the multidimensional first image into a second component, and embedding values ​​that vary in stages according to frequency into a third component; The image generating method according to any one of Supplementary Note 11 to Supplementary Note 13, comprising:

[0112] (Appendix 17) generating a multidimensional first image representing a frequency characteristic at each time of each piece of multidimensional time series data based on the pieces of multidimensional time series data; generating a single second image by combining the multidimensional first images by inputting the generated multidimensional first images into an encoder obtained by training an autoencoder to convert the multidimensional first images into a single image using the multidimensional first images as training data; An image generating method for causing a computer to execute a process including the steps of:

[0113] (Appendix 18) The image generation method according to claim 17, wherein the encoder converts a multidimensional reconstructed image of the multidimensional first image obtained by inputting the multidimensional first image converted from the multidimensional time series data into multidimensional reconstructed time series data, into the autoencoder, which includes an encoder and a decoder, and is the encoder in the autoencoder trained to minimize an error between the multidimensional time series data and the multidimensional reconstructed time series data.

[0114] (Appendix 19) a first generation unit that generates a multidimensional first image representing frequency characteristics at each time of each piece of multidimensional time series data based on the multidimensional time series data; a second generation unit that generates a single second image by combining the multidimensional first images weighted using a random matrix in which a different value is assigned to each frequency; An image generating device comprising:

[0115] (Appendix 20) a first generation unit that generates a multidimensional first image representing frequency characteristics at each time of each piece of multidimensional time series data based on the multidimensional time series data; a second generation unit that generates a single second image by combining the multidimensional first images by inputting the generated multidimensional first images into an encoder obtained by training an autoencoder to convert the multidimensional first images into a single image using the multidimensional first images as training data; An image generating device comprising: [Explanation of symbols]

[0116] 10, 210, 310 Image generating device 12, 212 1st generation part 14, 214, 314 2nd generation part 20 AE Training Department 21 Conversion unit 22 Encoding section 23 Decoding section 24 Inverse conversion unit 25 Loss calculation section 26 Encoder 30 Anomaly detection device 32 Training Department 34 Anomaly Detection Model 36 Detection unit 40 Computer 41 CPU 42 GPU 43 Memory 44 Storage device 45 Input / Output Devices 46 R / W device 47 Communication I / F 48 Bus 49 Storage medium 50, 250, 350 Image Generation Program 52, 252 First generation process control instruction 54, 254, 354 Second generation process control instructions 60 AE Training Process Control Instructions 66 Information storage area 100, 200, 300 Anomaly Detection System

Claims

1. generating a multidimensional first image representing a frequency characteristic at each time of each piece of multidimensional time series data based on the pieces of multidimensional time series data; generating a single second image by combining the multi-dimensional first images weighted using a random matrix in which different values ​​are assigned to different frequencies; An image generation program for causing a computer to execute a process including the above.

2. 2. The image generation program according to claim 1, wherein the process of generating the multidimensional first image includes converting the multidimensional time series data into time series data indicating feature quantities mapped onto multidimensional principal component axes by principal component analysis, and generating the multidimensional first image based on the converted time series data.

3. The image generation program according to claim 2, wherein the random matrix is ​​a matrix whose elements are values ​​obtained by multiplying a sparse matrix in which a weight of 1 is assigned to a predetermined number of first images randomly selected from the multidimensional first images and a weight of 0 is assigned to the other first images at each frequency by the contribution obtained by the principal component analysis, and normalizing the multiplied values ​​for each frequency.

4. The image generation program according to any one of claims 1 to 3, wherein the process of generating the second image includes weighting the multidimensional first image using a different random matrix at each time.

5. 5. The image generation program according to claim 4, wherein the process of generating the second image includes embedding a weighted value for the first image into a first component of the R, G, and B components of the RGB image, embedding values ​​that gradually differ depending on frequency into a second component, and embedding values ​​that gradually differ depending on the passage of time into a third component.

6. the process of generating the multidimensional first image includes generating, for each time series data, two types of the multidimensional first image representing two different types of frequency characteristics; the process of generating the second image includes embedding a weighted value for the first type of the multidimensional first image into a first component among the R component, the G component, and the B component of the RGB image, embedding a weighted value for the second type of the multidimensional first image into a second component, and embedding a value that varies stepwise according to frequency into a third component; The image generating program according to any one of claims 1 to 3, comprising:

7. generating a multidimensional first image representing a frequency characteristic at each time of each piece of multidimensional time series data based on the pieces of multidimensional time series data; generating a single second image by combining the multidimensional first images by inputting the generated multidimensional first images into an encoder obtained by training an autoencoder to convert the multidimensional first images into a single image using the multidimensional first images as training data; An image generation program for causing a computer to execute a process including the above.

8. The image generation program of claim 7, wherein the encoder converts a multidimensional reconstructed image of the multidimensional first image obtained by inputting the multidimensional first image converted from the multidimensional time series data into the autoencoder including an encoder and a decoder, into multidimensional reconstructed time series data, and the encoder is trained to minimize the error between the multidimensional time series data and the multidimensional reconstructed time series data.

9. generating a plurality of types of multidimensional first images representing different types of frequency characteristics at each time of each of the multidimensional time series data based on the respective multidimensional time series data; generating a single second image by combining the plurality of types of multi-dimensional first images; An image generation program for causing a computer to execute a process including the above.

10. generating a multidimensional first image representing a frequency characteristic at each time of each piece of multidimensional time series data based on the pieces of multidimensional time series data; generating a single second image by combining the multi-dimensional first images weighted using a random matrix in which different values ​​are assigned to different frequencies; An image generating method for causing a computer to execute a process including the steps of:

11. generating a multidimensional first image representing a frequency characteristic at each time of each piece of multidimensional time series data based on the pieces of multidimensional time series data; generating a single second image by combining the multidimensional first images by inputting the generated multidimensional first images into an encoder obtained by training an autoencoder to convert the multidimensional first images into a single image using the multidimensional first images as training data; An image generating method for causing a computer to execute a process including the steps of:

12. a first generation unit that generates a multidimensional first image representing frequency characteristics of each piece of multidimensional time series data at each time point, based on each piece of multidimensional time series data; a second generation unit that generates a single second image by combining the multidimensional first images weighted using a random matrix in which a different value is assigned to each frequency; An image generating device comprising:

13. a first generation unit that generates a multidimensional first image representing frequency characteristics of each piece of multidimensional time series data at each time point, based on each piece of multidimensional time series data; a second generation unit that generates a single second image by combining the multidimensional first images by inputting the generated multidimensional first images into an encoder obtained by training an autoencoder to convert the multidimensional first images into a single image using the multidimensional first images as training data; An image generating device comprising:

Images