System and method for analyzing time series data including image

Abstract

The present invention relates to a system and method for analyzing time-series data, wherein the time-series data including an image is analyzed to detect abnormalities, etc., and provides a system for analyzing time-series data and an operation method therefor, the system comprising: a numerical data receiving unit for receiving time-series numerical data for a plurality of variables; an image data receiving unit for receiving time-series captured image data; a data processing unit for generating a feature map using the time-series numerical data and the image data; a data training unit for generating a time-series data analysis model by using a plurality of feature maps generated on the basis of a plurality of time points; and a data analysis unit for analyzing time-series data by using the generated time-series data analysis model.

Description

System and method for analyzing time series data including images

[0001] The present invention relates to a system and method for analyzing time series data including images, and more particularly, to a system and method for creating an artificial intelligence model that analyzes time series data of various variables and image data input as a time series, and analyzing time series data using the same.

[0002] As artificial intelligence technology develops, technologies that analyze various time-series monitoring data received to analyze industrial environments and detect or predict abnormal situations are being widely used.

[0003] In particular, the convolutional neural network (CNN) is a structure that extracts features from data and identifies patterns in the features, and has the advantage of being able to quickly analyze large-scale multivariate time series data, so it is being used to analyze multivariate time series data.

[0004] Prior art, Korean Patent Publication No. 10-2023-0033312, "CNN-based multivariate data processing system and CNN-based multivariate data processing method," is a technology that monitors and analyzes multivariate time series data using such a synthetic neural network. However, these prior arts have a problem in that it is difficult to consider the characteristics due to the correlation between each variable because they extract and analyze features from each multivariate time series variable. In addition, there is a problem in that it is difficult to accurately analyze the actual situation because only numerical data is analyzed.

[0005] Therefore, rather than simply analyzing the characteristics of each variable, a time series data analysis method is required that considers the correlation between variables and analyzes image data in relation to them.

[0006] The purpose of the present invention is to simultaneously analyze time series numerical data and image data to derive accurate analysis results.

[0007] The purpose of the present invention is to provide analysis results by analyzing time series numerical data of various variables.

[0008] The purpose of the present invention is to enable analysis of time series data by considering characteristics resulting from correlations between various variables, rather than by considering only the individual characteristics of various variables.

[0009] The present invention aims to increase analysis accuracy through correlation analysis between numerical data and image data by deriving a single characteristic map and analyzing the same rather than performing the analysis on numerical data and image data separately.

[0010] In order to achieve this purpose, a time series data analysis system according to an embodiment of the present invention may be configured to include a numerical data receiving unit that receives time series numerical data for a plurality of variables, an image data receiving unit that receives image data captured in a time series manner, a data processing unit that generates a characteristic map using the time series numerical data and the image data, a data learning unit that generates a time series data analysis model using a plurality of characteristic maps generated based on a plurality of time points, and a data analysis unit that analyzes time series data using the generated time series data analysis model.

[0011] At this time, the data processing unit may be configured to generate an original one-dimensional matrix by organizing time series numerical data of the past n points in time into an n * 1 matrix for each of the plurality of variables based on each point in time, generate a numerical characteristic map based on the generated original one-dimensional matrix for each of the plurality of variables, generate an image characteristic map using at least one of the images captured within the n points in time among the received image data, and generate the characteristic map by combining the numerical characteristic map and the image characteristic map.

[0012] In addition, the data processing unit may be configured to generate a numerical feature map having n columns using the original one-dimensional matrix, generate an image feature map having n columns based on at least one of the images captured within the n viewpoints among the received image data, and then generate the feature map by combining the numerical feature map and the image feature map.

[0013] In addition, the data processing unit may be configured to generate a combination one-dimensional matrix by combining two or more of the generated original one-dimensional matrices to form an n * 1 matrix for each combination, and to generate the numerical feature map using the original one-dimensional matrix and the combination one-dimensional matrix.

[0014] At this time, if the number of the plurality of variables is m, the data processing unit may be configured to generate a combined two-dimensional matrix by accumulating the original one-dimensional matrices to be combined for each of the cases of combining the generated original one-dimensional matrices from 2 to m, and input the generated combined two-dimensional matrix into a 1*k convolution layer (k is the number of original one-dimensional matrices to be combined) to generate a combined one-dimensional matrix.

[0015] The present invention can obtain the effect of deriving accurate analysis results by simultaneously analyzing time series numerical data and image data.

[0016] The present invention can obtain the effect of providing analysis results by analyzing time series numerical data of various variables.

[0017] The present invention can achieve the effect of enabling analysis of time series data by considering not only the individual characteristics of various variables, but also the characteristics resulting from correlations between various variables.

[0018] The present invention can achieve the effect of increasing analysis accuracy through correlation analysis between numerical data and image data by deriving and analyzing a single characteristic map rather than conducting analysis on numerical data and image data separately.

[0019] FIG. 1 is a diagram illustrating the internal configuration of a time series data analysis system according to one embodiment of the present invention.

[0020] FIG. 2 is a diagram illustrating an example of time series numerical data and time series image data input to a time series data analysis system according to an embodiment of the present invention.

[0021] FIG. 3 is a diagram illustrating a process of extracting feature data from time series numerical data and time series image data in a time series data analysis system according to an embodiment of the present invention.

[0022] FIG. 4 is a diagram illustrating an example of extracting feature data from time series numerical data and time series image data in a time series data analysis system according to an embodiment of the present invention.

[0023] FIG. 5 is a diagram showing an example of a feature extraction module used in a time series data analysis system according to an embodiment of the present invention.

[0024] FIG. 6 is a diagram showing an example of an inception module used in a time series data analysis system according to one embodiment of the present invention.

[0025] FIG. 7 is a diagram illustrating a neural network structure used in a time series data analysis system according to one embodiment of the present invention.

[0026] Figure 8 is a flowchart showing the flow of a time series data analysis method according to one embodiment of the present invention.

[0027] In order to achieve this purpose, a time series data analysis system according to an embodiment of the present invention may be configured to include a numerical data receiving unit that receives time series numerical data for a plurality of variables, an image data receiving unit that receives image data captured in a time series manner, a data processing unit that generates a characteristic map using the time series numerical data and the image data, a data learning unit that generates a time series data analysis model using a plurality of characteristic maps generated based on a plurality of time points, and a data analysis unit that analyzes time series data using the generated time series data analysis model.

[0028] At this time, the data processing unit may be configured to generate an original one-dimensional matrix by organizing time series numerical data of the past n points in time into an n * 1 matrix for each of the plurality of variables based on each point in time, generate a numerical characteristic map based on the generated original one-dimensional matrix for each of the plurality of variables, generate an image characteristic map using at least one of the images captured within the n points in time among the received image data, and generate the characteristic map by combining the numerical characteristic map and the image characteristic map.

[0029] In addition, the data processing unit may be configured to generate a numerical feature map having n columns using the original one-dimensional matrix, generate an image feature map having n columns based on at least one of the images captured within the n viewpoints among the received image data, and then generate the feature map by combining the numerical feature map and the image feature map.

[0030] In addition, the data processing unit may be configured to generate a combination one-dimensional matrix by combining two or more of the generated original one-dimensional matrices to form an n * 1 matrix for each combination, and to generate the numerical feature map using the original one-dimensional matrix and the combination one-dimensional matrix.

[0031] At this time, if the number of the plurality of variables is m, the data processing unit may be configured to generate a combined two-dimensional matrix by accumulating the original one-dimensional matrices to be combined for each of the cases of combining the generated original one-dimensional matrices from 2 to m, and input the generated combined two-dimensional matrix into a 1*k convolution layer (k is the number of original one-dimensional matrices to be combined) to generate a combined one-dimensional matrix.

[0032] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the attached drawings. In describing the present invention, detailed descriptions of known components or functions will be omitted if they are deemed to obscure the gist of the invention. Furthermore, specific numerical values ​​used in describing embodiments of the present invention are merely exemplary and do not limit the scope of the invention.

[0033] The time-series data analysis system according to the present invention may be configured as a server equipped with a central processing unit (CPU) and memory, and capable of connecting to other terminals via a communication network such as the Internet. However, the present invention is not limited to the configuration of the central processing unit and memory. Furthermore, the time-series data analysis system according to the present invention may be physically configured as a single device, or may be implemented in a distributed manner across multiple devices.

[0034] FIG. 1 is a diagram illustrating the internal configuration of a time series data analysis system according to one embodiment of the present invention.

[0035] As illustrated in the drawing, a time series data analysis system (101) according to one embodiment of the present invention may be configured to include a numerical data receiving unit (110), an image data receiving unit (120), a data processing unit (130), a data learning unit (140), and a data analysis unit (150).

[0036] Each component may be a software module that operates within the same physical computer system, and may be configured so that two or more physically separated computer systems can operate in conjunction with each other, and various embodiments that include the same function fall within the scope of the present invention.

[0037] The numerical data receiver (110) receives time-series numerical data for multiple variables. Time-series numerical data is continuously generated and collected over time and may include various information for monitoring industrial sites or equipment, such as factories. For example, status information, such as temperature and humidity, of a specific space may be continuously collected, and information, such as voltage and current, of power supplied to specific equipment may also be time-series data.

[0038] The time series numerical data received from the numerical data receiving unit (110) may be data measured at multiple points in time. For example, when voltage and current information is measured and collected at 1-second intervals, the time series numerical data may be data at each point in time at 1-second intervals.

[0039] Time series numerical data received from the numerical data receiving unit (110) can be synchronized at different times for analysis. In cases where there is a difference in the time at which data is measured depending on the type of data, data measured at the same time can be extracted to match the time, or data can be corrected for some data to correct the data so that it is in the same form as if it were measured at the same time, thereby achieving synchronization.

[0040] The image data receiving unit (120) receives image data captured in a time-series manner. The image data may be data received from a camera module, such as a CCTV, installed to check the status of a location or equipment being monitored. The image data may be an image captured at a specific point in time from a video received in real time, or an image captured as a still image periodically.

[0041] The image data received from the image data receiving unit (120) can be synchronized with each point in time of the time series numerical data so that it can be analyzed in relation to the time series numerical data, and when extracting the time series numerical data by a certain time interval and analyzing it, one or more image data captured within the time interval can be used. This allows for simultaneous analysis of numerically derived information and image information about the scene to produce accurate analysis results.

[0042] The data processing unit (130) generates a feature map using the time series numerical data and the image data. A feature map refers to data for analysis created by extracting features from data to enable rapid analysis in an artificial intelligence model such as a convolutional neural network (CNN). In general, a convolutional neural network uses a number of convolutional layers that extract features through filters to extract feature information that can be used for actual abnormal state detection, prediction, etc., and by learning using this, a necessary neural network model can be derived.

[0043] The data processing unit (130) generates an original one-dimensional matrix by organizing time series numerical data of the past n points in time into an n * 1 matrix for each of the plurality of variables based on each point in time, generates a numerical characteristic map based on the generated original one-dimensional matrix for each of the plurality of variables, generates an image characteristic map using at least one of the images captured within the n points in time among the received image data, and generates the characteristic map by combining the numerical characteristic map and the image characteristic map.

[0044] When the time series numerical data processed in the data processing unit (130) is composed of multiple variables, the characteristics of the numerical data are derived to create a numerical characteristic map, and an image characteristic map is derived using images captured within the time range of the numerical data, thereby enabling them to be combined and analyzed.

[0045] The image data used in the data processing unit (130) can be specified as image data captured at a predetermined point among the start point, midpoint, and end point of the time interval of the numerical data, depending on the characteristics of the analysis target. This can be selected in various ways depending on the characteristics of the analysis target, and the present invention is not limited by this selection method. However, the image data must be captured within the time interval range of the numerical data, so that the characteristics of the numerical data and the characteristics of the image data can be considered simultaneously for the same time interval.

[0046] More specifically, the data processing unit (130) generates a numerical feature map having n columns using the original one-dimensional matrix, generates an image feature map having n columns based on at least one of the images captured within the n viewpoints among the received image data, and then generates the feature map by combining the numerical feature map and the image feature map.

[0047] In order to generate an image feature map in the data processing unit (130), a preprocessing process such as size adjustment and a simplex process may be performed on the original image, and various methods may be applied as long as an image feature map having the same columns as time-series numerical data can be derived. The process of deriving the image feature map can derive the feature map as a matrix of a desired size through a convolution operation using a filter depending on the structure used when analyzing the image using a convolutional neural network (CNN). The present invention is not limited by a specific method for deriving a feature map from an image.

[0048] The data processing unit (130) generates a combination 1-dimensional matrix by combining two or more of the above-generated original 1-dimensional matrices to form an n * 1 matrix for each combination, and generates the numerical characteristic map using the original 1-dimensional matrix and the combination 1-dimensional matrix.

[0049] Typically, when analyzing multivariate time series data, feature maps are derived from each variable and then synthesized to create data for analysis. However, during the feature derivation process, features due to correlations between variables may be removed, which can lead to a problem of reduced analysis accuracy.

[0050] Accordingly, the data processing unit (130) generates an original one-dimensional matrix by organizing time series data of the past n points in time for each of the plurality of variables into an n * 1 matrix based on each point in time, and generates a combined one-dimensional matrix by combining two or more of the generated original one-dimensional matrices to configure an n * 1 matrix for each combination, and generates the feature map using the original one-dimensional matrix and the combined one-dimensional matrix. Through this, rather than simply extracting the feature map of the time series data for each variable, it extracts the feature map from the combination of two or more variables, so that it is possible to analyze even the features derived from the correlation between the variables.

[0051] When the number of the plurality of variables is m, the data processing unit (130) accumulates the original one-dimensional matrices to be combined for each of the cases of combining the generated original one-dimensional matrices from 2 to m to generate a combined two-dimensional matrix, and inputs the generated combined two-dimensional matrix into a 1*k convolution layer (k is the number of original one-dimensional matrices to be combined) to generate a combined one-dimensional matrix.

[0052] For example, if there are 5 variables in total, 5 original 1-dimensional matrices each composed of one variable, 10 combined 1-dimensional matrices generated with 2 variable combinations each, 10 combined 1-dimensional matrices generated with 3 variable combinations each, 5 combined 1-dimensional matrices generated with 4 variable combinations each, and 1 combined 1-dimensional matrix generated with 5 variable combinations are generated, resulting in a total of 31 1-dimensional matrices. If these are accumulated, a 2-dimensional matrix of n * 31 is derived as a feature map, and as many feature maps as the number of channels can be derived.

[0053] The data processing unit (130) accumulates the original one-dimensional matrices to be combined to generate a combined two-dimensional matrix, and inputs the generated combined two-dimensional matrix into a 1*k convolution layer (k is the number of original one-dimensional matrices to be combined) to generate a combined one-dimensional matrix. When combining two or more variables, the original one-dimensional matrices for each variable are accumulated to generate a two-dimensional matrix. Since the original one-dimensional matrix is ​​an n*1 matrix, when combining two variables, an n*2 two-dimensional matrix is ​​generated, and when combining three variables, an n*3 two-dimensional matrix is ​​generated.

[0054] Therefore, when a 1 * 2 convolution layer is applied to an n * 2 2-dimensional matrix created by combining two variables, an n * 1 1-dimensional matrix is ​​derived again. Also, for an n * 3 2-dimensional matrix created by combining three variables, a 1 * 3 convolution layer is applied to derive an n * 1 1-dimensional matrix.

[0055] In addition, the data processing unit (130) generates the feature map by combining the data generated by applying a 1*1 convolution layer to the original 1-dimensional matrix and the combined 1-dimensional matrix. Through this, it is possible to confirm the features according to the time series flow of each variable in the data obtained by applying a 1*1 convolution layer to the original 1-dimensional matrix, and it is possible to confirm the features derived by the correlation between variables through the combined 1-dimensional matrix data derived by the combination of variables.

[0056] The data learning unit (140) creates a time series data analysis model using multiple feature maps generated based on multiple time points. By extracting continuously input time series data at n time points and generating feature maps, multiple feature maps can be generated. For each feature map derived time point, tagging, such as occurrence of an abnormality, is performed to derive learning data composed of multiple feature maps, and by learning the feature maps, a time series data analysis model that derives analysis results for time series data is created. Since the features of image data are included in the feature map, it can be configured in the same form as when creating a model for analyzing numerical data.

[0057] The data analysis unit (150) analyzes time series data using the generated time series data analysis model. By performing learning using tagged learning data for multiple feature maps derived from past time series data, a time series data analysis model capable of detecting and predicting abnormal situations for newly input time series data is derived. Then, by dividing real-time input time series data into n time points equally and inputting them, analysis results for the current status can be derived.

[0058] FIG. 2 is a diagram illustrating an example of time series numerical data and time series image data input to a time series data analysis system according to an embodiment of the present invention.

[0059] As shown in the drawing, time series data is data that is continuously input at specific time intervals. Therefore, in order to analyze the situation at a specific time, it is necessary to extract a set of data from a set number of consecutive time points.

[0060] The drawing shows a case where input data for a total of 25 variables are input in a time series manner. By applying the same time point criterion, n consecutive time point data are derived for all 25 variables, and this is applied to the time series data analysis system of the present invention for analysis.

[0061] n, which corresponds to the size of the window in the drawing, can be set differently depending on the characteristics of the time series data, and an appropriate size can be selected to sufficiently analyze the causal relationship according to the time series flow.

[0062] By constructing a data set using n past data each time a point in time passes by 1, it is possible to construct the data in each data set so that the data overlaps, and it is also possible to construct the next data set with n data after the extracted n data. This data extraction method can be constructed differently depending on the form of the data, the results to be analyzed, etc., and the present invention is not limited thereby.

[0063] Additionally, as illustrated in the drawing, image data captured at any one point in time among n points in time that constitute a single data set can be used as input data. The point in time at which the image data is captured only needs to fall within the n points in time. As illustrated in the drawing, the image data may be captured at the most recent point in time, or may be captured at the middle or earliest point in time among the n points in time.

[0064] In some cases, multiple image data sets may be used. In this case, multiple image data sets captured within n time points can be used. When multiple image data sets are used, multiple image feature maps can be derived, combined with numerical feature maps to derive a single feature map, and analysis can be conducted based on this map.

[0065] FIG. 3 is a diagram illustrating a process of extracting feature data from time series numerical data and time series image data in a time series data analysis system according to an embodiment of the present invention.

[0066] The drawing shows the process of deriving an n * 431 feature map for a time series section n using time series numerical data for five variables and image data captured within the time series section.

[0067] First, let's look at the numerical data. As explained above, when a 1-dimensional matrix of n * 1 is input for each of the five variables a, b, c, d, and e, a 1 * 1 convolution layer is applied as shown at the top to derive five n * 1 matrices.

[0068] Afterwards, through combinations of two variables each, an n * 2 matrix is ​​derived for 10 combinations of (a, b), (a, c), (a, d), (a, e), (b, c), (b, d), (b, e), (c, d), (c, e), (d, e), and an n * 2 matrix is ​​derived for each combination by applying a 1 * 2 convolution layer.

[0069] In addition, through combinations of 3 variables each, an n * 3 matrix is ​​derived for 10 combinations of (a, b, c), (a, b, d), (a, b, e), (a, c, d), (a, c, e), (a, d, e), (b, c, d), (b, c, e), (b, d, e), (c, d, e), and an n * 3 matrix is ​​derived for each combination by applying a 1 * 3 convolution layer.

[0070] In addition, through combinations of 4 variables each, an n * 4 matrix is ​​derived for 5 combinations of (a, b, c, d), (a, b, c, e), (a, b, d, e), (a, c, d, e), (b, c, d, e), and an n * 4 matrix is ​​derived for each combination by applying a 1 * 4 convolution layer.

[0071] Finally, a 1 * 5 convolution layer is applied to the 1 * 5 matrix that combines all of (a, b, c, d, e) to derive one n * 1 matrix.

[0072] Through this, a total of 31 n * 1 matrices are derived, and these are accumulated to derive an n * 31 matrix, which is then used as a feature map.

[0073] Depending on the number of channels in the convolutional layer, the number of n * 31 feature maps derived from one data set may vary.

[0074] In addition, for image data, a preprocessing process and a flattening process are performed to derive an n * 400 image feature map. The n * 400 image feature map derived in this way is accumulated with the previously derived n * 31 numerical feature map to derive a final n * 431 feature map, and by using this, it is possible to derive analysis results that comprehensively analyze time-series numerical data and image data of various variables.

[0075] FIG. 4 is a diagram illustrating an example of extracting feature data from time series numerical data and time series image data in a time series data analysis system according to an embodiment of the present invention.

[0076] The drawing shows an actual embodiment of the process of FIG. 3. As shown in the drawing, time-series numerical data may be numerical data of various variables input according to changes in time, and an n-column numerical characteristic map is derived using this time-series numerical data.

[0077] Additionally, image data goes through a preprocessing and simplexing process to derive an image feature map, and the data obtained by combining these is used as the final feature map.

[0078] FIG. 5, FIG. 6 and FIG. 7 are diagrams showing examples of a feature extraction module, an inception module and a neural network structure using the same used in a time series data analysis system according to one embodiment of the invention.

[0079] The time series data analysis system of the present invention can use a network based on a convolutional neural network (CNN), and can utilize a feature extraction module and an inception module configured as shown in FIGS. 4 and 5, and can apply these to configure a neural network structure as shown in FIG. 6.

[0080] Such a structure is only one embodiment for applying the present invention, and the present invention is not limited by such a neural network structure.

[0081] Figure 8 is a flowchart showing the flow of a time series data analysis method according to one embodiment of the present invention.

[0082] The time series data analysis method according to one embodiment of the present invention is a method that operates in a time series data analysis system (101) equipped with a central processing unit (CPU) and memory, and the description of the time series data analysis system (101) described above can be applied as is. Therefore, even if there is no separate description below, it is self-evident that all contents described to explain the time series data analysis system can be applied as is to implementing the time series data analysis method.

[0083] The numerical data reception step (S801) receives time-series numerical data for multiple variables. Time-series numerical data is continuously generated and collected over time and can include a variety of information for monitoring industrial sites or equipment, such as factories. For example, status information, such as temperature and humidity, of a specific space may be continuously collected, or information, such as voltage and current, of power supplied to specific equipment may be time-series data.

[0084] The time series numerical data received in the numerical data receiving step (S801) may be data measured at multiple points in time. For example, if voltage and current information is measured and collected at 1-second intervals, the time series numerical data may be data at each point in time at 1-second intervals.

[0085] In the numerical data receiving step (S801), time series numerical data received can be synchronized for analysis. In cases where there is a difference in the time at which data is measured depending on the type of data, data measured at the same time can be extracted to match the time, or data can be corrected for some data to correct the data so that it is in the same form as if it were measured at the same time, thereby achieving synchronization.

[0086] The image data reception step (S802) receives image data captured in a time-series fashion. The image data may be data received from a camera module, such as a CCTV, installed to check the status of a location or equipment being monitored. The image data may also be an image captured at a specific point in time from a video received in real time, or an image captured periodically as a still image.

[0087] In order to enable analysis in relation to the time-series numerical data, the image data received in the image data receiving step (S802) can be synchronized with each point in time-series numerical data, and when extracting and analyzing the time-series numerical data by a certain time interval, one or more image data captured within the time interval can be used. This allows for simultaneous analysis of numerically derived information and image information about the scene to yield accurate analysis results.

[0088] The data processing step (S803) generates a feature map using the time series numerical data and the image data. A feature map refers to data for analysis created by extracting features from data to enable rapid analysis in an artificial intelligence model such as a convolutional neural network (CNN). Typically, a convolutional neural network uses multiple convolutional layers that extract features through filters to extract feature information that can be used for actual abnormal condition detection and prediction, and by learning using this, a necessary neural network model can be derived.

[0089] The data processing step (S803) generates an original one-dimensional matrix by organizing time series numerical data of the past n points in time into an n * 1 matrix for each of the plurality of variables based on each point in time, generates a numerical feature map based on the generated original one-dimensional matrix for each of the plurality of variables, generates an image feature map using at least one of the images captured within the n points in time among the received image data, and generates the feature map by combining the numerical feature map and the image feature map.

[0090] When the time series numerical data processed in the data processing step (S803) consists of multiple variables, the characteristics of the numerical data are derived to create a numerical characteristic map, and an image characteristic map is derived using images captured within the time range of the numerical data, thereby enabling them to be combined and analyzed.

[0091] The image data used in the data processing step (S803) can be specified as image data captured at a predetermined point among the start point, midpoint, and end point of the time interval of the numerical data, depending on the characteristics of the analysis target. This can be selected in various ways depending on the characteristics of the analysis target, and the present invention is not limited by this selection method. However, the image data must be captured within the time interval range of the numerical data, so that the characteristics of the numerical data and the characteristics of the image data can be considered simultaneously for the same time interval.

[0092] In more detail, the data processing step (S803) generates a numerical feature map having n columns using the original one-dimensional matrix, generates an image feature map having n columns based on at least one of the images captured within the n time points among the received image data, and then generates the feature map by combining the numerical feature map and the image feature map.

[0093] In order to generate an image feature map in the data processing step (S803), a preprocessing process such as resizing the original image and a simplex process can be performed, and various methods can be applied if an image feature map having the same columns as time-series numerical data can be derived. The process of deriving the image feature map can derive the feature map as a matrix of a desired size through a convolution operation using a filter depending on the structure used when analyzing the image using a convolutional neural network (CNN). The present invention is not limited by a specific method for deriving a feature map from an image.

[0094] The data processing step (S803) generates a combination one-dimensional matrix by combining two or more of the above-generated original one-dimensional matrices to form an n * 1 matrix for each combination, and generates the numerical feature map using the original one-dimensional matrix and the combination one-dimensional matrix.

[0095] Typically, when analyzing multivariate time series data, feature maps are derived from each variable and then synthesized to create data for analysis. However, during the feature derivation process, features due to correlations between variables may be removed, which can lead to a problem of reduced analysis accuracy.

[0096] Accordingly, the data processing step (S803) generates an original one-dimensional matrix by organizing time series data of the past n points in time for each of the multiple variables based on each point in time into an n * 1 matrix, and generates a combined one-dimensional matrix by combining two or more of the generated original one-dimensional matrices to configure an n * 1 matrix for each combination, and generates the feature map using the original one-dimensional matrix and the combined one-dimensional matrix. Through this, rather than simply extracting the feature map of the time series data for each variable, it is possible to extract the feature map from the combination of two or more variables, thereby enabling analysis of features derived from the correlation between the variables.

[0097] In the data processing step (S803), if the number of the above-mentioned multiple variables is m, the original one-dimensional matrices to be combined are accumulated for each of the cases of combining the generated original one-dimensional matrices from 2 to m to generate a combined two-dimensional matrix, and the generated combined two-dimensional matrix is ​​input into a 1*k convolution layer (k is the number of original one-dimensional matrices to be combined) to generate a combined one-dimensional matrix.

[0098] For example, if there are 5 variables in total, 5 original 1-dimensional matrices each composed of one variable, 10 combined 1-dimensional matrices generated with 2 variable combinations each, 10 combined 1-dimensional matrices generated with 3 variable combinations each, 5 combined 1-dimensional matrices generated with 4 variable combinations each, and 1 combined 1-dimensional matrix generated with 5 variable combinations are generated, resulting in a total of 31 1-dimensional matrices. If these are accumulated, a 2-dimensional matrix of n * 31 is derived as a feature map, and as many feature maps as the number of channels can be derived.

[0099] The data processing step (S803) generates a combined two-dimensional matrix by accumulating the original one-dimensional matrices to be combined, and inputs the generated combined two-dimensional matrix into a 1*k convolution layer (k is the number of original one-dimensional matrices to be combined) to generate a combined one-dimensional matrix. When combining two or more variables, the original one-dimensional matrices for each variable are accumulated to generate a two-dimensional matrix. Since the original one-dimensional matrix is ​​an n*1 matrix, when combining two variables, an n*2 two-dimensional matrix is ​​generated, and when combining three variables, an n*3 two-dimensional matrix is ​​generated.

[0100] Therefore, when a 1 * 2 convolution layer is applied to an n * 2 2-dimensional matrix created by combining two variables, an n * 1 1-dimensional matrix is ​​derived again. Also, for an n * 3 2-dimensional matrix created by combining three variables, a 1 * 3 convolution layer is applied to derive an n * 1 1-dimensional matrix.

[0101] In addition, the data processing step (S803) generates the feature map by combining the data generated by applying a 1*1 convolution layer to the original 1-dimensional matrix and the combined 1-dimensional matrix. Through this, it is possible to confirm the characteristics according to the time series flow of each variable in the data obtained by applying a 1*1 convolution layer to the original 1-dimensional matrix, and it is possible to confirm the characteristics derived by the correlation between variables through the combined 1-dimensional matrix data derived from the combination of variables.

[0102] The data learning step (S804) creates a time series data analysis model using multiple feature maps generated based on multiple time points. By extracting n time points from continuously input time series data and generating feature maps, multiple feature maps can be generated. For each feature map derived time point, tags such as the occurrence of an anomaly are performed to derive learning data composed of multiple feature maps. By learning this, a time series data analysis model that derives analysis results for time series data is created. Since the characteristics of image data are included in the feature map, it can be configured in the same manner as when creating a model for analyzing numerical data.

[0103] The data analysis step (S805) analyzes time series data using the generated time series data analysis model. By performing training using tagged learning data for multiple feature maps derived from past time series data, a time series data analysis model capable of detecting and predicting abnormal situations for newly input time series data is derived. Then, by dividing real-time input time series data into n time points equally and inputting them, analysis results for the current state can be derived.

[0104] The time series data analysis method according to the present invention can be produced as a program for causing a computer to execute the program and recorded on a computer-readable recording medium.

[0105] Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical recording media such as CDROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, and flash memory.

[0106] Examples of program instructions include not only machine language codes, such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules to perform processing according to the present invention, and vice versa.

[0107] Although the present invention has been described above with reference to embodiments, those skilled in the art can modify and change the present invention in various ways without departing from the spirit and scope of the present invention as set forth in the following claims.

[0108] The present invention relates to a time series data analysis system and method for analyzing time series data including images to detect abnormalities, etc., and provides a time series data analysis system and an operation method thereof, including a numerical data receiving unit for receiving time series numerical data for a plurality of variables, an image data receiving unit for receiving image data photographed in a time series manner, a data processing unit for generating a characteristic map using the time series numerical data and the image data, a data learning unit for generating a time series data analysis model using a plurality of characteristic maps generated based on a plurality of time points, and a data analysis unit for analyzing the time series data using the generated time series data analysis model.

Claims

1. A numerical data receiving unit that receives time series numerical data for multiple variables; An image data receiving unit that receives image data captured in a time series manner; A data processing unit that generates a characteristic map using the above time series numerical data and the above image data; A data learning unit that creates a time series data analysis model using multiple feature maps generated based on multiple time points. A data analysis unit that analyzes time series data using the above-generated time series data analysis model. Time series data analysis system.

2. In paragraph 1, The above data processing unit Based on each point in time, the time series numerical data of the past n points in time are organized into an n * 1 matrix for each of the above multiple variables to create an original one-dimensional matrix, Generate a numerical feature map based on the original one-dimensional matrix for each of the multiple variables generated above, Generating an image feature map using at least one of the images captured within the n time points among the received image data, Generating the feature map by combining the above numerical feature map and the image feature map. A time series data analysis system featuring .

3. In paragraph 2, The above data processing unit Using the above original one-dimensional matrix, a numerical feature map having n columns is generated, After generating an image feature map having n columns based on at least one of the images captured within the n time points among the received image data, Generating the feature map by combining the above numerical feature map and the image feature map. A time series data analysis system featuring .

4. In paragraph 3, The above data processing unit By combining two or more of the above-generated original one-dimensional matrices and constructing an n * 1 matrix for each combination, a combined one-dimensional matrix is ​​generated, Generating the numerical feature map using the original one-dimensional matrix and the combined one-dimensional matrix. A time series data analysis system featuring .

5. In paragraph 4, The above data processing unit If the number of the above multiple variables is m, For all cases of combining the above-generated original one-dimensional matrices from 2 to m, the original one-dimensional matrices to be combined are accumulated to generate a combined two-dimensional matrix, Generating a combined 1-dimensional matrix by inputting the above-generated combined 2-dimensional matrix into a 1*k convolution layer (k is the number of original 1-dimensional matrices to be combined). A time series data analysis system featuring .

6. A time series data analysis method operating in a time series data analysis system equipped with a central processing unit and memory, A numerical data receiving step for receiving time series numerical data for multiple variables; An image data receiving step for receiving image data captured in a time series manner; A data processing step of generating a feature map using the above time series numerical data and the above image data; A data learning step for creating a time series data analysis model using multiple feature maps generated based on multiple time points; and A data analysis step for analyzing time series data using the above-generated time series data analysis model. Methods for analyzing time series data.

7. In paragraph 6, The above data processing steps are Based on each point in time, the time series numerical data of the past n points in time are organized into an n * 1 matrix for each of the above multiple variables to create an original one-dimensional matrix, Generate a numerical feature map based on the original one-dimensional matrix for each of the multiple variables generated above, Generating an image feature map using at least one of the images captured within the n time points among the received image data, Generating the feature map by combining the above numerical feature map and the image feature map. A time series data analysis method characterized by .

8. In paragraph 7, The above data processing steps are Using the above original one-dimensional matrix, a numerical feature map having n columns is generated, After generating an image feature map having n columns based on at least one of the images captured within the n time points among the received image data, Generating the feature map by combining the above numerical feature map and the image feature map. A time series data analysis method characterized by .

9. In paragraph 8, The above data processing steps are By combining two or more of the above-generated original one-dimensional matrices and constructing an n * 1 matrix for each combination, a combined one-dimensional matrix is ​​generated, Generating the numerical feature map using the original one-dimensional matrix and the combined one-dimensional matrix. A time series data analysis method characterized by .

10. In paragraph 9, The above data processing steps are If the number of the above multiple variables is m, For all cases of combining the above-generated original one-dimensional matrices from 2 to m, the original one-dimensional matrices to be combined are accumulated to generate a combined two-dimensional matrix, Generating a combined 1-dimensional matrix by inputting the above-generated combined 2-dimensional matrix into a 1*k convolution layer (k is the number of original 1-dimensional matrices to be combined). A time series data analysis method characterized by .

11. A computer-readable recording medium having recorded thereon a program for causing a computer to execute the method of any one of clauses 6 to 10.

Images