Information comparison method and system based on real-time processing of big data

Abstract

The present application relates to the field of data processing technology, and in particular to an information comparison method and system based on real-time processing of big data, the method comprising: calculating the amount of information at each moment in the time series data, dividing the time series data into multiple data segments according to the amount of information; performing anomaly detection on the first information amount sequence composed of the information amount of each data segment to locate the abnormal data segment, and forming a second information amount sequence according to the information amount of the normal data segment; calculating the sequence similarity of the second information amount sequence between two time series data, if the sequence similarity is less than the first threshold, the comparison result is dissimilar, otherwise, weighted summing the sequence similarities of each normal data segment based on the information amount of the normal data segment to obtain the weighted similarity; comparing the weighted similarity with the second threshold to obtain the comparison result. Through the technical solution of the present application, the comparison result of the time series data can be obtained quickly and accurately.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to an information comparison method and system based on real-time processing of big data. Background Art

[0002] Time series data is data collected at multiple moments, usually observations arranged in chronological order. It is widely used in many fields such as finance, communications, and marketing. For example, video surveillance, stock trends, and voice calls are all time series data.

[0003] At present, the patent application document with application publication number CN114528334A discloses a fast similarity search method in a time series database, including: obtaining time series data and marking it as original data; obtaining waveform data of the original data, wherein the waveform data is obtained by a polynomial fitting method; obtaining time series data corresponding to similar waveforms from a time series database according to the waveform data, and marking it as similar data. Wherein, obtaining waveform data of time series data according to the polynomial fitting method includes: fitting the time series data by a polynomial fitting method to obtain a corresponding time series function, wherein the determination coefficient corresponding to the time series function is not less than 0.9; obtaining waveform data corresponding to the time series data according to the time series function; wherein the waveform data includes a number of moments and corresponding first-order derivative values.

[0004] The above method constructs the waveform data of the original data by randomly selecting several moments and the first-order derivative values ​​of several moments, and then judges whether any two time series data are similar based on the waveform data. However, since the observation values ​​in the time series data change continuously with time and the information expressed at different moments is different, the waveform data cannot be guaranteed to accurately reflect the information of the time series data by randomly selecting several moments, resulting in inaccurate information comparison results. Summary of the invention

[0005] In order to solve the technical problem of inaccurate information comparison results, the present application provides an information comparison method and system based on real-time processing of big data, which can quickly and accurately obtain the comparison results of time series data.

[0006] In a first aspect, the present application provides an information comparison method based on real-time processing of big data, the comparison method comprising: calculating the amount of information at each moment in the time series data, dividing the time series data into multiple data segments according to the amount of information, so that the difference in the amount of information between adjacent data segments is maximized, and the amount of information is the multi-scale entropy within the adjacent time range; performing anomaly detection on a first information amount sequence composed of the amount of information of each data segment to locate the abnormal data segment, and forming a second information amount sequence according to the amount of information of the normal data segment; calculating the sequence similarity of the second information amount sequence between any two time series data, in response to the sequence similarity being less than a first threshold, the comparison result is dissimilar, otherwise, performing weighted summation on the sequence similarities of each normal data segment based on the amount of information of the normal data segment to obtain a weighted similarity; comparing the weighted similarity and the second threshold to obtain a comparison result.

[0007] Since the observation values ​​at each moment in the time series data are different, the amount of information contained at each moment in the time series data is different. First, the amount of information at each moment in the time series data is quantified by multi-scale entropy within the adjacent time range, and the time series data is divided into multiple data segments according to the amount of information; the abnormal data segments are located according to the amount of information of different data segments to remove the noise in the time series data and obtain the second information amount sequence that can truly reflect the trend of the change in the amount of information of the time series data; the sequence similarity of the second information amount sequence between the two time series data is calculated. If the sequence similarity is less than the first threshold, the comparison result is directly obtained. Otherwise, the sequence similarities of each normal data segment are weighted and summed based on the information amount of the normal data segment to obtain the weighted similarity; the comparison result is obtained by comparing the weighted similarity and the second threshold. Since the length of the second information amount sequence is less than the length of the time series data, the calculation amount of the sequence similarity between the second information amount sequences is greatly reduced, while the efficiency of information comparison is improved, the information amount of each normal data segment is considered to obtain an accurate weighted similarity, thereby ensuring the accuracy of the information comparison result.

[0008] Preferably, the method for calculating the amount of information also includes: taking the multi-scale entropy within the adjacent time range at any moment as the first information entropy; within the adjacent time range, deleting the observation value at the moment and then calculating the multi-scale entropy again to obtain the second information entropy; and taking the difference between the first information entropy and the second information entropy as the amount of information at the moment.

[0009] For the moment The first information entropy can accurately measure the amount of information in the time series within the adjacent time range, while the second information entropy can accurately measure the deletion time. After the observation value of , the information amount of the time series in the adjacent time range; the first information entropy minus the second information entropy can be obtained The amount of information.

[0010] Preferably, dividing the time series data into a plurality of data segments according to the amount of information includes: defining the class diameter of the data segment, the data segment Class diameter for: , and Data segment The start and end times of For the moment The amount of information, For data segment The median of the amount of information at each moment in the ; the sum of the class diameters of each data segment is taken as the loss function value, and the time series data is clustered in an ordered manner to obtain multiple data segments corresponding to the minimum loss function value.

[0011] Class diameter The larger the value, the more data segment The amount of information at each moment in the time series tends to be consistent. Ordered sample clustering can continuously find the optimal segmentation point in the time series data to achieve the segmentation of the time series data; when the loss function value takes the minimum value, the corresponding division result can ensure that the amount of information at each moment in each data segment tends to be consistent, thereby ensuring that the difference in the amount of information in adjacent data segments is maximized.

[0012] Preferably, the amount of information in a data segment is equal to the average amount of information at each moment in the data segment, and the process of anomaly detection includes: respectively calculating the absolute value of the difference in the amount of information between any data segment and two adjacent data segments to obtain the change in the amount of information of the data segment; calculating the average and standard deviation of all changes in the amount of information, and constructing normal intervals and abnormal intervals based on the criterion of 3 times the standard deviation; in response to the fact that all changes in the amount of information of a data segment are in the abnormal interval, marking the data segment as an abnormal data segment, otherwise, marking the data segment as a normal data segment.

[0013] Preferably, the time series data and time series data Sequence similarity The calculation method is: calculate the DTW distance of the second information volume sequence of two time series data, the sequence similarity Negatively correlated with DTW distance.

[0014] Preferably, the time series data and time series data Sequence similarity The calculation method is: transform the time series data Pan After a moment, get the time series data and time series data The sum of the absolute values ​​of the differences of the observations at the same time in the overlapping area is calculated; under the constraints, adjust , get the minimum value of the sum of the absolute values ​​of the differences, the sequence similarity is negatively correlated with the minimum value of the sum of the absolute values ​​of the differences, and the constraint condition is that the overlapping area is not less than the time series data and time series data A preset multiple of the minimum length.

[0015] Preferably, the time series data and time series data The weighted similarity The calculation method is as follows: determine the matching pairs of data segments in two time series data; weighted similarity for: , is the number of matching pairs, and For matching pairs Medium time series data and time series data The amount of information in the data segment, and Separately, time series data and time series data The sum of the information content of the normal data segment, For matching pairs The sequence similarity between two data segments.

[0016] Since different data segments contain different amounts of information, normal data segments with larger amounts of information have a greater impact on the information comparison results. To ensure the information comparison results, the sequence similarities of normal data segments are weighted and summed according to the amount of information, so that the weighted similarity can truly reflect the similarity of the information contained in the time series data.

[0017] Preferably, the process of determining the matching pair includes: calculating the time series data Normal data segment and time series data The sequence similarity of each normal data segment in The data segment corresponding to the maximum sequence similarity in is taken as the matching result to obtain a set of matching pairs.

[0018] Preferably, the process of determining the matching pair includes: organizing the numbers of the normal data segments in the time series data into a data segment sequence; defining the data segment spacing, and using the DTW algorithm to determine the matching results of each normal data segment in the two time series data to obtain a matching pair, and the data segment spacing is negatively correlated with the sequence similarity.

[0019] Time Series Data Medium data segment and time series data The matching pairs of data segments in the time series data directly affect the calculation accuracy of the weighted similarity. Since there is a time sequence between the data segments in the time series data, the idea of ​​dynamic programming in the DTW algorithm is used to quickly and accurately determine the matching results of the data segments in the two time series data.

[0020] In a second aspect of the present application, there is also provided an information comparison system based on real-time processing of big data, comprising a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, an information comparison method based on real-time processing of big data as described in the first aspect of the present application is implemented.

[0021] The technical solution of this application has the following beneficial technical effects:

[0022] Since the observation values ​​at each moment in the time series data are different, the amount of information contained at each moment in the time series data is different. First, the amount of information at each moment in the time series data is quantified by multi-scale entropy within the adjacent time range, and the time series data is divided into multiple data segments according to the amount of information; the abnormal data segments are located according to the amount of information of different data segments to remove the noise in the time series data and obtain the second information amount sequence that can truly reflect the trend of the change in the amount of information of the time series data; the sequence similarity of the second information amount sequence between the two time series data is calculated. If the sequence similarity is less than the first threshold, the comparison result is directly obtained. Otherwise, the sequence similarities of each normal data segment are weighted and summed based on the information amount of the normal data segment to obtain the weighted similarity; the comparison result is obtained by comparing the weighted similarity and the second threshold. Since the length of the second information amount sequence is less than the length of the time series data, the calculation amount of the sequence similarity between the second information amount sequences is greatly reduced, while the efficiency of information comparison is improved, the information amount of each normal data segment is considered to obtain an accurate weighted similarity, thereby ensuring the accuracy of the information comparison result. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] By reading the detailed description below with reference to the accompanying drawings, the above and other purposes, features and advantages of the exemplary embodiments of the present application will become easy to understand. In the accompanying drawings, several embodiments of the present application are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0024] Figure 1 is a flow chart of an information comparison method based on real-time processing of big data according to an embodiment of the present application;

[0025] Figure 2 is a schematic diagram of a path in a matrix according to an embodiment of the present application;

[0026] Figure 3 It is a structural block diagram of an information comparison system based on real-time processing of big data according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0028] It should be understood that when the terms "first", "second", etc. are used in the claims, specification and drawings of the present application, they are only used to distinguish different objects, rather than to describe a specific order. The terms "include" and "comprise" used in the specification and claims of the present application indicate the presence of the described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or their collections.

[0029] According to the first aspect of the present application, the present application provides an information comparison method based on real-time processing of big data, which is used to determine whether any two time series data are similar, and can be applied in multiple scenarios such as database retrieval, similarity judgment, and data status detection. Specifically, in the database retrieval scenario, determine whether the input time series data is similar to all the time series data in the database, and display the similar time series data. In the data status detection scenario, determine whether the collected real-time time series data is similar to the standard time series data in the normal state. If similar, it means that the real-time time series data is in a normal state.

[0030] Figure 1 1 is a flow chart of an information comparison method based on real-time processing of big data according to an embodiment of the present application. Figure 1 As shown, the information comparison method based on real-time processing of big data includes steps S101 to S104, which are described in detail below.

[0031] S101, calculating the amount of information at each moment in the time series data, and dividing the time series data into multiple data segments according to the amount of information, so that the difference in the amount of information between adjacent data segments is maximized, and the amount of information is the multi-scale entropy within the adjacent time range.

[0032] In one embodiment, the time series data includes observation values ​​at each moment. For example, when the time series data is a voltage sequence, the observation value is the voltage value at each moment; when the time series data is a voice sequence, the observation value is the text code at each moment.

[0033] The observation values ​​at each moment in the time series data are constantly changing, and the amount of information contained in the observation values ​​at different moments is different. The multi-scale entropy within the adjacent time range is used as the amount of information at the corresponding moment. Among them, the adjacent time range includes the moment, and multiple moments adjacent to the moment on both sides; in the embodiment of the present application, the adjacent time range includes a total of 17 moments.

[0034] Among them, multiscale entropy (MSE) is an information entropy that expands sample entropy to multiple time scales, so as to accurately measure the complexity of a time series when the time scale is uncertain, that is, to accurately measure the amount of information contained in a time series.

[0035] In another embodiment, the method for calculating the amount of information also includes: taking the multi-scale entropy within the adjacent time range of any moment as the first information entropy; within the adjacent time range, deleting the observation value at the moment and then calculating the multi-scale entropy again to obtain the second information entropy; and taking the difference between the first information entropy and the second information entropy as the amount of information at the moment.

[0036] Among them, for the time The first information entropy can accurately measure the amount of information in the time series within the adjacent time range, while the second information entropy can accurately measure the deletion time. After the observation value of , the information amount of the time series in the adjacent time range; the first information entropy minus the second information entropy can be obtained The amount of information.

[0037] In one embodiment, after obtaining the amount of information at each moment, the time series is divided according to the amount of information at each moment to obtain multiple data segments, each data segment includes an observation value at at least one moment, and the amount of information at multiple moments in the data segment is basically consistent, and the difference in the amount of information between adjacent data segments reaches the maximum.

[0038] Specifically, dividing the time series data into multiple data segments according to the amount of information includes: defining the class diameter of the data segment, the data segment Class diameter for: , and Data segment The start and end times of For the moment The amount of information, For data segment The median of the amount of information at each moment in the ; the sum of the class diameters of each data segment is taken as the loss function value, and the time series data is clustered in an ordered manner to obtain multiple data segments corresponding to the minimum loss function value.

[0039] Among them, the class diameter The larger the value, the more data segment The amount of information at each moment tends to be consistent. Ordered sample clustering can continuously find the optimal segmentation point in the time series data to realize the segmentation of the time series data. First, the first optimal segmentation point is found in the time series data to divide the time series data into two data segments. Further, on the basis of the two data segments, the second optimal segmentation point is found to divide the time series data into three data segments. And so on, multiple data segments are obtained according to the corresponding division results when the loss function value takes the minimum value.

[0040] In this way, the time series data is divided into multiple data segments, and the difference in the amount of information between adjacent data segments is maximized. At the same time, the amount of information in each data segment can be quantified, where the amount of information in the data segment is equal to the average amount of information at each moment in the data segment.

[0041] S102, performing anomaly detection on a first information amount sequence formed by the information amount of each data segment to locate the abnormal data segment, and forming a second information amount sequence according to the information amount of the normal data segment.

[0042] In one embodiment, the information of each data segment is spliced ​​together in time order to obtain a first information sequence, which can reflect the change in the information of the time series data. In the process of collecting time series data, noise is inevitably introduced; for example, when collecting a voice sequence, the speaker may pause or repeat, resulting in the inability to collect valid information in the voice sequence; when collecting a voltage sequence, the sensor is disturbed by the external environment, which may cause a sudden change in the voltage value, thereby causing a change in the amount of information in the voltage sequence.

[0043] Specifically, the anomaly detection process includes: calculating the absolute value of the difference in information volume between any data segment and two adjacent data segments, respectively, to obtain the information volume change of the data segment; calculating the mean and standard deviation of all information volume changes, and constructing normal intervals and abnormal intervals based on the criterion of 3 times the standard deviation; in response to the information volume changes of a data segment being in the abnormal interval, marking the data segment as an abnormal data segment, otherwise, marking the data segment as a normal data segment.

[0044] Among them, according to the criterion of 3 times the standard deviation, the normal interval can be obtained as , the abnormal interval is and The union of; and are the mean and standard deviation of all information changes respectively. One data segment corresponds to at least one information change; in the first information sequence, the data segments at both ends correspond to one information change, and the other data segments correspond to two information changes.

[0045] In other embodiments, a box plot may be used to determine normal intervals and abnormal intervals, thereby locating abnormal data segments in the first information volume sequence.

[0046] In one embodiment, after deleting the abnormal data segment, the information volume of the normal data segment is arranged in chronological order to obtain a second information volume sequence, which can reflect the information volume change trend of the time series after removing the abnormal data.

[0047] S103, calculating the sequence similarity of the second information amount sequence between any two time series data. In response to the sequence similarity being less than the first threshold, the comparison result is dissimilar. Otherwise, weighted summation is performed on the sequence similarities of each normal data segment based on the information amount of the normal data segment to obtain weighted similarity.

[0048] In one embodiment, the second information amount sequence of each time series data can be obtained according to the same method. The second information amount sequence can truly reflect the information amount change trend of the time series. The sequence similarity of the second information amount sequence between any two time series data can be used to determine whether the information amount change trends of the two time series data are similar.

[0049] Specifically, time series data and time series data Sequence similarity The calculation method is: calculate the DTW distance of the second information volume sequence of two time series data, the sequence similarity Negatively correlated with DTW distance. Sequence similarity for:

[0050] , Represents the DTW distance.

[0051] In other embodiments, the time series data and time series data Sequence similarity The calculation method is: transform the time series data Pan After a moment, get the time series data and time series data The sum of the absolute values ​​of the differences of the observations at the same time in the overlapping area is calculated; under the constraints, adjust , get the minimum value of the sum of the absolute values ​​of the differences, the sequence similarity is negatively correlated with the minimum value of the sum of the absolute values ​​of the differences, and the constraint condition is that the overlapping area is not less than the time series data and time series data A preset multiple of the minimum length.

[0052] The preset multiple is 0.7. for:

[0053] , is the minimum sum of the absolute values ​​of the differences.

[0054] In response to the sequence similarity being less than the first threshold, it indicates that the information amount change trend difference between the two time series data is large, and they are not similar data. Therefore, there is no need to calculate the weighted similarity, and the comparison result can be directly obtained, which reduces the amount of calculation and improves the efficiency of information comparison. In response to the sequence similarity being not less than the first threshold, it indicates that the information amount change trend difference between the two time series data is small, and they may be similar data. Therefore, it is necessary to perform weighted summation of the sequence similarities of each normal data segment based on the information amount of the normal data segment to obtain the weighted similarity, and accurately judge whether the two time series data are similar based on the weighted similarity.

[0055] The value of the first threshold is 0.5.

[0056] In one embodiment, since different data segments contain different amounts of information, normal data segments with larger amounts of information have a greater impact on the information comparison results. To ensure the information comparison results, the sequence similarities of normal data segments are weighted and summed according to the amount of information, so that the weighted similarity can truly reflect the similarity of the information contained in the time series data. and time series data The weighted similarity The calculation method is as follows: determine the matching pairs of data segments in two time series data; weighted similarity for: , is the number of matching pairs, and For matching pairs Medium time series data and time series data The amount of information in the data segment, and Separately, time series data and time series data The sum of the information content of the normal data segment, For matching pairs The sequence similarity between two data segments.

[0057] Among them, time series data Medium data segment and time series data The matching pairs of data segments in the data directly affect the weighted similarity The calculation accuracy of a matching pair includes two normal data segments from different time series data. Specifically, the process of determining a matching pair includes: calculating the time series data Normal data segment and time series data The sequence similarity of each normal data segment in The data segment corresponding to the maximum sequence similarity in is taken as the matching result to obtain a set of matching pairs.

[0058] In this way, we get the time series data The matching results of each normal data segment in the time series data are determined Medium data segment and time series data Matching pairs of data segments in .

[0059] In another embodiment, since there is a temporal sequence between the data segments in the time series data, the sequence between the data segments can be considered in the process of determining the matching pair, and multiple normal data segments in the time series data are regarded as a new time series data, recorded as a data segment sequence. Including 5 normal data segments, the data segment sequence is recorded as , where the numerical value in the data segment sequence is the data segment number. Specifically, the process of determining the matching pair includes: forming a data segment sequence with the numbers of the normal data segments in the time series data; defining the data segment spacing, and using the DTW algorithm to determine the matching results of each normal data segment in the two time series data to obtain a matching pair, and the data segment spacing is negatively correlated with the sequence similarity.

[0060] Among them, according to the sequence similarity From the calculation formula, we can know that the data segment spacing is .

[0061] Among them, the DTW algorithm is a dynamic time warping algorithm, which uses dynamic programming methods to extend and shorten time series, align two time series, and calculate the similarity of the two time series based on the aligned time series.

[0062] For example, the time series data Includes 5 normal data segments, time series data Including 6 normal data segments, a 5×6 matrix is ​​first constructed. Using the DTW algorithm, a path can be obtained in the matrix, and a point on the path corresponds to a set of matching pairs. Figure 2 is a schematic diagram of a path in a matrix according to an embodiment of the present application, the path includes , , , , , and There are 7 points in total, and 7 matching pairs are obtained; among them, point The corresponding matching pair is time series data Normal data segment 5 and timing data in Normal data segment 4 in.

[0063] S104, comparing the weighted similarity with the second threshold to obtain a comparison result.

[0064] In one embodiment, the comparison result is similar or dissimilar, and in response to the weighted similarity being less than a second threshold, the comparison result is dissimilar, otherwise, the comparison result is similar. The second threshold is set to 0.5.

[0065] The above describes the technical principle and implementation details of the information comparison method based on real-time processing of big data of the present application through specific embodiments. Due to the different observation values ​​at each moment in the time series data, the amount of information contained at each moment in the time series data is different. First, the amount of information at each moment in the time series data is quantified by multi-scale entropy within the adjacent time range, and the time series data is divided into multiple data segments according to the amount of information; the abnormal data segments are located according to the amount of information of different data segments to remove the noise in the time series data, and the second information amount sequence that can truly reflect the trend of the change of the amount of information in the time series data is obtained; the sequence similarity of the second information amount sequence between the two time series data is calculated. If the sequence similarity is less than the first threshold, the comparison result is directly obtained. Otherwise, the sequence similarity of each normal data segment is weighted and summed based on the amount of information of the normal data segment to obtain the weighted similarity; the comparison result is obtained by comparing the weighted similarity and the second threshold. Since the length of the second information amount sequence is less than the length of the time series data, the calculation amount of the sequence similarity between the second information amount sequences is greatly reduced, and the efficiency of information comparison is improved. At the same time, the information amount of each normal data segment is considered to obtain an accurate weighted similarity, thereby ensuring the accuracy of the information comparison result.

[0066] According to the second aspect of the present application, the present application also provides an information comparison system based on real-time processing of big data. Figure 3 is a structural block diagram of an information comparison system based on real-time processing of big data according to an embodiment of the present application. Figure 3 As shown, the system 50 includes a processor and a memory, the memory stores computer program instructions, and when the computer program instructions are executed by the processor, an information comparison method based on real-time processing of big data according to the first aspect of the present application is implemented. The system also includes other components well known to those skilled in the art, such as a communication bus and a communication interface, whose settings and functions are known in the art, so they are not described here.

[0067] In the present application, the aforementioned memory may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. For example, a computer-readable storage medium may be any suitable magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory RRAM (Resistive Random Access Memory), a dynamic random access memory DRAM (Dynamic Random Access Memory), a static random access memory SRAM (Static Random-Access Memory), an enhanced dynamic random access memory EDRAM (Enhanced Dynamic Random Access Memory), a high-bandwidth memory HBM (High-Bandwidth Memory), a hybrid memory cube HMC (Hybrid Memory Cube), etc., or any other medium that can be used to store the required information and can be accessed by an application, a module, or both. Any such computer storage medium may be part of a device or accessible or connectable to a device. Any application or module described in this application may be implemented using computer-readable / executable instructions that may be stored or otherwise maintained by such a computer-readable medium.

[0068] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0069] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the patent application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent application shall be subject to the attached claims.

Claims

1. An information comparison method based on real-time processing of big data, characterized in that: The comparison method comprises: Calculate the amount of information at each moment in the time series data, where the time series data includes observation values ​​at each moment, and when the time series data is a speech sequence, the observation values ​​are text codes at each moment; Divide the time series data into multiple data segments according to the amount of information so that the difference in the amount of information between adjacent data segments is maximized, and the amount of information is the multi-scale entropy within the adjacent time range; Performing anomaly detection on a first information quantity sequence formed by the information quantity of each data segment to locate the abnormal data segment, and forming a second information quantity sequence according to the information quantity of the normal data segment; Calculate the sequence similarity of the second information amount sequence between any two time series data. In response to the sequence similarity being less than the first threshold, the comparison result is dissimilar. Otherwise, perform weighted summation on the sequence similarities of each normal data segment based on the information amount of the normal data segment to obtain weighted similarity. Comparing the weighted similarity with the second threshold to obtain a comparison result; Time Series Data and time series data The weighted similarity The calculation method is: Determine matching pairs of data segments in two time series data; weighted similarity for: , is the number of matching pairs, and For matching pairs Medium time series data and time series data The amount of information in the data segment, and Separately, time series data and time series data The sum of the information content of the normal data segment, For matching pairs The sequence similarity of the two data segments in ; The process of determining matching pairs includes: Computing time series data Normal data segment and time series data The sequence similarity of each normal data segment in The data segment corresponding to the maximum sequence similarity is taken as the matching result to obtain a set of matching pairs; Dividing time series data into multiple data segments based on the amount of information includes: Defines the class diameter of the data segment, the data segment Class diameter for: , and Data segment The start and end times of For the moment The amount of information, For data segment The median of the amount of information at each moment; Taking the sum of the class diameters of each data segment as the loss function value, the time series data is clustered in an ordered manner to obtain multiple data segments corresponding to the minimum loss function value; Time Series Data and time series data Sequence similarity The calculation method is: Time series data Pan After a moment, get the time series data and time series data The sum of the absolute values ​​of the differences between the observations at the same time in the overlapping area is calculated; Adjusting under constraints , get the minimum value of the sum of the absolute values ​​of the differences, the sequence similarity is negatively correlated with the minimum value of the sum of the absolute values ​​of the differences, and the constraint condition is that the overlapping area is not less than the time series data and time series data A preset multiple of the minimum length.

2. The information comparison method based on real-time processing of big data according to claim 1 is characterized in that: The method for calculating the amount of information also includes: taking the multi-scale entropy within the adjacent time range of any moment as the first information entropy; within the adjacent time range, deleting the observation value at the moment and then calculating the multi-scale entropy again to obtain the second information entropy; and taking the difference between the first information entropy and the second information entropy as the amount of information at the moment.

3. The information comparison method based on real-time processing of big data according to claim 1 is characterized in that: The amount of information in a data segment is equal to the average amount of information at each moment in the data segment. The anomaly detection process includes: Calculate the absolute value of the information amount difference between any data segment and two adjacent data segments respectively to obtain the information amount change of the data segment; Calculate the mean and standard deviation of all information changes, and construct normal and abnormal intervals based on the 3-times standard deviation criterion; In response to changes in the amount of information of a data segment being in an abnormal range, the data segment is marked as an abnormal data segment; otherwise, the data segment is marked as a normal data segment.

4. The information comparison method based on real-time processing of big data according to claim 1 is characterized in that: The process of determining matching pairs includes: The numbers of normal data segments in time series data are combined into a data segment sequence; the data segment spacing is defined, and the DTW algorithm is used to determine the matching results of each normal data segment in two time series data to obtain matching pairs. The data segment spacing is negatively correlated with the sequence similarity.

5. An information comparison system based on real-time processing of big data, characterized in that: It comprises a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, an information comparison method based on real-time processing of big data according to any one of claims 1 to 4 is implemented.

Images