Database native time sequence clustering method and device

Abstract

The invention provides a database native time sequence clustering method and device, and the method comprises the steps: carrying out the initialization sorting of all pages of a database for storing native time sequences, and enabling the native time sequences to comprise subsequences with the same length; an approximate clustering result of each page is obtained, the approximate clustering results are stored through metadata, and the approximate clustering result of each page comprises r approximate clustering centers, clustering weights corresponding to the approximate clustering centers and the average distance between subsequences in clusters; using the approximate clustering result of the first page of the database as an initial aggregation result, performing page-by-page aggregation on the next page from the first page of the database, and updating the approximate clustering result of the aggregated pages through metadata; and after all pages are aggregated, performing global clustering on the final aggregated data, and accelerating the global clustering process based on the updated approximate clustering result. Complete matrix information does not need to be stored, so that the storage overhead is greatly reduced, and the complexity of input and output operations is also reduced.

Description

Technical Field

[0001] The present invention relates to the field of database storage technology, and in particular to a database native time series clustering method and device. Background Art

[0002] With the rapid development of Internet of Things (IoT) technology, a large number of sensors and devices have generated massive amounts of information time series data containing rich patterns. These data can be used to mine subsequence patterns in different time periods through cluster analysis, providing support for downstream applications such as anomaly detection, pattern recognition, and prediction. In this context, time series clustering has become an important task in data analysis, among which advanced methods can efficiently cluster according to the shape of time series and are applicable to a variety of IoT scenarios.

[0003] However, there are many special challenges in the IoT scenario, making time series clustering more complicated. First, due to the large number of devices in the IoT and the frequent data flow, time series data is usually stored in the database in a disordered manner. This disordered storage is mostly caused by the LSM tree structure supporting high-intensity writing. Second, time series clustering often requires repeated analysis for different time ranges or time filtering conditions. The frequent computing requirements put higher requirements on the efficiency of existing methods.

[0004] Existing time series clustering methods mainly operate in an out-of-database form, that is, all time series data is loaded into memory before clustering. This method requires data to be sorted in chronological order, which consumes a lot of input and output and preprocessing time, and cannot support incremental calculations. When dealing with frequent queries or clustering over a long period of time in the Internet of Things, existing methods need to recalculate from scratch each time, which significantly reduces efficiency. Summary of the invention

[0005] The present invention provides a database native time series clustering method and device, which are used to solve the defects of the time series clustering algorithm in the prior art, such as high computational complexity, high time cost and low efficiency when processing large-scale data, realize an efficient time series clustering method in the database, and adapt to the database storage characteristics.

[0006] The present invention provides a database native time series clustering method, comprising the following steps.

[0007] Initializing and sorting each page of a database storing a native time series, wherein the native time series includes subsequences of the same length; Obtaining approximate clustering results of each of the pages, and storing the approximate clustering results through metadata, wherein the approximate clustering results of each page include r approximate clustering centers, clustering weights corresponding to each approximate clustering center, and average distances between subsequences in each cluster; Using the approximate clustering result of the first page of the database as the initial aggregation result, starting from the first page of the database, the subsequent pages are aggregated page by page, and the approximate clustering results of the aggregated pages are updated through metadata; When all pages are aggregated, the final aggregated data of each page is globally clustered, and the global clustering process is accelerated based on the updated approximate clustering results.

[0008] According to a database native time series clustering method provided by the present invention, the obtaining of approximate clustering results of each of the pages specifically includes: Define the length of each of the pages as l A subsequence of is a complete subsequence, and its length is less than l The subsequence of is an incomplete subsequence; For the complete subsequences in each of the pages, a basic clustering algorithm is used to obtain r approximate cluster centers, clustering weights corresponding to each approximate cluster center, and an average distance between subsequences in each cluster, and the r approximate cluster centers, clustering weights corresponding to each approximate cluster center, and an average distance between data in each cluster are stored through metadata, wherein the basic clustering algorithms include K-Means, K-Shape, and LSH-based; For the incomplete subsequence in each of the pages, the incomplete subsequence is stored via metadata.

[0009] According to a database native time series clustering method provided by the present invention, the aggregating the subsequent pages page by page specifically includes: Determining a page relationship between the first page and the second page, wherein the page relationship includes adjacent pages, complementary pages, and overlapping pages; According to the judgment result of the page relationship, based on the page aggregation strategy, the data of the second page is aggregated to the first page; According to the initial sorting result of the pages, the page relationship between the current aggregated page and the next page is continuously determined, and according to the determination result of the page relationship and based on the page aggregation strategy, the data of the next page is aggregated into the current aggregated page.

[0010] According to a database native time series clustering method provided by the present invention, the aggregated pages in the database are set as , the page to be aggregated is , The approximate cluster center of , clustering weight and the average distance of data within the cluster , j = 1, 2…r; The page aggregation strategy specifically includes: If the page , If the page is adjacent, With page Direct merger; If the page , If it is a complementary page, the page , Extract complementary subsequences from the generated new page , and in a new page Create new cluster centers in If the page , For overlapping pages, use page Overlapping subsequence pairs in the The shape distance and center point position of the overlapping subsequences are updated.

[0011] According to a database native time series clustering method provided by the present invention, if the page , If the page is adjacent, With page Direct merger, including: For each page to be aggregated The cluster centers in , aggregate results : ; like, ,but: ; otherwise: ; Among them, the ones marked with an asterisk The variables are all the current optimal values; ++r is the class of the current page, and a new class is generated in the aggregation result; Update cluster weights The average distance to the data within the cluster .

[0012] According to a database native time series clustering method provided by the present invention, if the page , If it is a complementary page, the page , Extract complementary subsequences from the generated new page , and in a new page Create new cluster centers in , including: Generate a new loading page , including the pages to be aggregated and aggregated pages Complementary data, among which, Include new approximate cluster centers: ; merge and The aggregation result of initialization The clustering weight and the distance of the data within the cluster are: , .

[0013] According to a database native time series clustering method provided by the present invention, if the page , For overlapping pages, use page Overlapping subsequence pairs in the The shape distance and center point position of the overlapping subsequences in the update are updated, including: set up Display page Middle The approximate cluster centers of the clusters, Page The approximate cluster centers of clusters are updated as , the update of the approximate cluster center is as follows: ; .

[0014] According to a database native time series clustering method provided by the present invention, the final aggregated data of each page is globally clustered, and the global clustering process is accelerated based on the updated approximate clustering result, which specifically includes: Based on each subsequence in the final aggregation result, an objective function is constructed with the goal of maximizing the sum of shape similarities between each subsequence and the cluster center to be predicted, wherein the cluster center to be predicted is the subsequence in the final aggregation result; Based on a greedy strategy, the objective function is approximately solved, and the solution process is accelerated by the updated approximate cluster centers to determine the approximate optimal cluster center combination of the native time series.

[0015] According to a database native time series clustering method provided by the present invention, the constructing of the objective function specifically includes: Defining a subsequence With cluster center The shape similarity function is The calculation formula is: ; in, represents the weight, represents the i-th subsequence; is the jth cluster center; Defining functions is the clustering effect of the current cluster center set, and the calculation formula is: ; in, is the length of the time series T to be clustered l The subsequence set of , C is the cluster center set; Define the objective function The calculation formula is: .

[0016] According to a database native time series clustering method provided by the present invention, based on a greedy strategy, the objective function is approximately solved, and the solution process is accelerated by the updated approximate cluster center, specifically including: Based on the greedy strategy, the objective function Perform an approximate solution; Set validity bounds to estimate subsequences and cluster centers The shape distance between them, the effective limit is expressed as: ; in, is the updated approximate cluster center; The solution process is accelerated by the effective bound.

[0017] The present invention also provides a database native time series clustering device, comprising the following modules: An initialization module, used to initialize and sort each page of a database storing a native time series, wherein the native time series includes subsequences of the same length; An acquisition module, used to acquire the approximate clustering results of each of the pages, and store the approximate clustering results through metadata, wherein the approximate clustering results of each page include r approximate clustering centers, clustering weights corresponding to each approximate clustering center, and average distances between subsequences in each cluster; A page-by-page aggregation module is used to use the approximate clustering result of the first page of the database as the initial aggregation result, starting from the first page of the database, and aggregating the subsequent pages page by page, and updating the approximate clustering results of the aggregated pages through metadata; The clustering module is used to perform global clustering on the final aggregated data of each page after all pages are aggregated, and to speed up the global clustering process based on the updated approximate clustering results.

[0018] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the database native time series clustering method described above is implemented.

[0019] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the database-native time series clustering method described in any one of the above is implemented.

[0020] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned database native time series clustering methods.

[0021] The database native time series clustering method and device provided by the present invention initialize and sort each page of the database storing the native time series, wherein the native time series includes subsequences of the same length; obtain the approximate clustering results of each page, and store the approximate clustering results through metadata, wherein the approximate clustering results of each page include r approximate clustering centers, clustering weights corresponding to each approximate clustering center, and the average distance between subsequences in each cluster; use the approximate clustering results of the first page of the database as the initial aggregation results, start from the first page of the database, aggregate the subsequent pages page by page, and update the approximate clustering results of the aggregated pages through metadata; when all pages are aggregated, globally cluster the final aggregated data, and speed up the global clustering process based on the updated approximate clustering results, without storing the complete matrix information, which not only greatly reduces the storage overhead, but also reduces the complexity of input and output operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 It is a flow chart of the database native time series clustering method provided by the present invention.

[0024] Figure 2 It is a workflow diagram of K-Shape in the database provided by the present invention.

[0025] Figure 3 This is the aggregation process of adjacent pages provided by the present invention.

[0026] Figure 4 It is the execution program of the K-Shape method in the database provided by the present invention.

[0027] Figure 5 This is the accelerated approximate calculation process of the objective function provided by the present invention.

[0028] Figure 6 It is the execution program of the greedy algorithm provided by the present invention.

[0029] Figure 7 It is the execution program of the acceleration algorithm provided by the present invention.

[0030] Figure 8 It is an execution program of the clustering method adapted in the database provided by the present invention.

[0031] Fig. 9 It is a structural schematic diagram of the database native time series clustering device provided by the present invention.

[0032] Fig.10 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] The present invention is described in detail below in conjunction with the accompanying drawings of the specification. The specific operating method in the method embodiment can also be applied to the device embodiment or the system embodiment. In the description of the present invention, unless otherwise specified, "at least one" includes one or more. "Multiple" refers to two or more. For example, at least one of A, B and C includes: A exists alone, B exists alone, A and B exist at the same time, A and C exist at the same time, B and C exist at the same time, and A, B and C exist at the same time. In the present invention, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a kind of association relationship that describes the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0035] In some specific embodiments of the present invention, Figure 1 As shown, this solution provides a database native time series clustering method, including: Step 100: Initialize and sort each page of a database storing a native time series, wherein the native time series includes subsequences of the same length; Step 200: Obtain approximate clustering results of each of the pages, and store the approximate clustering results through metadata, wherein the approximate clustering results of each page include r approximate clustering centers, clustering weights corresponding to each approximate clustering center, and average distances between subsequences in each cluster; Step 300: using the approximate clustering result of the first page of the database as the initial aggregation result, starting from the first page of the database, aggregating subsequent pages page by page, and updating the approximate clustering result of the aggregated pages through metadata; Step 400: After all pages are aggregated, globally cluster the final aggregated data of each page, and speed up the global clustering process based on the updated approximate clustering results.

[0036] It should be noted that the existing time series clustering solutions cannot be executed within the database and require the data to be sorted in chronological order, which consumes a lot of input, output and preprocessing time and cannot support incremental computing. When dealing with frequent queries or long-term clustering in the Internet of Things, the existing methods need to recalculate from scratch each time, which significantly reduces efficiency.

[0037] Therefore, the present invention reduces storage overhead and I / O operation complexity by storing approximate cluster centers, cluster weights corresponding to each approximate cluster center, and the average distance between subsequences in each cluster in each page; aggregates the data of each page page by page according to the metadata aggregation strategy between different pages, combines the database storage characteristics, and utilizes block structure, page structure and metadata support to improve clustering performance.

[0038] In some possible implementations of the present invention, obtaining the approximate clustering result of each of the pages specifically includes: Define the length of each of the pages as l A subsequence of is a complete subsequence, and its length is less than l The subsequence of is an incomplete subsequence; For the complete subsequences in each of the pages, a basic clustering algorithm is used to obtain r approximate cluster centers, clustering weights corresponding to each approximate cluster center, and an average distance between subsequences in each cluster, and the r approximate cluster centers, clustering weights corresponding to each approximate cluster center, and an average distance between data in each cluster are stored through metadata, wherein the basic clustering algorithms include K-Means, K-Shape, and LSH-based; For the incomplete subsequence in each of the pages, the incomplete subsequence is stored via metadata.

[0039] Specifically, this embodiment provides an implementation method for obtaining approximate clustering results of each of the pages. For complete subsequences in the page, the approximate clustering results are stored through metadata. For incomplete subsequences, the incomplete subsequences are stored through metadata to facilitate subsequent processing of the incomplete subsequences.

[0040] It can be understood that the number r of approximate cluster centers should be much larger than the target number k of the native time series.

[0041] In a possible embodiment, the data in each page may be approximately clustered by using K-Means, K-Shape or LSH-based methods to obtain an approximate clustering result.

[0042] For example, Figure 2 As shown in the figure, the K-Shape clustering method is a cutting-edge data clustering method that clusters subsequences by identifying the similarity at the shape level of data subsequences.

[0043] Specifically, for shape similarity: In K-Shape, the shape distance between two time series X and Y is defined as: (1) in, represents the cross-correlation operation, represents the norm of X.

[0044] For clustering objectives: The goal of the K-Shape clustering problem is to find the most appropriate set of cluster centers by maximizing shape similarity. , the specific formula is: (2) In a possible embodiment, the database adopts an LSM-Tree structure to store time series data in batches as unmodifiable files (called pages). The data in the page is sorted in chronological order, but timestamp conflicts or overlaps across pages may occur.

[0045] The definitions of subsequences and incomplete subsequences in the database are as follows: Given a time series and subsequence length ,Will Split into several subsequences : (3) where pt is the time of the time series, is the time interval, Is a non-negative integer.

[0046] like , it is called a complete subsequence, otherwise it is an incomplete subsequence.

[0047] In a possible embodiment, for a complete subsequence in a single page, a K-Shape algorithm is used to iteratively generate k cluster medoids, and all cluster medoids are In addition, for all members within each cluster, the sum matrix of the matrix products of each subsequence and its transposed Stored as metadata, the formula is as follows: (4) At the same time, the average shape distance within each cluster needs to be calculated and stores it as metadata, along with the values ​​of up to two incomplete subsequences.

[0048] In some possible implementations of the present invention, aggregating the subsequent pages page by page specifically includes: Determining a page relationship between the first page and the second page, wherein the page relationship includes adjacent pages, complementary pages, and overlapping pages; According to the judgment result of the page relationship, based on the page aggregation strategy, the data of the second page is aggregated to the first page; According to the initial sorting result of the pages, the page relationship between the current aggregated page and the next page is continuously determined, and according to the determination result of the page relationship and based on the page aggregation strategy, the data of the next page is aggregated into the current aggregated page.

[0049] Specifically, this embodiment provides an implementation method for aggregating the subsequent pages page by page, by judging the page relationship between the previous and next pages, and determining a specific page aggregation method according to the judgment result.

[0050] Specifically, there is a page and chunk structure in the database. Simply put, a time series data is split into different pages and stored. In order to aggregate the pre-calculated metadata of multiple pages, all pages are sorted according to the start time and merged one by one. Figure 2 As shown in (b), when merging a new page into the current aggregation result, three situations of pages may occur: completely adjacent (ideal situation), complementary, and overlapping situations.

[0051] In some possible implementations of the present invention, the aggregated pages in the database are set to , the page to be aggregated is , The approximate cluster center of , clustering weight and the average distance of data within the cluster , j = 1, 2…r; The page aggregation strategy specifically includes: If the page , If the page is adjacent, With page Direct merger; If the page , If it is a complementary page, the page , Extract complementary subsequences from the generated new page , and in a new page Create new cluster centers in If the page , For overlapping pages, use page Overlapping subsequence pairs in the The shape distance and center point position of the overlapping subsequences are updated.

[0052] Specifically, this embodiment provides an implementation method of a page aggregation strategy, which performs specific processing on pages according to the relationship between previous and next pages.

[0053] In a possible embodiment, if the page , If the page is adjacent, With page Direct merger, including: For each page to be aggregated The cluster centers in , aggregate results : (5) like, ,but: (6) otherwise: (7) Update cluster weights The average distance to the data within the cluster .

[0054] In the above embodiment, ++r is to generate a new class in the aggregation result for the class of the current page. j is a value to be calculated, and a nearest cluster center needs to be calculated each time; k represents the initial set number of clusters. If the distance is too large, the number of clusters is expanded and the center of the current page is assigned to the new cluster. The variables are all the current optimal values. Represents the average distance between sequences within a cluster.

[0055] In a possible embodiment, if the page , If it is a complementary page, the page , Extract complementary subsequences from the generated new page , and in a new page Create new cluster centers in , including: Generate a new loading page , including the pages to be aggregated and aggregated pages Complementary data, among which, Include new approximate cluster centers: (8) merge and The aggregation result of initialization The clustering weight and the distance of the data within the cluster are: (9), (10).

[0056] In a possible embodiment, if the page , For overlapping pages, use page Overlapping subsequence pairs in the The shape distance and center point position of the overlapping subsequences in the update are updated, including: set up Display page Middle The approximate cluster centers of the clusters, Page The approximate cluster centers of clusters are updated as , the update of the approximate cluster center is as follows: (11); (12).

[0057] Specifically, adjacent pages and The conditions that must be met for adjacent pages are: (13) like Figure 2 (a) and are adjacent pages, because their timestamps are consecutive in chronological order, and The end and The beginning of each is a complete subsequence.

[0058] The cluster centers of adjacent pages can be directly updated by updating the corresponding sum matrix To merge into the current clustering results, for the following pages Each cluster center of , if there are cluster centers that are close enough in the current aggregation result , directly merge them, otherwise let it become a new cluster center. Figure 3 As shown: (14) like ,but (15) otherwise: (16) Complementary pages and The conditions for being a complementary page are: (17) Figure 2 Page in (a) and It's a complementary page.

[0059] Generate new subsequences by targeting complementary parts , and contains a new cluster center Sum Matrix , you can add the page Merge into clustering results in order Overlap pages and The conditions for overlapping pages are: (18) In this case, the data with the same timestamp in the two pages needs to be updated, and the subsequence of the overlapping area is updated by considering the conflict point. Then the remaining part becomes a neighboring page and can be aggregated directly in the same way as neighboring pages. Updated cluster centers Sum Matrix The calculation formula is as follows: (19) (20) in .

[0060] In a possible embodiment, the above page aggregation strategy may be implemented through an algorithm.

[0061] Specifically, the pseudo code of the K-Shape method in the database is as follows Figure 4 As shown in the algorithm, proposition2, proposition3 and lemma1 are the above aggregation strategies. Initialize the aggregation results, then load the next page each time And merge with the current aggregation result. For adjacent pages , can be directly merged into the current aggregation result through the above strategy; for complementary pages , Mr. ,as well as , ,and , and then merge in order and ; For overlapping pages , update the conflicting area through the above strategy and extract the final cluster center. Finally, if all pages are aggregated, the k final cluster centers still need to ensure the number by merging close cluster center points.

[0062] The specific steps are: Step 1: Sort all pages by startTime Step 2: Then for each and Determine its page type, and then update the cluster center and sum matrix according to the above aggregation strategy Step 3: Finally merge the close cluster centers to ensure the number Step 4: Return clustering information.

[0063] In a possible embodiment, for complexity analysis, the aggregation time complexity of adjacent or complementary pages is , the time complexity of overlapping pages is at most . Let the total number of pages be N, the number of overlapping pages be M, and the average overlapping length be , then the total time complexity of the K-Shape method in the database is: (twenty one) In some possible implementations of the present invention, the final aggregated data of each page is globally clustered, and the global clustering process is accelerated based on the updated approximate clustering result, specifically including: Based on each subsequence in the final aggregation result, an objective function is constructed with the goal of maximizing the sum of shape similarities between each subsequence and the cluster center to be predicted, wherein the cluster center to be predicted is the subsequence in the final aggregation result; Based on a greedy strategy, the objective function is approximately solved, and the solution process is accelerated by the updated approximate cluster centers to determine the approximate optimal cluster center combination of the native time series.

[0064] Specifically, this embodiment provides an implementation method for globally clustering the final aggregated data and speeding up the clustering process, and clustering the final aggregation results through a clustering method based on center points and shapes adapted to the database.

[0065] Specifically, the method based on the center point and shape (Medoid-Shape) is first introduced in detail, and then on this basis, the method based on the center point and shape adapted in the database is explained.

[0066] When the subsequence length When it is large, the K-Shape method in the database requires space to store the matrix, and time to extract the shape from the matrix. Therefore, the storage and computational overhead of the K-Shape method are too high when processing long subsequences. To solve this problem, a clustering method based on center points and shapes (Medoid-Shape) is designed, which selects a subset of all subsequences as cluster centers.

[0067] In some possible implementations of the present invention, constructing the objective function specifically includes: Defining a subsequence With cluster center The shape similarity function is The calculation formula is: (twenty two), in, represents the weight, represents the i-th subsequence; is the jth cluster center; Defining functions is the clustering effect of the current cluster center set, and the calculation formula is: (twenty three), in, The length of the original time series T is l The subsequence set of , C is the cluster center set; Define the objective function The calculation formula is: (twenty four).

[0068] Specifically, the Medoid-Shape clustering goal is to find a subset , to maximize the following objective function formula (24).

[0069] Different from K-Shape, Medoid-Shape does not need to calculate any center point through matrix decomposition, but directly selects the actual center point.

[0070] Furthermore, in order to efficiently solve the Medoid-Shape clustering problem, the sub-model characteristics of the objective function are used to adopt a greedy algorithm for approximate optimization. The greedy strategy gradually constructs the center point set C by iteratively selecting the subsequence that can maximize the gain of the objective function.

[0071] In a possible embodiment, a shape similarity function is defined as formula (22).

[0072] Furthermore, the objective function represents the clustering effect of the current center point set C, as shown in formula (23).

[0073] Through the above definition, the designed greedy algorithm gradually selects the one that can maximize The subsequence of , finally get the approximately optimal set of cluster center points. For the approximate calculation of the objective function, such as Figure 5 The pseudo code for this part is as follows Figure 6 As shown, the Medoid-Shape method under the greedy strategy is demonstrated.

[0074] In some possible implementations of the present invention, the objective function is approximately solved based on a greedy strategy, and the solution process is accelerated by updating the approximate cluster center, specifically including: Based on the greedy strategy, the objective function Perform an approximate solution; Set validity bounds to estimate subsequences and cluster centers The shape distance between them, the effective limit is expressed as: (25) in, is the updated approximate cluster center; The solution process is accelerated by the effective bound.

[0075] Specifically, in order to further reduce the time complexity, the objective function is proposed Intuitively, any two time periods with similar shapes have similar properties in terms of shape distance. Based on this, a limit is introduced to effectively estimate the subsequence and cluster centers The shape distance between them is shown in formula (25).

[0076] This bound has been proven to hold, and the formula has been used to significantly reduce the time required for actual calculations. Figure 5 By visualizing the subsequence and The similarity between them shows how to use the approximate center point and formula to optimize the objective function The pseudo code for this part is as follows Figure 7 As shown, the algorithm is used to replace Figure 6 The sixth line of the algorithm accelerates the algorithm flow.

[0077] The Medoid-Shape process is divided into two steps: All subsequences Clustering into several approximate clusters , select representative points for each cluster ; Based on representative points And the current center point set C quickly updates the objective function value , and finally get an approximate set of center points.

[0078] For complexity analysis, after using the approximate method, the time complexity is reduced from Reduced to .in is the number of cluster centers, The total length of the time series is is the subsequence length, is the calculation time of approximate clustering.

[0079] On the basis of the above embodiments, a method based on center points and shapes for adaptation within a database is obtained.

[0080] Specifically, unlike the metadata content of a single page in K-Shape, in the embodiment of the present invention, the specific cluster center results of K-Shape are not stored as metadata, but approximate clustering is used on each page to store the approximate cluster center points. , the weight of clustering , and the distance between the average intra-cluster data The present invention no longer needs to store matrices, so the space cost of the Medoid-Shape adapted in the database is greatly reduced compared to the K-Shape adapted in the database.

[0081] First, in this method, the metadata aggregation strategy is as follows: For adjacent pages: Similar to the strategy in K-Shape, when aggregating adjacent pages, cluster centers that are close enough can be directly merged, or a new center can be generated.

[0082] When there is already a clustering result (from aggregated), and an approximate center point And the cluster weight (j=1,2,…,r), if is an adjacent page, then: For each Center of , can have an aggregation result : if , otherwise: Then we can further update the weights and the average distance .

[0083] For complementary pages: Similar to K-Shape, when facing a complementary page, you can also generate a new page Then merge and The current aggregation results: Include only new approximate center points: and The weights and distances are initialized as: For overlapping pages: You need to update the overlapping information, and then consider the overlapping pages as adjacent pages.

[0084] If you use Indicates an updated one, belonging to The subsequence of the current aggregation result will be updated as (belong Page The approximate cluster center point is updated as follows: In a possible embodiment, the above method can be summarized into the following four steps: Step 1: Initialize and start the aggregation results from the first page; Step 2: Load page by page and merge it with the aggregation result. Consider the three types of pages according to the above strategy. Step 3: Update metadata and use weighted average to update the approximate center point and the average distance within the cluster ; Step 4: Final merging, using the greedy algorithm to determine the final cluster center.

[0085] In a possible embodiment, the above method can be implemented by a specific algorithm, such as Figure 8 As shown, the complete pseudo code of the In Database Medoid-Shape clustering method adapted in the database is shown, including the above four steps.

[0086] Analysis of time complexity: The complexity of the merge level is as follows: Adjacent and complementary: ,in represents the approximate number of clusters, Indicates the subsequence length; overlapping: ,in is the average length of the overlap; Greedy selection of the center point: ,in is the number of approximate central points; Total complexity (for the case of N total pages, M overlapping pages): .

[0087] The database native time series clustering method provided by the present invention adopts a block structure and a page structure when storing data in the database, and provides corresponding metadata support. By making full use of the characteristics of these structures and an efficient aggregation strategy, the existing K-Shape method with excellent clustering effect but high time cost is firstly modified to adapt to the storage characteristics of the database, thereby significantly improving the clustering performance; at the same time, a clustering method based on center point-shape is designed to solve the efficiency bottleneck of the existing method in long time series processing; further, the clustering method based on center point-shape is adapted and optimized in the database, so that it can quickly complete the clustering operation through the metadata aggregation strategy; finally, by deploying it in the open source time series database Apache IoTDB, the efficiency problem of the existing time series clustering method in the database is solved, and the storage characteristics of the database are better adapted.

[0088] Specifically, on the first aspect, by utilizing the metadata stored in the database, the K-Shape method is improved and optimized, and the K-Shape method is implemented in the database. In each data page, metadata of multiple complete subsequences and up to two incomplete subsequences are stored, including cluster center points, shape matrix products of subsequences, and average shape distances within each cluster. In this way, the K-Shape method does not need to load and process all the data from scratch each time, but gradually performs incremental calculations based on the clustering results within the page, thereby effectively reducing the computational overhead. In addition, in order to solve the problem of cluster merging across pages, page aggregation strategies for three scenarios are designed: adjacent pages, complementary pages, and overlapping pages. These strategies ensure the consistency and efficiency of clustering results under the conditions of disordered storage of time series by updating cluster center points, generating new clustering metadata, or processing overlapping areas.

[0089] Secondly, a clustering method based on center point-shape is proposed, which uses center points instead of eigenvectors as the basis for clustering. Specifically, this method determines the cluster assignment by selecting a subset of the subsequence as the representative points of the cluster and calculating the shape distance between these representative points and the current cluster center point. This approach avoids the complex feature decomposition steps in the traditional K-Shape method and significantly reduces the time complexity. On this basis, a greedy strategy is also introduced to quickly approach the global optimal solution with a suboptimal solution while maintaining the shape distance optimization. This method effectively improves the clustering efficiency of long time series, and at the same time can be close to or better than the original K-Shape method in terms of clustering accuracy.

[0090] Thirdly, we further combined the storage characteristics of the database and developed a center-shape-based clustering method within the database. Similar to the K-Shape method within the database, the approximate center points and the average shape distance between subsequences are stored in each page without the need to store the complete matrix information. This improvement not only greatly reduces the storage overhead, but also reduces the complexity of input and output operations. When aggregating pages, the center-shape-based clustering method adopts an incremental clustering update strategy to avoid reloading all the original data, but directly performs fast clustering and merging based on the stored approximate center points. In addition, for special scenarios of overlapping pages, the consistency and accuracy of the clustering results are further ensured by updating the shape distance and center point position of overlapping subsequences.

[0091] The database native time series clustering device provided by the present invention is described below. The database native time series clustering device described below and the database native time series clustering method described above can be referred to each other.

[0092] In some specific embodiments of the present invention, Fig. 9 As shown, this solution provides a database native time series clustering device, including: Initialization module 91, used to initialize and sort each page of the database storing the native time series, wherein the native time series includes subsequences of the same length; An acquisition module 92 is used to acquire the approximate clustering results of each of the pages and store the approximate clustering results through metadata, wherein the approximate clustering results of each page include r approximate clustering centers, clustering weights corresponding to each approximate clustering center, and average distances between subsequences in each cluster; The page-by-page aggregation module 93 is used to use the approximate clustering result of the first page of the database as the initial aggregation result, start from the first page of the database, aggregate the subsequent pages page by page, and update the approximate clustering results of the aggregated pages through metadata; The clustering module 94 is used to perform global clustering on the final aggregated data of each page after all pages are aggregated, and to speed up the global clustering process based on the updated approximate clustering results.

[0093] Fig.10 An example of a physical structure diagram of an electronic device is shown in FIG. Fig.10As shown, the electronic device may include: a processor (processor) 1010 , a communication interface (Communications Interface) 1020 , a memory (memory) 1030 and a communication bus 1040 , wherein the processor 1010 , the communication interface 1020 , and the memory 1030 communicate with each other via the communication bus 1040 . The processor 1010 can call the logic instructions in the memory 1030 to execute the database native time series clustering method, which includes: initializing the sorting of each page of the database storing the native time series, the native time series including subsequences of the same length; obtaining the approximate clustering results of each of the pages, and storing the approximate clustering results through metadata, the approximate clustering results of each page including r approximate clustering centers, the clustering weights corresponding to each approximate clustering center, and the average distance between subsequences in each cluster; using the approximate clustering result of the first page of the database as the initial aggregation result, starting from the first page of the database, the subsequent pages are aggregated page by page, and the approximate clustering results of the aggregated pages are updated through metadata; when all pages are aggregated, the final aggregated data of each of the pages is globally clustered, and the global clustering process is accelerated based on the updated approximate clustering results.

[0094] In addition, the logic instructions in the above-mentioned memory 1030 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0095] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the database native time series clustering method provided by the above methods, the method including: initializing sorting of each page of the database storing the native time series, the native time series including subsequences of the same length; obtaining approximate clustering results of each of the pages, and storing the approximate clustering results through metadata, the approximate clustering results of each page including r approximate clustering centers, clustering weights corresponding to each approximate clustering center, and the average distance between subsequences in each cluster; using the approximate clustering result of the first page of the database as the initial aggregation result, starting from the first page of the database, the subsequent pages are aggregated page by page, and the approximate clustering results of the aggregated pages are updated through metadata; when all pages are aggregated, the final aggregated data of each of the pages is globally clustered, and the global clustering process is accelerated based on the updated approximate clustering results.

[0096] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the database native time series clustering method provided by the above-mentioned methods, the method comprising: initializing sorting of each page of the database storing the native time series, the native time series comprising subsequences of the same length; obtaining approximate clustering results of each of the pages, and storing the approximate clustering results through metadata, the approximate clustering results of each page comprising r approximate clustering centers, clustering weights corresponding to each approximate clustering center, and the average distance between subsequences within each cluster; using the approximate clustering result of the first page of the database as the initial aggregation result, starting from the first page of the database, aggregating subsequent pages page by page, and updating the approximate clustering results of the aggregated pages through metadata; when all pages are aggregated, globally clustering the final aggregated data of each of the pages, and speeding up the global clustering process based on the updated approximate clustering results.

[0097] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0098] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A database native time series clustering method, characterized in that: include: Initializing and sorting each page of a database storing a native time series, wherein the native time series includes subsequences of the same length; Obtaining approximate clustering results of each of the pages, and storing the approximate clustering results through metadata, wherein the approximate clustering results of each page include r approximate clustering centers, clustering weights corresponding to each approximate clustering center, and average distances between subsequences in each cluster; Using the approximate clustering result of the first page of the database as the initial aggregation result, starting from the first page of the database, the subsequent pages are aggregated page by page, and the approximate clustering results of the aggregated pages are updated through metadata; When all pages are aggregated, the final aggregated data of each page is globally clustered, and the global clustering process is accelerated based on the updated approximate clustering results.

2. The database native time series clustering method according to claim 1, characterized in that: The obtaining of the approximate clustering results of the pages specifically includes: Define the length of each of the pages as l A subsequence of is a complete subsequence, and its length is less than l The subsequence of is an incomplete subsequence; For the complete subsequences in each of the pages, a basic clustering algorithm is used to obtain r approximate cluster centers, clustering weights corresponding to each approximate cluster center, and an average distance between subsequences in each cluster, and the r approximate cluster centers, clustering weights corresponding to each approximate cluster center, and an average distance between data in each cluster are stored through metadata, wherein the basic clustering algorithms include K-Means, K-Shape, and LSH-based; For the incomplete subsequence in each of the pages, the incomplete subsequence is stored via metadata.

3. The database native time series clustering method according to claim 1 or 2, characterized in that: The aggregating the subsequent pages page by page specifically includes: Determining a page relationship between the first page and the second page, wherein the page relationship includes adjacent pages, complementary pages, and overlapping pages; According to the judgment result of the page relationship, based on the page aggregation strategy, the data of the second page is aggregated to the first page; According to the initial sorting result of the pages, the page relationship between the current aggregated page and the next page is continuously determined, and according to the determination result of the page relationship and based on the page aggregation strategy, the data of the next page is aggregated into the current aggregated page.

4. The database native time series clustering method according to claim 3, characterized in that: Set the aggregated pages in the database to , the page to be aggregated is , The approximate cluster center of , clustering weight and the average distance of data within the cluster , j = 1, 2…r; The page aggregation strategy specifically includes: If the page , If the page is adjacent, With page Direct merger; If the page , If it is a complementary page, the page , Extract complementary subsequences from the generated new page , and in a new page Create new cluster centers in If the page , For overlapping pages, use page Overlapping subsequence pairs in the The shape distance and center point position of the overlapping subsequences are updated.

5. The database native time series clustering method according to claim 4, characterized in that: If the page , If the page is adjacent, With page Direct merger, including: For each page to be aggregated The cluster centers in , aggregate results : ； like, ,but: ； otherwise: ； Among them, the ones marked with an asterisk The variables are all the current optimal values; ++r is the class of the current page, and a new class is generated in the aggregation result; Update cluster weights The average distance to the data within the cluster .

6. The database native time series clustering method according to claim 4, characterized in that: If the page , If it is a complementary page, the page , Extract complementary subsequences from the generated new page , and in a new page Create new cluster centers in , including: Generate a new loading page , including the pages to be aggregated and aggregated pages Complementary data, among which, Include new approximate cluster centers: ； merge and The aggregation result of initialization The clustering weight and the distance of the data within the cluster are: ， 。 7. The database native time series clustering method according to claim 4, characterized in that: If the page , For overlapping pages, use page Overlapping subsequence pairs in the The shape distance and center point position of the overlapping subsequences in the update are updated, including: set up Display page Middle The approximate cluster centers of the clusters, Page The approximate cluster centers of clusters are updated as , the update of the approximate cluster center is as follows: ； 。 8. The database native time series clustering method according to claim 1, characterized in that: The globally clustering the final aggregated data of each of the pages and speeding up the global clustering process based on the updated approximate clustering results specifically includes: Based on each subsequence in the final aggregation result, an objective function is constructed with the goal of maximizing the sum of shape similarities between each subsequence and the cluster center to be predicted, wherein the cluster center to be predicted is the subsequence in the final aggregation result; Based on a greedy strategy, the objective function is approximately solved, and the solution process is accelerated by the updated approximate cluster centers to determine the approximate optimal cluster center combination of the native time series.

9. The database native time series clustering method according to claim 8, characterized in that: The constructing of the objective function specifically includes: Defining a subsequence With cluster center The shape similarity function is The calculation formula is: ； in, represents the weight, represents the i-th subsequence; is the jth cluster center; Defining functions is the clustering effect of the current cluster center set, and the calculation formula is: ； in, is the length of the time series T to be clustered l The subsequence set of , C is the cluster center set; Define the objective function The calculation formula is: 。 10. The database native time series clustering method according to claim 9, characterized in that: Based on the greedy strategy, the objective function is approximately solved, and the solution process is accelerated by the updated approximate cluster center, specifically including: Based on the greedy strategy, the objective function Perform an approximate solution; Set validity bounds to estimate subsequences and cluster centers The shape distance between them, the effective limit is expressed as: ； in, is the updated approximate cluster center; The solution process is accelerated by the effective bound.

11. A database native time series clustering device, characterized in that: include: An initialization module, used to initialize and sort each page of a database storing a native time series, wherein the native time series includes subsequences of the same length; An acquisition module, used to acquire the approximate clustering results of each of the pages, and store the approximate clustering results through metadata, wherein the approximate clustering results of each page include r approximate clustering centers, clustering weights corresponding to each approximate clustering center, and average distances between subsequences in each cluster; A page-by-page aggregation module is used to use the approximate clustering result of the first page of the database as the initial aggregation result, starting from the first page of the database, aggregating the subsequent pages page by page, and updating the approximate clustering results of the aggregated pages through metadata; The clustering module is used to perform global clustering on the final aggregated data of each page after all pages are aggregated, and to speed up the global clustering process based on the updated approximate clustering results.

Images