Data processing method, device and equipment, readable storage medium and program product

Abstract

The embodiment of the invention discloses a data processing method and device, equipment, a readable storage medium and a program product, and relates to the technical field of communication. Comprising the following steps: converting an original table of a row type storage structure into a target table of a column type storage structure; obtaining access statistical information of the target table; according to the access statistical information, clustering column fields in the target table according to popularity to obtain a clustering result; and determining a compression strategy and / or a storage strategy of the column field according to the clustering result. The embodiment of the invention can improve the data processing efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of communication, and in particular to a data processing method and device, equipment, readable storage medium and program product. BACKGROUND

[0002] When processing large-scale structured data (such as financial transactions, Internet of Things collection, industrial monitoring data, etc.), a relational database system usually adopts a unified row storage structure or a column storage structure for data organization. Columnar storage is widely used in time series databases, analytical databases and other scenarios due to its compression-friendly and batch operation-friendly characteristics. At the same time, to improve system performance and reduce resource consumption, more and more database systems introduce a hot and cold data management mechanism.

[0003] However, the hot and cold data layering technology in the prior art usually determines the data hotness in units of entire tables, entire regions or time periods, which leads to lagging recognition or misjudgment of hot and cold data, affecting the data processing effect of the database system. SUMMARY

[0004] Embodiments of the present application provide a data processing method, device, equipment, readable storage medium and program product to improve data processing efficiency.

[0005] In a first aspect, the embodiments of the present application provide a data processing method, comprising:

[0006] Converting an original table in a row storage structure into a target table in a column storage structure;

[0007] Obtaining access statistical information of the target table;

[0008] According to the access statistical information, clustering column fields in the target table according to hotness to obtain a clustering result;

[0009] According to the clustering result, determining a compression strategy and / or a storage strategy of the column fields.

[0010] Optionally, the converting an original table in a row storage structure into a target table in a column storage structure comprises:

[0011] Columnar conversion of a single original table in a row storage structure to obtain the target table in a column storage structure, wherein each column field in the target table includes a time range index and an intra-segment position index;

[0012] The time range index is used to represent the generation time range of the data in the column field, and the intra-segment position index is used to represent the position of each column field in the target table.

[0013] Optionally, the converting the original table of the row storage structure into the target table of the column storage structure comprises:

[0014] In the case that the plurality of original tables of row storage structure are row storage structures, at least part of row fields of the plurality of original tables of row storage structure are converted into column fields to obtain a global column field space, and the global column field space is taken as the target table, wherein each column field has a unique identifier.

[0015] Optionally, the clustering the column fields in the target table according to the access statistical information to obtain a clustering result comprises:

[0016] converting the access statistical information into an access feature vector;

[0017] processing the access feature vector by using a clustering algorithm to obtain the clustering result, wherein the clustering result at least comprises: a first heat cluster, a second heat cluster, and a third heat cluster;

[0018] wherein the access heat or access frequency of the first heat cluster, the second heat cluster, and the third heat cluster is sequentially sorted from high to low, or the access interval standard deviation of the first heat cluster, the second heat cluster, and the third heat cluster is sequentially sorted from low to high.

[0019] Optionally, the clustering the column fields in the target table according to the access statistical information to obtain a clustering result comprises:

[0020] converting the access statistical information into an access behavior vector;

[0021] obtaining an access track of the column field according to the access behavior vector;

[0022] processing the access track by using a clustering algorithm to obtain the clustering result, wherein the clustering result at least comprises:

[0023] a first classification, a second classification, a third classification, and a fourth classification;

[0024] wherein the access heat or access frequency of the first classification, the second classification, the third classification, and the fourth classification is sequentially sorted from high to low, or the access interval standard deviation of the first classification, the second classification, the third classification, and the fourth classification is sequentially sorted from low to high.

[0025] Optionally, the access statistical information comprises access statistical information in a sliding time window of each column field of the target table.

[0026] Optionally, the method further comprises:

[0027] Reorganizing the column fields belonging to the fourth classification to form reorganized column fields, wherein the reorganized column fields support batch processing.

[0028] Optionally, the method further comprises:

[0029] Obtaining an adjustment index of the column fields;

[0030] Adjusting at least one of the following according to the adjustment index:

[0031] a sliding time window, the compression strategy, the storage strategy, parameters of the clustering algorithm, clustering rules, column field division granularity.

[0032] Optionally, for a target table obtained based on a single row storage structure original table, the adjustment index comprises at least one of the following: cold-hot switching frequency, misclassification rate; or,

[0033] For a target table obtained based on a plurality of row storage structure original tables, the adjustment index comprises at least one of the following: compression ratio of cold segment migration, restoration time consumption, access delay statistical information.

[0034] In a second aspect, an embodiment of the present application provides a data processing apparatus, comprising:

[0035] A first conversion module, configured to convert a row storage structure original table into a column storage structure target table;

[0036] A first obtaining module, configured to obtain access statistical information of the target table;

[0037] A second obtaining module, configured to cluster column fields in the target table according to the access statistical information to obtain a clustering result;

[0038] A first determining module, configured to determine a compression strategy and / or a storage strategy of the column fields according to the clustering result.

[0039] Optionally, the first conversion module is further configured to convert a single row storage structure original table into the column storage structure target table, wherein each column field in the target table comprises a time range index and an intra-segment position index;

[0040] The time range index is used to represent a generation time range of data in the column field, and the intra-segment position index is used to represent a position of each column field in the target table.

[0041] Optionally, the first conversion module is further configured to, in a case where the original table of the plurality of row storage structures is a row storage structure, convert at least part of row fields of the original table of the plurality of row storage structures into column fields to obtain a global column field space, and use the global column field space as the target table, wherein each column field has a unique identifier.

[0042] Optionally, the first obtaining module is further configured to:

[0043] convert the access statistical information into an access feature vector;

[0044] process the access feature vector by using a clustering algorithm to obtain the clustering result, wherein the clustering result at least includes: a first heat cluster, a second heat cluster, and a third heat cluster;

[0045] wherein the access heat or access frequency of the first heat cluster, the second heat cluster, and the third heat cluster is sequentially ranked from high to low, or the access interval standard deviation of the first heat cluster, the second heat cluster, and the third heat cluster is sequentially ranked from low to high.

[0046] Optionally, the first obtaining module is further configured to convert the access statistical information into an access behavior vector.

[0047] obtain an access track of the column field according to the access behavior vector;

[0048] process the access track by using a clustering algorithm to obtain the clustering result, wherein the clustering result at least includes:

[0049] a first classification, a second classification, a third classification, and a fourth classification.

[0050] wherein the access heat or access frequency of the first classification, the second classification, the third classification, and the fourth classification is sequentially ranked from high to low, or the access interval standard deviation of the first classification, the second classification, the third classification, and the fourth classification is sequentially ranked from low to high.

[0051] Optionally, the access statistical information includes access statistical information in a sliding time window of each column field of the target table.

[0052] Optionally, the apparatus further includes a third obtaining module configured to reorganize column fields belonging to the fourth classification to form reorganized column fields, wherein the reorganized column fields support batch processing.

[0053] Optionally, the apparatus further comprises a fourth obtaining module configured to obtain an adjustment index of the column field; and a first processing module configured to adjust at least one of the following according to the adjustment index:

[0054] a sliding time window, the compression strategy, the storage strategy, a parameter of the clustering algorithm, a clustering rule, and a column field division granularity.

[0055] Optionally, for a target table obtained based on a single row storage structure original table, the adjustment index comprises at least one of the following: a cold-hot switching frequency and a misclassification rate; or,

[0056] For a target table obtained based on a plurality of row storage structure original tables, the adjustment index comprises at least one of the following: a compression ratio of cold segment migration, a restoration time consumption, and access delay statistical information.

[0057] In a third aspect, an embodiment of the present application further provides a communication device, comprising a memory, a processor, and a program stored in the memory and capable of running on the processor, and the processor implements the steps in the data processing method as described above when executing the program.

[0058] In a fourth aspect, an embodiment of the present application further provides a readable storage medium, and the readable storage medium stores a program, and the program is executed by a processor to implement the steps in the data processing method as described above.

[0059] In a fifth aspect, an embodiment of the present application further provides a computer program product, comprising computer instructions, and the computer instructions are executed by a processor to implement the steps in the data processing method as described above.

[0060] In the embodiment of the present application, based on a column storage structure, the access statistical information of a column field and a clustering algorithm are combined to dynamically identify the heat state of data, and then a differentiated compression and / or storage strategy of the data is determined, adaptive hierarchical management of column field data is realized, and the data processing efficiency of the system is improved. BRIEF DESCRIPTION OF DRAWINGS

[0061] Figure 1 is a flowchart of the data processing method provided by the embodiment of the present application;

[0062] Figure 2 is a structural diagram of the data processing apparatus provided by the embodiment of the present application. DETAILED DESCRIPTION

[0063] In the embodiment of the present application, the term "and / or" describes the association relationship of the associated objects, and indicates that there can be three relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone. The character " / " generally represents an "or" relationship between the associated objects before and after it.

[0064] The term "multiple" in the embodiments of the present application refers to two or more, and other quantifiers are similar.

[0065] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0066] Referring to Figure 1 , Figure 1 is a flowchart of a data processing method provided by the embodiments of the present application, and the method can be applied to electronic devices such as terminals and servers. As shown in Figure 1 , the method comprises the following steps:

[0067] Step 101: converting a row storage structure original table into a column storage structure target table.

[0068] Step 102: obtaining access statistical information of the target table.

[0069] Step 103: clustering column fields in the target table according to the access statistical information and the heat, to obtain a clustering result.

[0070] Step 104: determining a compression strategy and / or a storage strategy of the column fields according to the clustering result.

[0071] In the embodiments of the present application, based on the column storage structure, the heat state of data is dynamically identified by combining the access statistical information of column fields and the clustering algorithm, and then the differentiated compression and / or storage strategy of data is determined, so that adaptive hierarchical management of column field data is realized, and the data processing efficiency of the system is improved.

[0072] In step 101, for a single row storage structure original table, the original table can be converted into a column table to obtain the column storage structure target table. In the target table, the content of each column field (or column field) is an independent data segment (Column Segment), and each data segment comprises a time range index and an intra-segment position index. The time range index is used to indicate the generation time range of the data in the column field, and the intra-segment position index is used to indicate the position of each column field in the target table. The division granularity of the column field is configurable, for example, each segment can contain 1000-5000 records by default, so as to ensure the logical integrity between column fields and facilitate statistics and scheduling.

[0073] In the embodiments of the present application, the specific implementation manner of converting the columnization of the original table is not limited.

[0074] In step 101, for the plurality of original tables in row storage structure, if the original tables are in row storage structure, at least part of the row fields of the plurality of original tables in row storage structure can be converted into columnization to obtain a global column field space, and the global column field space is taken as the target table. Of course, the plurality of original tables in row storage structure can also be in column storage structure, and in this case, the column fields of the tables can be uniformly converted into columnization.

[0075] In the embodiments of the present application, the column fields of all tables are abstracted into a global column field space, each column field has a unique identifier, including but not limited to the table, field, time interval and segment location information. The system uniformly manages these column fields, which facilitates cross-table access pattern analysis and scheduling. The plurality of original tables in row storage structure can be business key tables, log tables, etc., and the at least part of the row fields can be part of the row fields of all tables or all row fields.

[0076] In the embodiments of the present application, the access row space can be uniformly constructed across multiple logical tables, and the field column segments in multiple tables are uniformly abstracted into a global column segment set, so as to realize cross-table level hot and cold recognition and compression strategy reuse, and improve the overall scheduling efficiency. Compared with the single-table independent scheduling mode in the traditional system, this mode improves the reusability of hot and cold strategies and the unified management ability of compressed resources, and can adapt to the actual needs of concurrent access of multiple tables and multiple fields in a complex business system.

[0077] In the embodiments of the present application, in step 102, the access statistics information can be obtained by intercepting and analyzing the access request (such as structured query language (SQL)) of the user. The access statistics information includes but is not limited to access frequency, access table information, access field information, access mode, access timestamp, etc.

[0078] By means of the built-in access log tracker, each SQL request such as SELECT, UPDATE, etc. is intercepted and analyzed, and the column field identifier (ID), request type (reading or writing), occurrence time, client identifier, average interval, recent active time, etc. are extracted. For each column field, an independent sliding time window can also be maintained, and the above information obtained in the sliding time window is counted to obtain the access statistics information of each column field in the sliding time window of the target table. The length of the sliding time window can be set as required, for example, 30 minutes, etc. In addition, the sliding time window can also be refreshed periodically, for example, refreshed once every 5 minutes by default, etc.

[0079] For the target table obtained based on multiple tables, a behavior coding model can also be uniformly defined for the global SQL request log to convert different operations or access behaviors into standard behavior vectors. For example, reading field X in table A once is coded as (A, X, R, T), R represents the read operation, and T is the timestamp; updating field Y in table B is coded as (B, Y, W, T), and W represents the write operation. Subsequently, the system constructs the time series access trajectory of the column field as the input of subsequent behavior modeling.

[0080] In the case of obtaining access statistics, in step 103, for the target table obtained by columnar conversion of the original table based on a single row storage structure, the access statistics can be converted into an access feature vector, wherein the access feature vector includes at least one of access frequency, access interval standard deviation, read-write ratio, and request source quantity. The access interval standard deviation is calculated according to the access frequency, and the request source quantity can represent which user or which business the access request comes from. Then, the access feature vector is processed by using a clustering algorithm to obtain the clustering result, wherein the clustering result at least includes: a first heat cluster, a second heat cluster, and a third heat cluster; wherein the access heat or access frequency of the first heat cluster, the second heat cluster, and the third heat cluster are sorted from high to low in turn, or the access interval standard deviation of the first heat cluster, the second heat cluster, and the third heat cluster are sorted from low to high in turn.

[0081] The clustering algorithm includes but is not limited to a density-based clustering algorithm, such as Density-Based Spatial Clustering of Applications with Noise (DBSCAN). The greater the access heat or access frequency, the higher the heat of the column field; otherwise, the smaller. The smaller the access interval standard deviation rate, the higher the heat of the column field; otherwise, the smaller. Therefore, in the embodiments of the present application, the first heat cluster, the second heat cluster, and the third heat cluster can be obtained based on the clustering algorithm. For example, the first heat cluster, the second heat cluster, and the third heat cluster can be high-frequency cluster (hot data), medium-frequency cluster (warm data), and low-frequency cluster (cold data), respectively. The clustering result is used as a data cold and hot judgment basis to mark each column field and store it in the metadata index table.

[0082] Thus, in the embodiments of the present application, the row data in the relational database is columnized based on column storage, and the access behavior is tracked independently after the data is divided into column fields. By using a sliding time window to record the access frequency, interval, read-write ratio and other characteristics, an access feature vector at the column segment level is formed, which can accurately capture the data access activity at the field level and segment level, avoid performance fluctuations caused by cold and hot misjudgment, and improve the granularity and accuracy of cold and hot identification.

[0083] In step 103, for the target table obtained by columnizing conversion based on multiple original tables, the access statistical information can be converted into an access behavior vector, and the access trajectory of the column field can be obtained according to the access behavior vector. Then, the access trajectory is processed by using a clustering algorithm to obtain the clustering result. The clustering algorithm includes but is not limited to a time series clustering algorithm based on shape similarity (such as K-Shape or Dynamic Time Warping (DTW) clustering). The clustering result at least includes a first classification, a second classification, a third classification, and a fourth classification. The order of the access heat or access frequency of the first classification, the second classification, the third classification, and the fourth classification is from high to low, or the order of the access interval standard deviation of the first classification, the second classification, the third classification, and the fourth classification is from low to high. The greater the access heat or access frequency, the higher the heat of the column field; otherwise, the smaller. The smaller the access interval standard deviation rate, the higher the heat of the column field; otherwise, the smaller. Therefore, in the embodiments of the present application, the first classification, the second classification, the third classification, and the fourth classification can be obtained based on the clustering algorithm. For example, the first classification, the second classification, the third classification, and the fourth classification are "persistent hot segment", "periodic hot segment", "short-term hot segment", and "long-term cold segment", respectively. Such clustering does not depend on a fixed time window threshold, but captures the trend changes of access patterns, and is more suitable for sudden behaviors and periodic business fluctuations.

[0084] In an embodiment of the present application, for the case of the target table obtained by columnizing conversion based on multiple original tables, the column fields belonging to the fourth classification can also be reorganized to form reorganized column fields, wherein the reorganized column fields support batch processing, thereby improving the storage efficiency of the data of the fourth classification. Here, the column fields of the fourth classification can be repackaged according to column field attribution, time proximity, and access history similarity to form optimized archive data blocks, i.e., reorganized column fields. These data blocks support batch compression, batch writing to object storage, and generating compressed metadata index for subsequent query optimization module. This process can be performed during the system load valley period to avoid interference with online business.

[0085] By constructing an access behavior vector, combining unsupervised models such as density clustering and time series clustering, and clustering all column fields, dynamic classification can be performed. Since this mechanism does not require manual threshold setting, it can adapt to changes in business access patterns and automatically correct recognition errors. Compared with traditional static hot and cold recognition methods based on access frequency thresholds or fixed time windows, the unsupervised clustering algorithm based on density or time series is introduced to automatically model data access patterns, recognize periodic, bursty, and long-tail behaviors, and has higher environmental adaptability and behavior trend perception ability, improving the robustness and dynamic response capability of hot and cold division.

[0086] In an embodiment of the present application, an adjustment index of the column field can also be obtained, and at least one of the following is adjusted according to the adjustment index: a sliding time window (such as the length of the sliding time window), the compression strategy, the storage strategy, the parameters of the clustering algorithm, and the clustering rule.

[0087] For a target table obtained from an original table based on a single row storage structure, the adjustment index includes at least one of the following: hot and cold switching frequency, misclassification rate; or for a target table obtained from an original table based on multiple row storage structures, the adjustment index includes at least one of the following: compression ratio of cold segment migration, restoration time consumption, and access delay statistical information.

[0088] For example, the system periodically performs a clustering task (or a hot and cold division evaluation task) of data, compares the sampled access history with the current clustering result, and calculates the hot and cold switching frequency, misclassification rate, and other indexes of the column field. If the data of a certain class frequently fluctuates between cold and hot states (for example, the switching between cold and hot states reaches a preset number within a certain time range, and both the time range and the preset number can be set as needed), the system triggers parameter self-adjustment logic, including adjusting the sliding window size, retraining the clustering model, or refining the column field division granularity.

[0089] Through this feedback loop mechanism of "access characteristics-heat division-compression strategy-storage allocation", the actual use situation (such as compression ratio, decompression time consumption, access hit rate, etc.) after compression can be continuously collected, and the above adjustable information can be dynamically adjusted to build a data-driven self-optimization mechanism, realize online evolution of clustering parameters, compression strategies, segment division granularity, and other core functions, and ensure long-term stability and adaptability of the system, and gradually evolve the hot and cold recognition model that is most suitable for the current business load, and improve the overall system performance. At the same time, this error control mechanism can ensure that the data recovery quality is not affected under different compression ratios; through the adaptive compression strategy, the compression parameters can be adjusted according to the change trend of the data to ensure that data loss is minimized during the compression process. Compared with existing static compression mechanisms without feedback and adjustment, the embodiments of the present application have stronger long-term optimization capability and system stability.

[0090] For another example, the system reserves the compression ratio, restore time consumption and access delay statistics of each cold segment migration, and archives them in association with the original cluster label and behavior pattern record. By comparing the execution effects of different strategy groups, the system periodically optimizes the hotness classification rule, clustering algorithm parameter and behavior encoding structure, realizes online evolution of the cold and hot recognition mechanism, and improves the long-term strategy adaptation degree and system robustness.

[0091] In step 104, for different classified column fields, corresponding compression strategies, storage strategies, etc. can be set.

[0092] For example, the system allocates compression strategies and storage paths to column fields according to states: the first hotness cluster (hot data) uses lightweight compression (such as LZ4), resides in main memory or a solid state disk (SSD) high-speed layer; the second hotness cluster (warm data) uses medium compression (such as Snappy) and is migrated to a general disk; the third hotness cluster (cold data) uses a high compression ratio algorithm (such as Zstandard (ZSTD)) and is stored in a low-cost medium (such as an archive storage, an object storage bucket). At the same time, the third hotness cluster (cold data) allows asynchronous decompression and batch prefetching mechanism to reduce decompression delay.

[0093] For another example, the first classification (“persistent hot segment”) is bound to a memory or a non-volatile memory express (NVMe) layer, and deep compression is disabled; the second classification (“periodic hot segment”) or the third classification (“short-term hot segment”) enables timed decompression and time-sharing migration; the fourth classification (“long-term cold segment”) is directly written to an object storage, and a maximum compression rate parameter is enabled. The system constructs a mapping table of column fields to storage layers, and supports hotness-aware page cache scheduling, thereby guaranteeing that hot segments are preferentially hit in cache.

[0094] By mapping different storage strategies and compression strategies for different classifications, dynamic balance of compression ratio and access delay can be achieved. In the embodiments of the present application, by introducing a real-time modeling mechanism of access patterns and a hotness-aware framework at the column level, combined with dynamic adaptive control of compression strategies, an integrated cold and hot data management process from perception, judgment to response is realized, thereby improving the comprehensive ability of the database system in terms of data processing efficiency, resource utilization and storage performance.

[0095] As can be seen from the above description, in the embodiments of the present application, through columnar storage optimization, access frequency monitoring, behavior feature clustering and differentiated compression mechanism, the cold and hot distribution characteristics of data access can be dynamically recognized, and then automatic multi-level storage strategy matching is realized, thereby improving the overall access performance and storage efficiency of the system.

[0096] The embodiment of the application proposes a relational database intelligent cold and hot data hierarchical storage method based on columnarization and access mode analysis. The design goal is to improve the access efficiency and resource utilization of the database system when facing massive data. The method is based on columnar storage structure, combined with sliding window statistics, access behavior modeling and clustering algorithm, dynamically identifies the data heat state, and combines the differential compression and storage strategy to realize the adaptive hierarchical management of cold and hot data for column fields. In the following, the implementation process of the embodiment of the application is described by taking the target table based on the columnar storage structure of a single table and the target table constructed by multiple tables as examples.

[0097] In an embodiment of the application, the target table based on the columnar storage structure of a single table, the process can include:

[0098] Step 11, the original row storage table is converted according to the field, the columnar storage structure is constructed, and the target table of the columnar storage structure is obtained.

[0099] Wherein, the content of each column field is organized as an independent data segment (Column Segment), and a time range index and a segment position index are established on each data segment. The column field division granularity can be configured, and by default, each segment contains 1000-5000 records, which guarantees the logical integrity between column fields and facilitates statistics and scheduling.

[0100] Step 12, the system built-in access log tracker intercepts and analyzes each SQL request such as SELECT, UPDATE, etc., such as the accessed column field ID, the request type (read / write), the occurrence time and the client identification. Each column field maintains an independent sliding time window (such as 30 minutes), records the access frequency, average interval, latest active time and other key indicators in the sliding time window as access statistics information, and periodically (default 5 minutes) refreshes.

[0101] Step 13, for each sliding time window, the system converts the access statistics information of all column fields into an access feature vector, including access frequency, access interval standard deviation, read-write ratio, request source quantity and other multi-dimensional indicators. The system uses a density-based clustering algorithm (such as DBSCAN) to divide the column fields into multiple heat clusters: high-frequency cluster (hot), medium-frequency cluster (warm) and low-frequency cluster (cold). The clustering model automatically adapts to the access distribution under the current load without manual threshold setting.

[0102] Step 14, the clustering result is used as the basis for cold and hot judgment, each column field is marked as hot, warm and cold state, and stored in the metadata index table.

[0103] The system assigns compression strategies and storage paths to column fields according to states: hot data uses lightweight compression (such as LZ4), resides in main memory or SSD high-speed layer; warm data uses moderate compression (such as Snappy) and migrates to ordinary disks; cold data uses high compression ratio algorithms (such as ZSTD) and is stored in low-cost media (such as archival storage, object storage buckets). At the same time, cold data allows asynchronous decompression and batch prefetching mechanism to reduce decompression delay.

[0104] Step 15, the system periodically performs cold-hot partition evaluation tasks, compares the sampling access history with the current clustering results, and calculates indicators such as cold-hot switching frequency and misclassification rate. If certain data frequently fluctuates between cold and hot states, the system will trigger parameter self-adjustment logic, including adjusting the size of the sliding window, retraining the clustering model, or refining the column field division granularity. This feedback loop mechanism can gradually evolve the cold-hot identification model that best suits the current business load.

[0105] In an embodiment of the present application, a target table is constructed based on a plurality of tables to build a columnar storage structure, which process can include:

[0106] Step 21, select a plurality of target tables (such as business critical tables, log tables, etc.) in the system, and perform unified columnar processing on all column fields of the target tables to obtain the target table.

[0107] Among them, the column fields of all tables are abstracted into a global column field space, each column field has a unique identifier, including the table it belongs to, the field, the time interval, and the position information within the segment. The system uniformly manages these column fields to facilitate cross-table access pattern analysis and scheduling.

[0108] Step 22, the system has an access log tracker that intercepts and analyzes each SQL request such as SELECT, UPDATE, etc., such as column field ID, request type (read / write), time of occurrence, and client identification. Each column field maintains an independent sliding time window (such as 30 minutes) that records access frequency, average interval, and recent active time within the sliding time window as access statistics, and is refreshed periodically (default 5 minutes).

[0109] Specifically, for global SQL request logs, the system defines a unified behavior coding model to convert different operations into standardized behavior vectors. For example, reading field X in table A once is coded as (A, X, R, T), where T is the timestamp; updating field Y in table B is coded as (B, Y, W, T). Then, the system constructs the time series access trajectory of the column field as the input of the behavior modeling.

[0110] Step 23, the system uses a time series clustering algorithm based on shape similarity (such as K-Shape or Dynamic Time Warping clustering) to cluster all column field access trajectories. The clustering results reflect the similarity of access patterns of different column fields over time, thereby identifying multiple behavior patterns such as "persistent hot segment", "periodic hot segment", "short-term hot segment", and "long-term cold segment". This clustering does not rely on fixed time window thresholds, but captures access pattern trend changes, making it more suitable for handling sudden behavior and periodic business fluctuations.

[0111] Step 24, according to the clustering label, the system assigns a differentiated storage strategy to each type of column field. For example, "persistent hot segment" is bound to memory or NVMe layer, and deep compression is disabled; "periodic hot segment" or "short-term hot segment" enables timed decompression and time-sharing migration; "long-term cold segment" is directly written to object storage, and the maximum compression ratio parameter is enabled.

[0112] The system constructs a mapping table of column fields to storage layers and supports hot-aware page cache scheduling to ensure that hot segments are preferentially hit in the cache.

[0113] Step 25, to improve the efficiency of long-term storage of cold data, the system periodically performs batch reorganization processing on "long-term cold segments". According to the column field attribution, time proximity, and access history similarity, the cold segments are repackaged to form optimized archival data blocks. These data blocks support batch compression and batch writing to object storage, and generate compression metadata indexes for subsequent query optimization modules. This process is performed during the system load valley period to avoid interference with online business.

[0114] Step 26, the system retains the compression ratio, restoration time consumption, and access delay statistics of each cold segment migration, and archives them in association with the original clustering label and behavior pattern record. By comparing the execution effects of different strategy groups, the system periodically optimizes the hotness classification rules, clustering algorithm parameters, and behavior encoding structure, realizes the online evolution of the cold and hot recognition mechanism, and improves the long-term strategy adaptation degree and system robustness.

[0115] Through the above two technical implementation schemes for single table and multiple tables, the embodiments of the present application can achieve the following beneficial effects:

[0116] 1. Multi-dimensional cold and hot recognition based on columnar structure in relational databases, behavior-driven compression strategy adaptation and storage scheduling are realized, and good scalability, real-time performance, and controllability are maintained in multiple tables, multiple fields, and multiple access types scenarios, greatly improving the comprehensive performance of the system in big data scenarios, while reducing data storage and operation and maintenance costs.

[0117] 2. In the embodiments of the present application, by introducing columnization and compression technology, the occupation of data on physical storage space is greatly reduced under the premise of ensuring data integrity. For enterprises and institutions that handle massive time series data, storage device investment and later operation and maintenance costs can be directly saved, especially under the cloud computing and big data platform, which has cost advantages.

[0118] 3. The columnized and compressed data structure is more suitable for modern parallel processing and vectorized computing, greatly improving the response speed and throughput capacity of the database in query, analysis and other operations. Therefore, the scheme of the embodiments of the present application can enhance the data processing efficiency of enterprises in real-time monitoring, intelligent manufacturing, financial risk control and other key business scenarios, improve the system throughput capacity and response efficiency, bring better customer experience and decision support capability.

[0119] 4. The scheme of the embodiments of the present application can be widely integrated into existing relational database products as a plug-in or core functional module, enabling traditional databases to upgrade in the direction of "time series awareness" and "efficient compression", with good technical barriers and scalability. For database product developers and industry solution providers, it will enhance their product technical competitiveness and market expansion capability, gain stronger market differentiation advantage, and expand business opportunities in emerging fields such as industrial Internet of Things, smart city and energy management.

[0120] Referring to Figure 2 , Figure 2 is a structural diagram of a data processing apparatus provided by the embodiments of the present application. As shown in Figure 2 , the data processing apparatus comprises:

[0121] A first conversion module 201 is configured to convert an original table in a row storage structure into a target table in a column storage structure; a first acquisition module 202 is configured to acquire access statistical information of the target table; a second acquisition module 203 is configured to cluster column fields in the target table according to the access statistical information to obtain a clustering result; and a first determination module 204 is configured to determine a compression strategy and / or a storage strategy of the column fields according to the clustering result.

[0122] Optionally, the first conversion module 201 is further configured to convert an original table in a single row storage structure into a target table in a column storage structure, wherein each column field in the target table comprises a time range index and an intra-segment position index.

[0123] The time range index is configured to represent the generation time range of data in the column field, and the intra-segment position index is configured to represent the position of each column field in the target table.

[0124] Optionally, the first conversion module 201 is further configured to, in a case where the original table of the plurality of row storage structures is a row storage structure, convert at least part of row fields of the original table of the plurality of row storage structures into column fields to obtain a global column field space, and use the global column field space as the target table, wherein each column field has a unique identifier.

[0125] Optionally, the first obtaining module 202 is further configured to:

[0126] convert the access statistical information into an access feature vector;

[0127] process the access feature vector by using a clustering algorithm to obtain the clustering result, wherein the clustering result at least includes: a first heat cluster, a second heat cluster, and a third heat cluster;

[0128] wherein the access heat or access frequency of the first heat cluster, the second heat cluster, and the third heat cluster is sequentially ranked from high to low, or the access interval standard deviation of the first heat cluster, the second heat cluster, and the third heat cluster is sequentially ranked from low to high.

[0129] Optionally, the first obtaining module 202 is further configured to convert the access statistical information into an access behavior vector.

[0130] obtain an access track of the column field according to the access behavior vector;

[0131] process the access track by using a clustering algorithm to obtain the clustering result, wherein the clustering result at least includes:

[0132] a first classification, a second classification, a third classification, and a fourth classification;

[0133] wherein the access heat or access frequency of the first classification, the second classification, the third classification, and the fourth classification is sequentially ranked from high to low, or the access interval standard deviation of the first classification, the second classification, the third classification, and the fourth classification is sequentially ranked from low to high.

[0134] Optionally, the access statistical information includes access statistical information in a sliding time window of each column field of the target table.

[0135] Optionally, the apparatus further includes a third obtaining module configured to reorganize column fields belonging to the fourth classification to form reorganized column fields, wherein the reorganized column fields support batch processing.

[0136] Optionally, the apparatus further comprises: a fourth acquisition module, configured to acquire an adjustment index of the column field; and a first processing module, configured to adjust at least one of the following according to the adjustment index:

[0137] a sliding time window, the compression strategy, the storage strategy, a parameter of the clustering algorithm, a clustering rule, and a column field division granularity.

[0138] Optionally, for a target table obtained based on a single row storage structure original table, the adjustment index comprises at least one of the following: a cold-hot switching frequency and a misclassification rate; or,

[0139] For a target table obtained based on a plurality of row storage structure original tables, the adjustment index comprises at least one of the following: a compression ratio of cold segment migration, a restoration time consumption, and access delay statistical information.

[0140] The apparatus provided in the embodiments of the present application can execute the method embodiments described above, and has similar implementation principles and technical effects, which will not be described here in detail.

[0141] It should be noted that the division of units in the embodiments of the present application is illustrative, and is only a logical functional division. In actual implementation, another division mode can be used. In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0142] When the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a processor-readable storage medium. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, includes several instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) execute all or part of the steps of the methods described in the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, and various program code storage media.

[0143] The embodiment of the present application provides a communication device, comprising a memory, a processor and a program stored in the memory and executable on the processor; the processor is used for reading the program in the memory to realize the steps in the data processing method.

[0144] The embodiment of the present application also provides a readable storage medium, and the readable storage medium stores a program, the program is executed by a processor to realize various processes of the data processing method embodiment, and the same technical effects can be achieved, and details are not repeated here. The readable storage medium can be any available medium or data storage device that can be accessed by the processor, including but not limited to a magnetic memory (for example, a floppy disk, a hard disk, a magnetic tape, a magneto-optical disk (MO) and the like), an optical memory (for example, a CD, a DVD, a BD, a HVD and the like), and a semiconductor memory (for example, a ROM, an EPROM, an EEPROM, a nonvolatile memory (NAND FLASH), a solid state disk (SSD)) and the like.

[0145] The embodiment of the present application also provides a computer program product, comprising computer instructions, which are executed by a processor to realize various processes of the data processing method embodiment, and the same technical effects can be achieved, and details are not repeated here.

[0146] It should be noted that in this paper, the term "including", "containing" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or includes elements inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, method, article or device including the element.

[0147] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment method can be realized by means of software and necessary general hardware platform, of course, it can also be realized by hardware, but in many cases, the former is a better embodiment. According to such understanding, the technical solutions of the present application can be embodied in the form of computer software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a plurality of instructions for making a terminal (which can be a mobile phone, computer, server, air conditioner or network equipment) execute the method described in each embodiment of the present application.

[0148] The embodiments of the present application are described above with reference to the accompanying drawings, but the present application is not limited to the specific embodiments described above, and the specific embodiments described above are merely illustrative, but not restrictive, and a person of ordinary skill in the art can make many forms under the inspiration of the present application without departing from the purpose of the present application and the scope protected by the claims.

Claims

1. A data processing method, characterized by, The method comprises the following steps: Converting a row storage structure original table into a column storage structure target table; Obtaining access statistical information of the target table; According to the access statistical information, clustering column fields in the target table according to heat to obtain a clustering result; According to the clustering result, determining a compression strategy and / or a storage strategy of the column fields.

2. The method of claim 1, wherein, The method of converting a row storage structure original table into a column storage structure target table comprises the following steps: Columnar conversion of a single row storage structure original table to obtain the column storage structure target table, wherein each column field in the target table comprises a time range index and an intra-segment position index; The time range index is used to represent the generation time range of the data in the column field, and the intra-segment position index is used to represent the position of each column field in the target table.

3. The method of claim 1, wherein, The method of converting a row storage structure original table into a column storage structure target table comprises the following steps: In the case that a plurality of row storage structure original tables are row storage structures, columnar conversion of at least part of the row fields of the plurality of row storage structure original tables to obtain a global column field space, and taking the global column field space as the target table, wherein each column field has a unique identifier.

4. The method of claim 1, wherein, The method of clustering column fields in the target table according to heat according to the access statistical information to obtain a clustering result comprises the following steps: Converting the access statistical information into an access feature vector; Processing the access feature vector by using a clustering algorithm to obtain the clustering result, wherein the clustering result at least comprises: a first heat cluster, a second heat cluster, and a third heat cluster; The access heat or access frequency of the first heat cluster, the second heat cluster, and the third heat cluster is sequentially sorted from high to low, or the access interval standard deviation of the first heat cluster, the second heat cluster, and the third heat cluster is sequentially sorted from low to high.

5. The method of claim 1, wherein, The method of clustering column fields in the target table according to heat according to the access statistical information to obtain a clustering result comprises the following steps: Converting the access statistical information into an access behavior vector; According to the access behavior vector, obtaining an access trajectory of the column fields; Processing the access trajectory by using a clustering algorithm to obtain the clustering result, wherein the clustering result at least comprises: A first classification, a second classification, a third classification, and a fourth classification; The access heat or access frequency of the first classification, the second classification, the third classification, and the fourth classification is sequentially sorted from high to low, or the access interval standard deviation of the first classification, the second classification, the third classification, and the fourth classification is sequentially sorted from low to high.

6. The method of claim 5, wherein, The method further comprises: Reorganizing the column fields belonging to the fourth classification to form reorganized column fields, wherein the reorganized column fields support batch processing.

7. The method according to claim 4 or 5, characterized in that, The access statistical information comprises access statistical information in a sliding time window of each column field of the target table.

8. The method of claim 7, wherein, The method further comprises: Obtaining an adjustment index of the column fields; According to the adjustment index, adjusting at least one of the following: A sliding time window, the compression strategy, the storage strategy, parameters of the clustering algorithm, clustering rules, column field division granularity.

9. The method of claim 8, wherein, For a target table obtained from an original table based on a single row storage structure, the adjustment index comprises at least one of: cold-hot switching frequency, misclassification rate; or, For a target table obtained from an original table based on multiple row storage structures, the adjustment index comprises at least one of: compression ratio of cold segment migration, restoration time consumption, access delay statistical information.

10. A data processing apparatus, characterized by, The method comprises: A first conversion module configured to convert an original table in a row storage structure into a target table in a column storage structure; A first acquisition module configured to acquire access statistical information of the target table; A second acquisition module configured to cluster column fields in the target table according to heat based on the access statistical information to obtain a clustering result; A first determination module configured to determine a compression strategy and / or a storage strategy of the column fields based on the clustering result.

11. A communication device comprising: A memory, a processor, and a program stored in the memory and executable on the processor; and the processor is configured to read the program in the memory to implement the steps in the data processing method according to any one of claims 1 to 9.

12. A computer readable storage medium for storing a program, characterized in that, The program is executed by the processor to implement the steps in the data processing method according to any one of claims 1 to 9.

13. A computer program product, characterised in that, The computer program comprises computer instructions, and the computer instructions are executed by the processor to implement the steps in the data processing method according to any one of claims 1 to 9.

Images