A computer data management system and method

Abstract

The application discloses a kind of computer data management system and method, it is related to data management technical field, comprising: reading heterogeneous data set and dividing into text, time series and binary data group according to data type, extract the core field of each data group and generate deduplication feature and index feature, based on data type Setting sliding window and generating window index feature abstract;According to sliding window, the integrated similarity score between data block is calculated, similar data block set is filtered and reference data block and redundant data block are determined, the initial clustering cluster is generated to reference data block clustering, if global deduplication rate and clustering preprocessing effectiveness do not satisfy preset condition, then reinitialize sliding window;Based on the search heat forecast value of initial clustering cluster, optimize cluster structure and generate cluster feature abstract, construct three-layer index structure including global index, intra-cluster index and feature index, test retrieval efficiency and optimize index level, significantly improve the storage efficiency and retrieval performance of heterogeneous data management.

Description

Technical Field

[0001] This invention relates to the field of data management technology, specifically to a computer data management system and method. Background Technology

[0002] In existing computer data management systems, data deduplication and index building are common optimization techniques. Data deduplication technology mainly reduces storage space usage by identifying and eliminating duplicate or highly similar data blocks; while index building improves data query efficiency by establishing an efficient data retrieval structure. Traditional methods typically use similarity comparison algorithms with fixed parameters for deduplication, such as hash comparison and sliding window comparison, and combine them with inverted indexes, B+ trees, and other structures to build indexes. These techniques perform well on single data types or small-scale datasets and can meet basic storage and retrieval needs.

[0003] However, existing technologies have obvious limitations. First, data deduplication and index building are usually performed independently. The deduplication process does not fully consider the needs of subsequent indexing, which leads to the need to re-extract features during index building, increasing computational overhead. At the same time, index building also fails to fully utilize the features of the deduplicated data, which can easily generate redundant indexes and reduce retrieval efficiency. Second, deduplication algorithms with fixed parameters, such as fixed window size and similarity threshold, are difficult to adapt to the feature differences of multiple data types, such as text, time series, and binary data, which can easily lead to missed or false positives. Moreover, the unordered distribution of deduplicated data further increases the complexity of index building.

[0004] Furthermore, in existing technologies, data deduplication and index maintenance are independent of each other and lack a linkage mechanism. When data access patterns change or data is updated, the system can only adjust the deduplication strategy or index structure separately and cannot achieve global optimization. For example, when the deduplication rate decreases, index redundancy increases, or when the index hit rate decreases, the deduplication logic is not optimized in conjunction with the deduplication process, resulting in the system performance not being able to maintain its optimal state. These problems limit the application effect of existing technologies in heterogeneous data management and efficient retrieval scenarios. Summary of the Invention

[0005] (a) Technical problems to be solved To address the shortcomings of existing technologies, this invention provides a computer data management system and method. By dynamically adjusting the sliding window and the comprehensive similarity threshold, it achieves accurate deduplication of multiple types of data and constructs a three-layer index structure linked with deduplication features. This solves the problems of low efficiency, fixed parameters, and maintenance difficulties caused by the separation of data deduplication and index construction in traditional methods, and significantly improves the storage efficiency and retrieval performance of heterogeneous data management.

[0006] (II) Technical Solution To achieve the above objectives, the present invention is implemented through the following technical solution: a computer data management method, comprising: reading a heterogeneous dataset and dividing it into text, time series and binary data groups according to data type; extracting the core fields of each data group and generating deduplication features and index features; performing standardization processing on the deduplication features; performing redundancy filtering on the index features; setting a sliding window based on data type and generating a window index feature summary. The sliding window is used to calculate the comprehensive similarity score between data blocks, filter the set of similar data blocks and determine the benchmark data block and redundant data blocks, cluster the benchmark data blocks to generate initial clusters, calculate the global deduplication rate and the effectiveness of cluster preprocessing, and reinitialize the sliding window if the global deduplication rate and the effectiveness of cluster preprocessing do not meet the preset conditions. Based on the initial cluster search popularity prediction, the cluster structure is optimized and cluster feature summaries are generated. A three-layer index structure including a global index, an intra-cluster index, and a feature index is constructed. The search efficiency is tested and the index level is optimized until the performance indicators are met.

[0007] Furthermore, the heterogeneous datasets are divided into text data groups, time-series data groups, and binary data groups according to data type. The core fields of the text data group include keywords, byte distribution, and topic tags. The core fields of the time-series data group include timestamps, numerical trends, and peak features. The core fields of the binary data group include file header identifiers, data segment lengths, and feature codes. The dual-track feature consists of deduplication features and index features. The deduplication feature of the text data block is a combination of keyword hash values ​​and byte sequence MD5 values, and the index feature is a combination of keyword frequency and topic tags. The deduplication feature of the time-series data block is a combination of numerical trend curves and adjacent data point difference sequences, and the index feature is a combination of statistical values ​​within a preset time window and peak features. The deduplication feature of the binary data block is a combination of file header identifiers and data segment hash values, and the index feature is a combination of file type and data segment feature codes.

[0008] Furthermore, the standardization processing of deduplication features includes: converting keywords in text data blocks to lowercase and deleting preset stop words; converting hash values ​​of binary data blocks to string format; and normalizing the numerical features of time-series data blocks to the [0, 1] interval. Redundancy filtering of index features includes: deleting low-frequency keywords in text data blocks and deleting invalid features in time-series data blocks. When the deduplication feature or index feature missing rate of a data block is greater than or equal to the missing threshold, it is marked as an abnormal data block and stored in an independent temporary directory. The initial sliding window size is dynamically set according to the data type. For each sliding window, the index features of all data blocks in the window are summarized to generate a unique window index feature summary for that window.

[0009] Further, the sliding distance of the sliding window is 1 data block or the data block corresponding to 1 preset time window, where the text and binary data group is 1 data block, and the time-series data group is the data block corresponding to 1 preset time window; the cosine similarity algorithm is used to calculate the deduplication feature similarity and the index feature similarity. After weighted summation of the deduplication feature similarity and the index feature similarity, the comprehensive similarity score of each pair of data blocks is calculated. The adjustment rules for the window size and the comprehensive similarity threshold T are as follows: if the proportion of data block pairs with a comprehensive similarity score ≥ T within the window ≥ M2, the window size is increased by 1 data block and T is increased by 0.05; if the proportion < M1, the window size is decreased by 1 data block and T is decreased by 0.05; if the proportion ≥ M1 and < M2, the window size and T remain unchanged.

[0010] Further, the set of similar data blocks is the data block pairs with a comprehensive similarity score ≥ the finally adjusted comprehensive similarity threshold T; the reference data block is the data block with the highest historical access frequency and 100% index feature integrity in the set of similar data blocks, which is stored in the core storage node and retains the complete dual-track features; the redundant data block only stores the difference information from the reference data block and the core storage address. The text data block records the byte modification position and content, the time-series data block records the numerical difference and the corresponding timestamp, the binary data block records the data segment modification flag and content, and the complete data body of the redundant data block is deleted but the complete index features are retained.

[0011] Further, the generation rule of the initial clustering clusters is as follows: the reference data blocks with an index feature similarity ≥ D2 are grouped into the same cluster, and the intersection of the index features of the reference data blocks within the cluster is extracted as the core index feature; if the index feature similarity between the reference data block and all the generated clusters is < D1, a new cluster is generated separately; the global deduplication rate is the ratio of the total storage volume of the redundant data blocks of all data groups to the total storage volume of the original data set, and the effectiveness of clustering preprocessing is the average value of the index feature similarities of the reference data blocks within all the initial clustering clusters; when the global deduplication rate < the deduplication rate threshold and the effectiveness of clustering preprocessing < the effectiveness threshold, return to re-initialize the sliding window and adjust the index feature weight.

[0012] Further, the retrieval heat prediction value is obtained by weighted summation of the historical access frequency of the reference data blocks within the cluster and the proportion of the historical hit frequency of the core index feature; the clustering cluster optimization rule is as follows: small clusters with a retrieval heat prediction value < Y1 and the number of reference data blocks within the cluster < 5 are merged with adjacent clusters with an index feature similarity ≥ D3; large clusters with a retrieval heat prediction value ≥ Y2 and the number of reference data blocks within the cluster ≥ 50 are subdivided according to the data type. The text data group is split according to the topic tags, the time-series data group is split according to the 24-hour time interval, and the binary data group is split according to the file type.

[0013] Furthermore, the cluster feature summary is generated by combining core identifier features and auxiliary retrieval features. The core identifier features are the intersection of the index features of all benchmark data blocks within the cluster. The auxiliary retrieval features include: the top 5 frequency of keywords and the mean byte distribution of text data clusters, the mean / variance of numerical features and the frequency of peak occurrence of time series data clusters, and the data segment length distribution range and feature code repetition rate of binary data clusters.

[0014] Furthermore, in the three-level index structure, the global index adopts a hybrid structure of hash table and ordered linked list. The hash table key is the hash value of the cluster feature summary, and the value is the cluster identifier, the number of data blocks, the search popularity prediction value, and the pointer to the storage address of the intra-cluster index. The ordered linked list is arranged in descending order according to the search popularity prediction value. The intra-cluster index adopts inverted index, B+ tree index, and hash index according to the data type. If the base data block associated with any index item has been deleted and there is no redundant data block associated with the index item, the index item is directly deleted, the storage space is released, and the index structure is updated. A computer data management system, comprising: The data preprocessing module reads the heterogeneous dataset and divides it into text, time series and binary data groups according to data type. It extracts the core fields of each data group and generates deduplication features and index features. It performs standardization processing on the deduplication features and redundancy filtering on the index features. It sets a sliding window based on data type and generates a window index feature summary. The clustering module calculates the comprehensive similarity score between data blocks using a sliding window, filters the set of similar data blocks and determines the benchmark data block and redundant data blocks, clusters the benchmark data blocks to generate initial clusters, calculates the global deduplication rate and the effectiveness of cluster preprocessing, and reinitializes the sliding window if the global deduplication rate and the effectiveness of cluster preprocessing do not meet the preset conditions. The index optimization module optimizes the cluster structure and generates cluster feature summaries based on the initial cluster search popularity prediction values. It constructs a three-layer index structure including a global index, an intra-cluster index, and a feature index, tests search efficiency, and optimizes the index hierarchy until the performance indicators are met.

[0015] (III) Beneficial Effects This invention provides a computer data management system and method, which has the following beneficial effects: (1) By dividing the heterogeneous dataset into text, time series and binary data groups according to data type, and extracting core fields to generate deduplication features and index features, efficient data classification and feature extraction are achieved. Standardization and redundancy filtering improve the consistency and quality of data, reduce the interference of invalid features, dynamically set sliding windows and generate window index feature summaries, laying the foundation for subsequent similarity calculation and clustering, significantly improving the accuracy and efficiency of data management, and providing an optimized data foundation for deduplication and index construction.

[0016] (2) By dynamically adjusting the sliding window size and the comprehensive similarity threshold, the set of similar data blocks is accurately screened, and the benchmark data block and redundant data block are determined, which effectively improves the accuracy and efficiency of data deduplication. The initial cluster is generated by clustering the benchmark data block, and the evaluation of the global deduplication rate and the effectiveness of cluster preprocessing is combined to ensure the optimization of data distribution and the improvement of storage efficiency. If the preset conditions are not met, the sliding window is re-initialized and the parameters are adjusted, which enhances the system's adaptive capability and provides a high-quality data foundation for subsequent index construction, significantly improving the overall performance of heterogeneous data management.

[0017] (3) The cluster structure is dynamically optimized by searching the heat prediction value, merging small clusters with low heat and splitting large clusters with high heat, generating cluster feature summaries containing core identifiers and auxiliary features, which significantly improves the accuracy and efficiency of data retrieval. The constructed three-layer index structure adapts to heterogeneous data by using index types such as mixed hash tables and B+ trees, enabling fast location and efficient query. Through continuous testing and optimization of the index level, the average retrieval response time is reduced and the accuracy is improved. At the same time, it supports high success rate reconstruction of redundant data, which improves the overall storage and retrieval performance of the system. Attached Figure Description

[0018] Figure 1 This is a schematic diagram illustrating the steps of the computer data management method of the present invention; Figure 2 This is a schematic diagram of the computer data management method of the present invention; Figure 3 This is a schematic diagram of the computer data management system structure of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Please see Figures 1-2 This invention provides a computer data management method, comprising the following steps: Step 1: Read the heterogeneous dataset and divide it into text, time series and binary data groups according to data type. Extract the core fields of each data group and generate deduplication features and index features. Perform standardization on the deduplication features and redundancy filtering on the index features. Set a sliding window based on data type and generate a window index feature summary. Step one includes the following: Step 101: The edge side reads the heterogeneous data set to be managed. The heterogeneous data set includes text data, time-series data, and binary data. The text data is like logs and documents, the time-series data is like sensor-acquired data, and the binary data is like pictures and executable files. The heterogeneous data set is divided into a text data group, a time-series data group, and a binary data group according to the data type. For the text data group, keywords, byte distribution, and topic tags are extracted as core fields. The byte distribution is like the occurrence frequency ratio of bytes from 0 to 255. For the time-series data group, timestamps, numerical trends, and peak features are extracted as core fields. For the binary data group, file header identifiers, data segment lengths, and feature codes are extracted as core fields; Step 102: For each data block in the text data group, the keyword hash value and the byte sequence MD5 value are calculated and combined as the deduplication feature, and the keyword occurrence frequency and the topic tag are counted and combined as the index feature. For each data block in the time-series data group, the numerical change trend curve and the difference sequence of adjacent data points are extracted and combined as the deduplication feature, and the statistical values within a preset time window and the peak features are calculated and combined as the index feature. The statistical values include the mean and variance, and the peak features include the peak size and the peak occurrence timestamp. For each data block in the binary data group, the file header identifier and the data segment hash value are extracted and combined as the deduplication feature, and the file type and the data segment feature code are determined and combined as the index feature. The deduplication feature and the index feature of each data block are bound to the metadata of the data block, forming a mapping relationship where the data block corresponds one-to-one with the dual-track features. The dual-track features are the deduplication feature and the index feature; Step 103: Standardize the deduplication features of all data blocks: Convert the keywords of text data blocks to lowercase uniformly, delete preset stop words such as "of, already, and", convert the hash values of binary data blocks to 64-bit string format uniformly, and normalize the numerical features of time-series data blocks to the interval [0, 1]. Perform redundant filtering on the index features of all data blocks: Delete low-frequency keywords with an occurrence frequency < 0.01% in text data blocks, and delete invalid features with a numerical fluctuation amplitude < 0.05 in time-series data blocks. Check the integrity of the dual-track features of each data block. If the missing rate of the deduplication feature or the index feature of a certain data block ≥ 10%, that is, the integrity of the dual-track features < 90%, then mark this data block as an abnormal data block and store it in a separate temporary directory without participating in the subsequent process; Step 104: Dynamically set the initial sliding window size based on data type: The sliding window for text data groups contains 5 consecutive data blocks, the sliding window for time-series data groups contains all data blocks corresponding to 3 consecutive preset time windows, and the sliding window for binary data groups contains 4 consecutive data segments; for each sliding window, summarize the index features of all data blocks within the window to generate a unique window index feature summary for that window; bind the window index feature summary to the corresponding sliding window to form a mapping relationship between the sliding window and the index feature summary.

[0021] When using this method, refer to steps 101 to 104: By dividing heterogeneous datasets into text, time-series, and binary data groups according to data type, and extracting core fields to generate deduplication and index features, efficient data classification and feature extraction are achieved. Standardization and redundancy filtering improve data consistency and quality, reduce interference from invalid features, and dynamically set sliding windows to generate window index feature summaries, laying the foundation for subsequent similarity calculations and clustering. This significantly improves the accuracy and efficiency of data management and provides an optimized data foundation for deduplication and index construction.

[0022] Step 2: Calculate the comprehensive similarity score between data blocks using a sliding window, filter the set of similar data blocks and determine the benchmark data block and redundant data blocks, cluster the benchmark data blocks to generate initial clusters, calculate the global deduplication rate and the effectiveness of cluster preprocessing, and reinitialize the sliding window if the global deduplication rate and the effectiveness of cluster preprocessing do not meet the preset conditions. Step two includes the following: Step 201: Starting from the beginning of the dataset of each data group, slide the data block by block according to the set initial sliding window size. Each slide distance is 1 data block or 1 data block corresponding to a preset time window. For text / binary data groups, it is 1 data block, and for time series data groups, it is 1 data block corresponding to a preset time window. For each data block in the current sliding window, perform dual-feature similarity calculation with other data blocks in the window. This includes using the cosine similarity algorithm to calculate the deduplication feature similarity and index feature similarity. After weighted summing of the deduplication feature similarity and index feature similarity, calculate the comprehensive similarity score for each pair of data blocks. The weight of the deduplication feature is 0.6 by default, and the weight of the index feature is 0.4 by default. Step 202: Preset an initial comprehensive similarity threshold T, defaulting to 0.85. Count the proportion of the number of data block pairs with a comprehensive similarity score ≥ T in the current sliding window to the total number of data block pairs in the window. If this proportion ≥ M2, defaulting to 30%, then expand the current window size by 1 data block, not exceeding the upper limit of 8 data blocks, and at the same time increase the comprehensive similarity threshold T by 0.05, not exceeding the upper limit of 0.95. If this proportion < M1, defaulting to 10%, then shrink the current window size by 1 data block, not lower than the lower limit of 2 data blocks, and at the same time decrease the comprehensive similarity threshold T by 0.05, not lower than the lower limit of 0.7. If this proportion ≥ M1 and < M2, then keep the current window size and the comprehensive similarity threshold T unchanged. Continue to slide and process subsequent data blocks according to the adjusted window size and comprehensive similarity threshold T. M1 and M2 are adjusted based on the actual situation. Step 203: After processing all sliding windows, filter out all data block pairs with a comprehensive similarity score ≥ the finally adjusted comprehensive similarity threshold T, and determine them as a set of similar data blocks. For each set of similar data blocks, filter out the data block with the highest historical access frequency and an index feature integrity of 100% as the reference data block. An integrity of 100% means no core fields are missing. Store it in the core storage node, retaining the complete dual-track feature. The remaining data blocks in the set are marked as redundant data blocks, only storing the difference information from the reference data block and the core storage address of the reference data block. For text data blocks, record the byte modification position and content; for time-series data blocks, record the numerical difference and the corresponding timestamp; for binary data blocks, record the data segment modification identifier and content. Delete the complete data body of the redundant data block, but retain its complete index feature. Step 204: Collect the reference data blocks of all data groups, and calculate the index feature similarity between each other based on the index features of the reference data blocks. Filter out the reference data blocks with an index feature similarity ≥ D2 and group them into the same initial clustering cluster. D2 defaults to 0.75. When setting the similarity threshold, ensure that similar reference data blocks are filtered out. Assign a unique cluster identifier to each initial clustering cluster, extract the intersection of the index features of all reference data blocks within the cluster as the core index feature of the cluster, count the total number of reference data blocks within the cluster, and associate and record the cluster identifier, core index feature, and number of data blocks to the clustering preprocessing log. If the index feature similarity of a reference data block to all the already generated initial clustering clusters is < D1, D1 defaults to 0.6, then generate a new initial clustering cluster for this reference data block alone, and supplement the record to the clustering preprocessing log according to the above rules. Step 205: Calculate the global deduplication rate and the clustering preprocessing effectiveness. The global deduplication rate is the ratio of the total storage volume of redundant data blocks in all data groups to the total storage volume of the original dataset in all data groups. The clustering preprocessing effectiveness is the average index feature similarity of the benchmark data blocks within all initial clusters. If the global deduplication rate is greater than or equal to the deduplication rate threshold of 25% and the clustering preprocessing effectiveness is greater than or equal to the effective threshold (the deduplication rate threshold is 25% by default and the effective threshold is 0.7 by default), proceed to step three. If the above conditions are not met, return to step 104 to reinitialize the sliding window size of each data group and adjust the dual feature weights. The index feature weights can be adjusted between 0.4 and 0.5. Repeat steps 201 to 205 until the conditions are met.

[0023] When using this method, refer to steps 201 to 205: By dynamically adjusting the sliding window size and comprehensive similarity threshold, the system accurately filters similar data block sets and identifies benchmark and redundant data blocks, effectively improving the accuracy and efficiency of data deduplication. Initial clusters are generated by clustering benchmark data blocks, and the evaluation of global deduplication rate and cluster preprocessing effectiveness ensures optimized data distribution and improved storage efficiency. If the preset conditions are not met, the sliding window is reinitialized and parameters are adjusted, enhancing the system's adaptability and providing a high-quality data foundation for subsequent index construction, significantly improving the overall performance of heterogeneous data management.

[0024] Step 3: Based on the initial cluster search popularity prediction, optimize the cluster structure and generate cluster feature summaries. Construct a three-layer index structure including a global index, an intra-cluster index, and a feature index. Test the search efficiency and optimize the index level until the performance indicators are met. Step three includes the following: Step 301: The cloud reads the generated clustering preprocessing log, extracts the cluster identifiers, core index features, and the number of benchmark data blocks within each initial cluster; based on the historical access frequency of each benchmark data block within the cluster and the proportion of the historical hit frequency of the core index feature, where the historical access frequency is the cumulative access count in the past 30 days and the proportion of the historical hit frequency is the total number of successfully matched retrieval requests with the core index feature of the cluster in all retrieval requests in the past 30 days. After normalization, calculate the predicted retrieval heat value for each initial cluster. The predicted retrieval heat value is obtained by weighted summation of the historical access frequency and the proportion of the historical hit frequency. The default weight of the historical access frequency is 0.6, and the default weight of the proportion of the historical hit frequency is 0.4; for small clusters where the predicted retrieval heat value < Y1 and the number of benchmark data blocks within the cluster < 5, Y1 is defaulted to 0.2, find adjacent clusters with an index feature similarity ≥ D3 and perform merging, D3 is defaulted to 0.8, and retain the intersection of the core index features and all benchmark data blocks of the merged cluster; for large clusters where the predicted retrieval heat value ≥ Y2 and the number of benchmark data blocks within the cluster ≥ 50, Y2 is defaulted to 0.8, and subdivide by data type: large clusters in the text data group are split by topic tags, large clusters in the time series data group are split by time intervals, with each 24 hours as a sub-interval, and large clusters in the binary data group are split by file types. Each split sub-cluster is re-assigned a unique cluster identifier and the core index features are updated; Step 302: For each index cluster optimized in Step 301, extract the intersection of all benchmark data block index features within the cluster as the core identification feature; count the distribution rules of the index features within the cluster: for text data clusters, count the top 5 frequencies of keyword occurrences and the mean byte distribution, for time series data clusters, count the mean / variance of numerical features and the peak occurrence frequency, for binary data clusters, count the distribution range of data segment lengths and the feature code repetition rate, and use the above distribution rules as auxiliary retrieval features; combine the core identification feature and the auxiliary retrieval features in the format of core identification feature + separator + auxiliary retrieval features to generate a unique cluster feature summary for each index cluster, and store it in the cluster index metadata table in an associated manner; Step 303: Construct the first-level global index: A hybrid structure of hash table and ordered linked list is adopted. The key of the hash table is the hash value of the cluster feature summary, and the value is the cluster identifier of the index cluster, the number of data blocks in the cluster, the search popularity prediction value, and the pointer to the storage address of the index in the cluster. The ordered linked list arranges the index clusters in descending order of the search popularity prediction value, supporting fast location of popular clusters. Construct the second-level intra-cluster index: The index clusters of text data groups adopt an inverted index structure. The key is the keyword, and the value is a list of storage addresses of the baseline data blocks containing the keyword and the address of the difference information of redundant data blocks. The index clusters of time-series data groups adopt a B+ tree index structure. The key is the timestamp, and the value is a list of storage addresses of data blocks corresponding to the timestamp. The index clusters of binary data groups adopt a hash index structure. The key is the data segment feature code, and the value is a list of storage addresses of the baseline data blocks of the corresponding data segment. Construct the third-level feature index: For the core identifier features of each index cluster, an independent feature index table is established. The table entries include the core feature value, the corresponding baseline data block storage address, the associated redundant data block difference information address, and the feature matching priority. The core identifier features have higher priority than the auxiliary retrieval features. Step 304: Collect a test dataset containing 1000 retrieval requests covering various data groups. Test and calculate the retrieval efficiency metrics for each index level: global index matching time, intra-cluster index location time, and feature index filtering time. Global index matching time is the time taken from the retrieval request to locating the target index cluster; intra-cluster index location time is the time taken from the index cluster to locating the base data block; and feature index filtering time is the time taken from the base data block to the target data. Optimize any level where any retrieval efficiency metric is greater than the metric threshold: for the global index, optimize the hash collision rate to ≤5% by adjusting the hash function parameters; for the intra-cluster index, cache the intra-cluster index of popular clusters with a retrieval popularity prediction value ≥0.7 in memory; for the feature index, delete entries corresponding to invalid features with a frequency <0.005. The metric threshold is 50ms by default. Traverse all index entries. If the base data block associated with any index entry has been deleted and there is no redundant data block associated with the index entry, delete the index entry directly, release storage space, and update the index structure. Step 305: Verify index performance using a standard retrieval test set covering text, time-series, and binary data types; statistically analyze core performance indicators: average retrieval response time, retrieval accuracy, and redundant data reconstruction success rate. Retrieval accuracy is the ratio of correctly matched retrieval results to the total number of retrieval requests, and redundant data reconstruction success rate is the ratio of successfully reconstructed redundant data to the total number of redundant data. If the average retrieval response time is ≤100ms, the retrieval accuracy is ≥98%, and the redundant data reconstruction success rate is ≥99.5%, then the index construction is complete. If any of the above indicators are not met, return to step 301 to adjust the cluster optimization parameters. The similarity threshold for merging small clusters can be lowered to 0.75, and the threshold for splitting large clusters can be lowered to 40 data blocks. Alternatively, optimize the index structure, such as replacing the B+ tree index with a hash index within the time-series data cluster. Repeat steps 301 to 305 until the performance requirements are met.

[0025] When using this method, refer to steps 301 to 305: By dynamically optimizing the cluster structure based on search popularity predictions, merging small clusters with low popularity and splitting large clusters with high popularity, cluster feature summaries containing core identifiers and auxiliary features are generated, significantly improving the accuracy and efficiency of data retrieval. The constructed three-layer index structure adapts to heterogeneous data through a hybrid hash table, B+ tree, and other index types, enabling fast location and efficient querying. Through continuous testing and optimization of the index hierarchy, the average retrieval response time is reduced and the accuracy is improved. At the same time, it supports high-success-rate reconstruction of redundant data, thereby improving the overall storage and retrieval performance of the system.

[0026] Please see Figure 3 The present invention also provides a computer data management system, comprising: a data preprocessing module, a clustering module, and an index optimization module, wherein: The data preprocessing module reads the heterogeneous dataset and divides it into text, time series and binary data groups according to data type. It extracts the core fields of each data group and generates deduplication features and index features. It performs standardization processing on the deduplication features and redundancy filtering on the index features. It sets a sliding window based on data type and generates a window index feature summary. The clustering module calculates the comprehensive similarity score between data blocks using a sliding window, filters the set of similar data blocks and determines the benchmark data block and redundant data blocks, clusters the benchmark data blocks to generate initial clusters, calculates the global deduplication rate and the effectiveness of cluster preprocessing, and reinitializes the sliding window if the global deduplication rate and the effectiveness of cluster preprocessing do not meet the preset conditions. The index optimization module optimizes the cluster structure and generates cluster feature summaries based on the initial cluster search popularity prediction values. It constructs a three-layer index structure including a global index, an intra-cluster index, and a feature index, tests search efficiency, and optimizes the index hierarchy until the performance indicators are met.

[0027] In the application, the various formulas mentioned are all calculated by removing dimensions and taking their numerical values. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The coefficients in the formulas are set by those skilled in the art according to the actual situation.

[0028] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented in software, the above embodiments can be implemented, in whole or in part, as a computer program product. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, and combinations thereof. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution.

[0029] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0030] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A computer data management method, characterized in that: include: Read heterogeneous datasets and divide them into text, time series and binary data groups according to data type. Extract the core fields of each data group and generate deduplication features and index features. Perform standardization on the deduplication features and redundancy filtering on the index features. Set a sliding window based on data type and generate window index feature summary. The comprehensive similarity score between data blocks is calculated using a sliding window. The cosine similarity algorithm is used to calculate the deduplication feature similarity and index feature similarity. The deduplication feature similarity and index feature similarity are weighted and summed to calculate the comprehensive similarity score for each pair of data blocks. A set of similar data blocks is selected and a benchmark data block and redundant data blocks are determined. The benchmark data block is the data block with the highest historical access frequency and 100% index feature completeness in the set of similar data blocks. The remaining data blocks in the set of similar data blocks are marked as redundant data blocks. Initial clusters are generated by clustering the benchmark data blocks. The global deduplication rate and cluster preprocessing effectiveness are calculated. If the global deduplication rate and cluster preprocessing effectiveness do not meet the preset conditions, the sliding window is reinitialized. Based on the initial cluster search popularity prediction, the cluster structure is optimized and cluster feature summaries are generated. A three-layer index structure including a global index, an intra-cluster index, and a feature index is constructed. The search efficiency is tested and the index level is optimized until the performance indicators are met.

2. The computer data management method according to claim 1, characterized in that: The heterogeneous datasets are divided into text data groups, time-series data groups, and binary data groups based on data type. The core fields of the text data group include keywords, byte distribution, and topic tags. The core fields of the time-series data group include timestamps, numerical trends, and peak characteristics. The core fields of the binary data group include file header identifiers, data segment lengths, and feature codes. The dual-track feature consists of deduplication features and index features. The deduplication feature for text data blocks is a combination of keyword hash values ​​and byte sequence MD5 values, and the index feature is a combination of keyword frequency and topic tags. The deduplication feature for time-series data blocks is a combination of numerical trend curves and the difference sequence of adjacent data points, and the index feature is a combination of statistical values ​​within a preset time window and peak characteristics. The deduplication feature for binary data blocks is a combination of file header identifiers and data segment hash values, and the index feature is a combination of file type and data segment feature codes.

3. The computer data management method according to claim 2, characterized in that: The standardization of deduplication features includes: converting keywords in text data blocks to lowercase and deleting preset stop words; converting hash values ​​in binary data blocks to string format; and normalizing numerical features of time-series data blocks to the [0, 1] interval. Redundancy filtering of index features includes: deleting low-frequency keywords in text data blocks and deleting invalid features in time-series data blocks. When the deduplication feature or index feature missing rate of a data block is greater than or equal to the missing threshold, it is marked as an abnormal data block and stored in an independent temporary directory. The initial sliding window size is dynamically set according to the data type. For each sliding window, the index features of all data blocks in the window are summarized to generate a unique window index feature summary for that window.

4. The computer data management method according to claim 1, characterized in that: The sliding distance of the sliding window is 1 data block or the data block corresponding to 1 preset time window. Among them, the text and binary data group is 1 data block, and the time series data group is the data block corresponding to 1 preset time window. The adjustment rules for the window size and the comprehensive similarity threshold T are as follows: If the proportion of data block pairs with a comprehensive similarity score ≥ T within the window ≥ M2, the window size is enlarged by 1 data block and T is increased by 0.05; if the proportion < M1, the window size is reduced by 1 data block and T is decreased by 0.05; if the proportion ≥ M1 and < M2, the window size and T remain unchanged.

5. A computer data management method according to claim 4, characterized in that: The set of similar data blocks is the data block pairs with a comprehensive similarity score ≥ the finally adjusted comprehensive similarity threshold T; the reference data block is stored in the core storage node and retains the complete dual-track feature; The redundant data block only stores the difference information from the reference data block and the core storage address. The text data block records the byte modification position and content. The time series data block records the numerical difference and the corresponding timestamp. The binary data block records the data segment modification identifier and content. The complete data body of the redundant data block is deleted but the complete index feature is retained.

6. The computer data management method according to claim 5, characterized in that: The generation rule of the initial clustering cluster is: The reference data blocks with an index feature similarity ≥ D2 are grouped into the same cluster, and the intersection of the index features of the reference data blocks within the cluster is extracted as the core index feature; if the index feature similarity between the reference data block and all the generated clusters is < D1, a new cluster is generated separately; the global deduplication rate is the ratio of the total storage volume of the redundant data blocks of all data groups to the total storage volume of the original data set, and the effectiveness of clustering preprocessing is the average value of the index feature similarities of the reference data blocks within all initial clustering clusters; When the global deduplication rate < the deduplication rate threshold and the effectiveness of clustering preprocessing < the effectiveness threshold, return to reinitialize the sliding window and adjust the index feature weights.

7. The computer data management method according to claim 1, characterized in that: The retrieval heat prediction value is obtained by weighted summation of the historical access frequencies of the reference data blocks within the cluster and the proportion of the historical hit frequencies of the core index feature. The clustering cluster optimization rule is: Small clusters with a retrieval heat prediction value < Y1 and the number of reference data blocks within the cluster < 5 are merged with adjacent clusters with an index feature similarity ≥ D3; Large clusters with a retrieval heat prediction value ≥ Y2 and the number of reference data blocks within the cluster ≥ 50 are subdivided according to the data type. The text data group is split according to the topic labels, the time series data group is split according to the 24-hour time interval, and the binary data group is split according to the file type.

8. A computer data management method according to claim 7, characterized in that: The cluster feature summary is generated by combining the core identification feature and the auxiliary retrieval feature. The core identification feature is the intersection of the index features of all reference data blocks within the cluster. The auxiliary retrieval features include: The top 5 keyword occurrence frequencies and the average byte distribution in the text data cluster, the mean / variance of the numerical features and the peak occurrence frequency in the time series data cluster, and the data segment length distribution range and the feature code repetition rate in the binary data cluster.

9. A computer data management method according to claim 8, characterized in that: In the three-level index structure, the global index adopts a hybrid structure of hash table and ordered linked list. The hash table key is the hash value of the cluster feature summary, and the value is the cluster identifier, the number of data blocks, the search popularity prediction value, and the pointer to the storage address of the intra-cluster index. The ordered linked list is arranged in descending order according to the search popularity prediction value. The intra-cluster index adopts inverted index, B+ tree index, and hash index according to the data type. If the base data block associated with any index item has been deleted and there is no redundant data block associated with the index item, the index item is directly deleted, the storage space is released, and the index structure is updated.

10. A computer data management system for implementing the method according to any one of claims 1 to 9, characterized in that: include: The data preprocessing module reads the heterogeneous dataset and divides it into text, time series and binary data groups according to data type. It extracts the core fields of each data group and generates deduplication features and index features. It performs standardization processing on the deduplication features and redundancy filtering on the index features. It sets a sliding window based on data type and generates a window index feature summary. The clustering module calculates the comprehensive similarity score between data blocks using a sliding window, filters the set of similar data blocks and determines the benchmark data block and redundant data blocks, clusters the benchmark data blocks to generate initial clusters, calculates the global deduplication rate and the effectiveness of cluster preprocessing, and reinitializes the sliding window if the global deduplication rate and the effectiveness of cluster preprocessing do not meet the preset conditions. The index optimization module optimizes the cluster structure and generates cluster feature summaries based on the initial cluster search popularity prediction values. It constructs a three-layer index structure including a global index, an intra-cluster index, and a feature index, tests search efficiency, and optimizes the index hierarchy until the performance indicators are met.

Images