Computer data management system and method

Abstract

The invention discloses a computer data management system and method, and relates to the technical field of data management, and the method comprises the steps: reading a heterogeneous data set, dividing the heterogeneous data set into text, time sequence and binary data groups according to data types, extracting a core field of each data group, and generating a deduplication feature and an index feature, setting a sliding window based on the data type and generating a window index feature abstract; calculating a comprehensive similarity score between the data blocks according to the sliding window, screening a similar data block set, determining a reference data block and a redundant data block, clustering the reference data block to generate an initial cluster, and if the global deduplication rate and the clustering preprocessing validity do not meet preset conditions, re-initializing the sliding window; according to the method, the cluster structure is optimized and the cluster feature abstract is generated based on the retrieval popularity predicted value of the initial clustering cluster, the three-layer index structure comprising the global index, the intra-cluster index and the feature index is constructed, the retrieval efficiency is tested, the index hierarchy is optimized, and the storage efficiency and the retrieval performance of heterogeneous data management are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data management, in particular to a computer data management system and method. BACKGROUND

[0002] In the existing computer data management system, data deduplication and index construction are common optimization methods. Data deduplication technology mainly identifies and eliminates duplicate or highly similar data blocks to reduce storage space occupation. Index construction improves data query efficiency by establishing an efficient data retrieval structure. Traditional methods usually use fixed parameter similarity comparison algorithms for deduplication, such as hash comparison, sliding window comparison, and index construction combined with inverted index, B+ tree structure. These technologies perform well on single data type or small-scale data sets and can meet basic storage and retrieval needs.

[0003] However, the existing technology has obvious limitations. First, data deduplication and index construction are usually performed independently. The deduplication process does not fully consider the needs of subsequent index construction, resulting in the need to extract features again during index construction, increasing computational overhead. At the same time, index construction does not fully utilize the features of deduplicated data, which can easily produce redundant indexes and reduce retrieval efficiency. Second, fixed parameter deduplication algorithms, such as fixed window size and similarity threshold, are difficult to adapt to the characteristics of multiple types of data, such as text, time series, and binary data, which can easily lead to missed or false positives. The distribution of deduplicated data is disordered, further increasing the complexity of index construction.

[0004] In addition, data deduplication and index maintenance in existing technology are independent of each other, lacking a linkage mechanism. When data access patterns change or data updates occur, the system can only adjust the deduplication strategy or index structure independently, and cannot achieve global optimization. For example, when the deduplication rate decreases, the index redundancy increases, or when the index hit rate decreases, the deduplication logic cannot be optimized, resulting in the system performance cannot be continuously maintained at the optimal state. These problems limit the application effect of existing technology in heterogeneous data management and efficient retrieval scenarios. SUMMARY

[0005] (I) Technical problems solved To overcome the shortcomings of the prior art, the present application provides a computer data management system and method, which realizes accurate deduplication of multiple types of data by dynamically adjusting the sliding window and comprehensive similarity threshold, and constructs a three-layer index structure linked with deduplication features. The present application solves the problems of low efficiency, parameter fixation, and maintenance difficulty caused by the separation of data deduplication and index construction in traditional methods, and significantly improves the storage efficiency and retrieval performance of heterogeneous data management.

[0006] (II) Technical solutions To achieve the above object, the present application is implemented by the following technical solutions: A computer data management method, comprising: reading a heterogeneous data set and dividing it into text, time series and binary data groups according to data types, extracting core fields of each data group and generating deduplication features and index features, performing standardization processing on the deduplication features, performing redundancy filtering on the index features, setting a sliding window based on the data types and generating a window index feature summary; calculating the comprehensive similarity score between data blocks in the sliding window, screening a similar data block set and determining a reference data block and a redundant data block, clustering the reference data block to generate an initial cluster, calculating the global deduplication rate and the clustering preprocessing effectiveness, and reinitializing the sliding window if the global deduplication rate and the clustering preprocessing effectiveness do not meet the preset conditions; Based on the search heat prediction value of the initial cluster, the cluster structure is optimized and a cluster feature summary is generated, a three-layer index structure including global index, intra-cluster index and feature index is constructed, the search efficiency is tested and the index level is optimized until the performance indicators are met.

[0007] Further, the heterogeneous data set is divided into text data group, time series data group and binary data group according to data types, the core fields of the text data group include keywords, byte distribution and topic label, the core fields of the time series data group include timestamp, numerical trend and peak feature, and the core fields of the binary data group include file header identifier, data segment length and feature code; The double-track feature is composed of deduplication features and index features, the deduplication features of the text data block are the combination of keyword hash value and byte sequence MD5 value, and the index features are the combination of keyword frequency and topic label; The deduplication features of the time series data block are the combination of numerical change trend curve and adjacent data point difference sequence, and the index features are the combination of statistical value in the preset time window and peak feature; The deduplication features of the binary data block are the combination of file header identifier and data segment hash value, and the index features are the combination of file type and data segment feature code.

[0008] Further, the standardization processing of the deduplication features includes: converting the keywords of the text data block to lowercase and deleting the preset stop words, converting the hash value of the binary data block to string format, and normalizing the numerical features of the time series data block to the interval [0, 1]; The redundancy filtering of the index features includes: deleting low-frequency keywords in the text data block and deleting invalid features in the time series data block; When the missing rate of the deduplication features or the index features of the data block is greater than or equal to the missing threshold, mark it as an abnormal data block and store it in a separate temporary directory; The size of the initial sliding window is dynamically set according to the data types, for each sliding window, the index features of all data blocks in the window are summarized to generate a unique window index feature summary of the window.

[0009] Further, the sliding distance of the sliding window is 1 data block or a data block corresponding to a preset time window, wherein the text and binary data group is 1 data block, and the time series data group is a data block corresponding to a preset time window; the cosine similarity algorithm is used to calculate the deduplication feature similarity and the index feature similarity, the comprehensive similarity score of each pair of data blocks is calculated after the deduplication feature similarity and the index feature similarity are weighted and summed, and the adjustment rule of the window size and the comprehensive similarity threshold T is: if the proportion of the data block pairs with the comprehensive similarity score ≥ T in the window is ≥ M2, the window size is expanded by 1 data block and T is increased by 0.05; if the proportion is < M1, the window size is reduced by 1 data block and T is reduced by 0.05; if the proportion is ≥ M1 and < M2, the window size and T remain unchanged.

[0010] Further, the similar data block set is a pair of data blocks with a comprehensive similarity score ≥ the final adjusted comprehensive similarity threshold T; the reference data block is a data block with the highest historical access frequency and an index feature completeness of 100% in the similar data block set, which is stored in the core storage node and retains complete double-track features; the redundant data block only stores the difference information and the core storage address of the reference data block, the text data block records the byte modification position and content, the time series data block records the value difference and the corresponding timestamp, and the binary data block records the data segment modification identifier and content. The complete data body of the redundant data block is deleted but the complete index feature is retained.

[0011] Further, the generation rule of the initial clustering cluster is: the reference data blocks with an index feature similarity ≥ D2 are classified into the same cluster, and the intersection of the index features of the reference data blocks in the cluster is extracted as the core index feature; if the index feature similarity of the reference data block and all generated clusters is < D1, a new cluster is generated separately; the global deduplication rate is the ratio of the total storage volume of the redundant data blocks of all data groups to the total storage volume of the original data set, and the clustering preprocessing effectiveness is the average of the index feature similarities of the reference data blocks in all initial clustering clusters; when the global deduplication rate < the deduplication rate threshold and the clustering preprocessing effectiveness < the effective threshold, the sliding window is reinitialized, and the index feature weight is adjusted.

[0012] Further, the retrieval heat prediction value is obtained by weighted summation of the historical access frequency of the reference data block in the cluster and the historical hit frequency proportion of the core index feature; the clustering cluster optimization rule is: the small cluster with a retrieval heat prediction value < Y1 and a number of reference data blocks in the cluster < 5 is merged with an adjacent cluster with an index feature similarity ≥ D3; the large cluster with a retrieval heat prediction value ≥ Y2 and a number of reference data blocks in the cluster ≥ 50 is subdivided according to data types, the text data group is split according to the theme label, the time series data group is split according to the 24-hour time interval, and the binary data group is split according to the file type.

[0013] Further, the cluster feature abstract is generated by combining the core identification feature and the auxiliary retrieval feature, the core identification feature is the intersection of the index features of all reference data blocks in the cluster, and the auxiliary retrieval feature includes: Top 5 keyword appearance frequency and byte distribution mean of the text data cluster, numerical feature mean / variance and peak appearance frequency of the time series data cluster, data segment length distribution range and feature code repetition rate of the binary data cluster.

[0014] Further, in the three-layer index structure, the global index adopts a hybrid structure of a hash table and an ordered list, the hash table key is the cluster feature abstract hash value, the value is the cluster identifier, the data block quantity, the retrieval heat prediction value and the cluster internal index storage address pointer, and the ordered list is arranged in descending order according to the retrieval heat prediction value; the cluster internal index adopts an inverted index, a B+ tree index and a hash index according to the data type; if any index item associated with the reference data block has been deleted and there is no redundant data block associated with the index item, the index item is directly deleted, the storage space is released and the index structure is updated; A computer data management system comprises: A data preprocessing module reads the heterogeneous data set and divides it into text, time series and binary data groups according to the data type, extracts the core fields of each data group and generates the deduplication features and index features, performs standardization processing on the deduplication features, performs redundancy filtering on the index features, sets a sliding window based on the data type and generates a window index feature abstract; A clustering module calculates the comprehensive similarity score between data blocks according to the sliding window, filters the similar data block set and determines the reference data block and the redundant data block, clusters the reference data block to generate an initial clustering cluster, calculates the global deduplication rate and the clustering preprocessing effectiveness, and reinitializes the sliding window if the global deduplication rate and the clustering preprocessing effectiveness do not meet the preset conditions; An index optimization module optimizes the cluster structure and generates the cluster feature abstract based on the retrieval heat prediction value of the initial clustering cluster, constructs a three-layer index structure including the global index, the cluster internal index and the feature index, tests the retrieval efficiency and optimizes the index level until the performance indicators are met.

[0015] (Three) beneficial effects The application provides a computer data management system and method, which has the following beneficial effects: (1) By dividing the heterogeneous data set into text, time series and binary data groups according to the data type, and extracting the core fields to generate the deduplication features and index features, efficient classification and feature extraction of data are realized, the standardization processing and redundancy filtering improve the consistency and quality of the data, reduce the interference of invalid features, a sliding window is dynamically set and a window index feature abstract is generated, which lays a foundation for subsequent similarity calculation and clustering, significantly improves the accuracy and efficiency of data management, and provides an optimized data basis for deduplication and index construction.

[0016] (2) By dynamically adjusting the size of the sliding window and the comprehensive similarity threshold, the similar data block set is accurately screened, and the reference data block and the redundant data block are determined, which effectively improves the accuracy and efficiency of data deduplication. By clustering the reference data block to generate an initial cluster, and combining the global deduplication rate and the evaluation of the effectiveness of the cluster preprocessing, the data distribution optimization and the storage efficiency improvement are ensured. If the preset condition is not met, the sliding window is reinitialized and the parameters are adjusted, which enhances the adaptive ability of the system and provides a high-quality data basis for subsequent index construction, significantly improving the overall performance of heterogeneous data management.

[0017] (3) By searching the heat prediction value to dynamically optimize the cluster structure, merging small clusters with low heat and splitting large clusters with high heat, a cluster feature abstract containing core identifiers and auxiliary features is generated, which significantly improves the accuracy and efficiency of data retrieval. The three-layer index structure constructed by mixed hash table, B+ tree and other index types adapts to heterogeneous data, realizes fast positioning and efficient query, reduces the average retrieval response time and improves the accuracy through continuous testing and optimization of the index level, while supporting high success rate reconstruction of redundant data, which overall improves the storage and retrieval performance of the system. BRIEF DESCRIPTION OF DRAWINGS

[0018] Fig. 1 The figure is a schematic diagram of the steps of the computer data management method of the present application. Fig. 2 The figure is a schematic diagram of the flow of the computer data management method of the present application. Fig. 3 The figure is a schematic diagram of the structure of the computer data management system of the present application. DETAILED DESCRIPTION

[0019] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0020] Please refer to Figs. 1-2 The present application provides a computer data management method, comprising the following steps: Step 1: Read the heterogeneous data set and divide it into text, time series and binary data groups according to the data type, extract the core fields of each data group and generate deduplication features and index features, perform standardization processing on the deduplication features and perform redundancy filtering on the index features, set the sliding window based on the data type and generate a window index feature abstract; The step 1 includes the following contents: Step 101: The edge end reads the heterogeneous data set to be managed, which contains text data, time series data, and binary data, such as logs, documents, sensor collected data, and pictures, executable files; the heterogeneous data set is divided into text data group, time series data group, and binary data group according to data types; the text data group is extracted as core fields, such as keywords, byte distribution, and topic labels, the byte distribution is the frequency ratio of 0-255 bytes, the time series data group is extracted as core fields, such as timestamp, numerical trend, and peak characteristics, and the binary data group is extracted as core fields, such as file header identifier, data segment length, and feature code; Step 102: For each data block in the text data group, calculate the keyword hash value and MD5 value of the byte sequence and combine them as the deduplication feature, and count the keyword frequency and topic label and combine them as the index feature; for each data block in the time series data group, extract the numerical trend curve and adjacent data point difference sequence and combine them as the deduplication feature, calculate the statistical value and peak feature in the preset time window and combine them as the index feature, the statistical value includes mean and variance, and the peak feature includes peak value and peak timestamp; for each data block in the binary data group, extract the file header identifier and data segment hash value and combine them as the deduplication feature, and determine the file type and data segment feature code and combine them as the index feature; bind the deduplication feature and index feature of each data block to the metadata of the data block to form a one-to-one mapping relationship between the data block and the double-track feature, that is, the deduplication feature and the index feature; Step 103: Perform standardization processing on the deduplication features of all data blocks: convert the keywords of the text data block to lowercase format, delete the preset stop words such as "of, have, and", convert the hash values of the binary data block to 64-bit string format, and normalize the numerical features of the time series data block to the [0, 1] interval; perform redundancy filtering on the index features of all data blocks: delete low-frequency keywords with a frequency <0.01% in the text data block, and delete invalid features with a numerical fluctuation amplitude <0.05 in the time series data block; check the integrity of the double-track features of each data block, if the missing rate of the deduplication feature or the index feature of a certain data block is ≥10%, that is, the integrity of the double-track features is <90%, mark the data block as an abnormal data block and store it in a separate temporary directory, which does not participate in the subsequent process; Step 104: dynamically setting the initial sliding window size based on the data type: the sliding window of the text data set contains 5 consecutive data blocks, the sliding window of the time series data set contains all data blocks corresponding to 3 consecutive preset time windows, and the sliding window of the binary data set contains 4 consecutive data segments; for each sliding window, the index features of all data blocks in the window are summarized to generate a unique window index feature summary for the window; the window index feature summary is bound to the corresponding sliding window to form a mapping relationship between the sliding window and the index feature summary.

[0021] In use, the contents of steps 101 to 104 are combined: By dividing the heterogeneous data set into text, time series and binary data sets according to the data type, and extracting the core field to generate the deduplication feature and the index feature, efficient classification and feature extraction of the data are realized, the consistency and quality of the data are improved by standardization processing and redundancy filtering, the interference of invalid features is reduced, the sliding window is dynamically set and the window index feature summary is generated, which lays a foundation for subsequent similarity calculation and clustering, significantly improves the accuracy and efficiency of data management, and provides an optimized data basis for deduplication and index construction.

[0022] Step two: calculating the comprehensive similarity score between data blocks according to the sliding window, screening the similar data block set and determining the reference data block and the redundant data block, clustering the reference data block to generate the initial clustering cluster, calculating the global deduplication rate and the clustering preprocessing effectiveness, and if the global deduplication rate and the clustering preprocessing effectiveness do not meet the preset conditions, reinitializing the sliding window; The step two includes the following contents: Step 201: according to the set initial sliding window size, starting from the starting position of the data set of each data group, sliding block by block, the sliding distance each time is 1 data block or 1 data block corresponding to a preset time window, 1 data block for text / binary data group, and 1 data block corresponding to a preset time window for time series data group; for each data block in the current sliding window, double-feature similarity calculation is performed with other data blocks in the window, including calculating the deduplication feature similarity and the index feature similarity using the cosine similarity algorithm, and calculating the comprehensive similarity score of each pair of data blocks after weighted summation of the deduplication feature similarity and the index feature similarity, the weight of the deduplication feature is 0.6 by default, and the weight of the index feature is 0.4 by default; Step 202: preset an initial comprehensive similarity threshold T, defaulting to 0.85, and count the proportion of the number of data block pairs with a comprehensive similarity score greater than or equal to T in the total number of data block pairs in the current sliding window; if the proportion is greater than or equal to M2, defaulting to 30%, the current window size is expanded by 1 data block, not exceeding the upper limit of 8 data blocks, and the comprehensive similarity threshold T is increased by 0.05, not exceeding the upper limit of 0.95; if the proportion is less than M1, defaulting to 10%, the current window size is reduced by 1 data block, not less than the lower limit of 2 data blocks, and the comprehensive similarity threshold T is reduced by 0.05, not less than the lower limit of 0.7; if the proportion is greater than M1 and less than M2, the current window size and the comprehensive similarity threshold T remain unchanged; continue to process the subsequent data blocks according to the adjusted window size and the comprehensive similarity threshold T, and M1 and M2 are adjusted based on the actual situation; Step 203: after processing all sliding windows, filter out all data block pairs with a comprehensive similarity score greater than or equal to the final adjusted comprehensive similarity threshold T, and determine them as a similar data block set; for each similar data block set, filter out the data block with the highest historical access frequency and an index feature completeness of 100% as the reference data block, and store it in the core storage node, retaining the complete dual-track feature; mark the remaining data blocks in the set as redundant data blocks, and only store the difference information with the reference data block and the core storage address of the reference data block, the byte modification position and content of the text data block, the value difference and corresponding timestamp of the time series data block, and the data segment modification identifier and content of the binary data block, and delete the complete data body of the redundant data block, but retain its complete index feature; Step 204: collect all reference data blocks of all data groups, calculate the index feature similarity between each other based on the index features of the reference data blocks; filter out reference data blocks with an index feature similarity greater than or equal to D2 as the same initial clustering cluster, D2 defaulting to 0.75, and the similarity threshold is set to ensure that similar reference data blocks are selected, assign a unique cluster identifier to each initial clustering cluster, extract the intersection of the index features of all reference data blocks in the cluster as the core index feature of the cluster, and record the cluster identifier, core index feature, and data block quantity to the clustering preprocessing log; if the index feature similarity between the reference data block and all generated initial clustering clusters is less than D1, D1 defaulting to 0.6, a new initial clustering cluster is generated for the reference data block, and the above rules are supplemented to the clustering preprocessing log; Step 205: Calculate the global deduplication rate and the clustering preprocessing effectiveness, the global deduplication rate is the ratio of the total storage volume of redundant data blocks of all data groups to the total storage volume of original data sets of all data groups, and the clustering preprocessing effectiveness is the average similarity of index features of reference data blocks in all initial clustering clusters; if the global deduplication rate is greater than or equal to a deduplication rate threshold of 25% and the clustering preprocessing effectiveness is greater than or equal to an effective threshold, the deduplication rate threshold is 25% by default, and the effective threshold is 0.7 by default, then go to Step Three; if the above conditions are not met, return to Step 104 to re-initialize the sliding window size of each data group, and adjust the double feature weight at the same time, the index feature weight can be adjusted between 0.4 and 0.5, and Steps 201 to 205 are repeated until the conditions are met.

[0023] In use, in combination with the contents of Steps 201 to 205: By dynamically adjusting the sliding window size and the comprehensive similarity threshold, the similar data block set is accurately screened, and the reference data block and the redundant data block are determined, which effectively improves the accuracy and efficiency of data deduplication. By clustering the reference data blocks to generate initial clustering clusters, and combining the evaluation of the global deduplication rate and the clustering preprocessing effectiveness, the data distribution optimization and storage efficiency improvement are ensured. If the preset conditions are not met, the sliding window is re-initialized and the parameters are adjusted, which enhances the adaptive ability of the system and provides a high-quality data basis for subsequent index construction, significantly improving the overall performance of heterogeneous data management.

[0024] Step Three: Based on the search heat prediction value of the initial clustering cluster, optimize the cluster structure and generate a cluster feature abstract, construct a three-layer index structure including global index, intra-cluster index and feature index, test the search efficiency and optimize the index level until the performance indicators are met. The Step Three includes the following contents: Step 301: The cloud reads the generated cluster preprocessing log, extracts the cluster identification, core index feature and the number of reference data blocks in each initial cluster, and calculates the search heat prediction value of each initial cluster based on the historical access frequency of each reference data block in the cluster and the historical hit frequency proportion of the core index feature. The historical access frequency is the cumulative access frequency in the past 30 days, and the historical hit frequency proportion is the total number of successful matches with the cluster core index feature in all search requests in the past 30 days. After normalization, the search heat prediction value is obtained by weighted summation of the search historical access frequency and the historical hit frequency proportion. The weight of the historical access frequency is 0.6 by default, and the weight of the historical hit frequency proportion is 0.4 by default. For small clusters with search heat prediction value < Y1 and number of reference data blocks in the cluster < 5, Y1 is 0.2 by default, find adjacent clusters with index feature similarity ≥ D3 and perform merging, D3 is 0.8 by default, and keep the intersection of the core index feature and all reference data blocks of the merged cluster. For large clusters with search heat prediction value ≥ Y2 and number of reference data blocks in the cluster ≥ 50, Y2 is 0.8 by default, and is divided according to data type: large clusters of text data group are split by topic label, large clusters of time series data group are split by time interval, and each 24 hours is a sub-interval, and large clusters of binary data group are split by file type. Each split sub-cluster is assigned a unique cluster identification and the core index feature is updated. Step 302: For each index cluster optimized in step 301, extract the intersection of all reference data block index features in the cluster as the core identification feature. Statistics the distribution of index features in the cluster: text data cluster statistics keyword appearance frequency ranking Top5, byte distribution mean, time series data cluster statistics numerical feature mean / variance, peak frequency, binary data cluster statistics data segment length distribution range, feature code repetition rate. The above distribution rules are used as auxiliary search features. Combine the core identification feature and the auxiliary search feature in the format of core identification feature + separator + auxiliary search feature to generate a unique cluster feature summary for each index cluster, and store it in the cluster index metadata table. Step 303: Constructing the first layer global index: using a hybrid structure of hash table + ordered linked list, the key of the hash table is the hash value of the cluster feature abstract, and the value is the cluster identifier of the index cluster, the number of data blocks in the cluster, the search popularity prediction value and the index storage address pointer in the cluster; the ordered linked list arranges the index clusters in descending order of search popularity prediction value, supporting fast positioning of popular clusters; constructing the second layer intra-cluster index: the index cluster of the text data group adopts an inverted index structure, the key is the keyword, and the value is the storage address list of the reference data block containing the keyword and the redundant data block difference information address; the index cluster of the time series data group adopts a B+ tree index structure, the key is the timestamp, and the value is the data block storage address list corresponding to the timestamp; the index cluster of the binary data group adopts a hash index structure, the key is the data segment feature code, and the value is the reference data block storage address list corresponding to the data segment; constructing the third layer feature index: for each core identification feature of the index cluster, an independent feature index table is established, and the table entry contains the core feature value, the corresponding reference data block storage address, the associated redundant data block difference information address and the feature matching priority, and the priority of the core identification feature is higher than that of the auxiliary search feature; Step 304: Collecting a test data set containing 1000 search requests covering various data groups, testing and calculating the search efficiency indicators of each index level: global index matching time, intra-cluster index positioning time, and feature index screening time. The global index matching time is the time spent from the search request to locating the target index cluster, the intra-cluster index positioning time is the time spent from the index cluster to locating the reference data block, and the feature index screening time is the time spent from the reference data block to screening the target data. For any index level with an indicator > indicator threshold, perform optimization: adjust the hash function parameters of the global index to optimize the hash collision rate to ≤ 5%, cache the intra-cluster index of the popular cluster with a search popularity prediction value ≥ 0.7 to the memory; delete the table entries corresponding to the invalid features with an occurrence frequency < 0.005, and the indicator threshold is 50 ms by default; traverse all index entries, if the reference data block associated with any index entry has been deleted and there is no redundant data block associated with the index entry, then directly delete the index entry, release the storage space and update the index structure; Step 305: Verify the index performance by using the standard retrieval test set covering text, time series, and binary data types; calculate the core performance indicators: average retrieval response time, retrieval accuracy, and redundant data reconstruction success rate; the retrieval accuracy is the ratio of the number of correct retrieval results to the total number of retrieval requests, and the redundant data reconstruction success rate is the ratio of the number of successfully reconstructed redundant data to the total number of redundant data; if the average retrieval response time is less than or equal to 100 ms, the retrieval accuracy is greater than or equal to 98%, and the redundant data reconstruction success rate is greater than or equal to 99.5%, the index construction is completed; if any of the above indicators is not met, return to step 301 to adjust the clustering cluster optimization parameters, the small cluster merging similarity threshold can be lowered to 0.75, the large cluster splitting threshold can be lowered to 40 data blocks, or the index structure can be optimized, such as replacing the B+ tree index with a hash index in the time series cluster, and repeating steps 301 to 305 until the performance requirements are met.

[0025] In use, in combination with the contents of steps 301 to 305: The clustering cluster structure is dynamically optimized by retrieval heat prediction value, low-heat small clusters are merged and high-heat large clusters are split, cluster feature abstracts containing core identifiers and auxiliary features are generated, the accuracy and efficiency of data retrieval are significantly improved, the three-layer index structure is constructed to adapt to heterogeneous data by mixing index types such as hash table and B+ tree, fast positioning and efficient query are achieved, the average retrieval response time is reduced and the accuracy is improved through continuous testing and optimization of index levels, while supporting high-success-rate reconstruction of redundant data, and the overall storage and retrieval performance of the system is improved.

[0026] Please refer to Fig. 3 The application also provides a computer data management system, comprising: a data preprocessing module, a clustering module, and an index optimization module, wherein: The data preprocessing module reads the heterogeneous data set and divides it into text, time series, and binary data groups according to data types, extracts core fields of each data group and generates deduplication features and index features, performs standardization processing on the deduplication features and redundancy filtering on the index features, sets a sliding window based on the data types and generates a window index feature abstract; The clustering module calculates the comprehensive similarity score between data blocks according to the sliding window, filters similar data block sets and determines the reference data block and the redundant data block, clusters the reference data block to generate an initial clustering cluster, calculates the global deduplication rate and the clustering preprocessing effectiveness, and reinitializes the sliding window if the global deduplication rate and the clustering preprocessing effectiveness do not meet the preset conditions; The index optimization module optimizes the cluster structure and generates a cluster feature abstract based on the retrieval heat prediction value of the initial clustering cluster, constructs a three-layer index structure containing global index, intra-cluster index, and feature index, tests the retrieval efficiency and optimizes the index level until the performance indicators are met.

[0027] In the application, the several formulas involved are calculated by taking the numerical value after de-dimensioning, and the formula is obtained by software simulation of a large number of collected data to reflect the most real situation. The coefficients in the formula are set by the person skilled in the art according to the actual situation.

[0028] The above embodiments can be implemented wholly or partially by software, hardware, firmware, or any other combination. When implemented by software, the above embodiments can be implemented wholly or partially in the form of a computer program product. Those skilled in the art can realize that the units and algorithm steps of the examples described in connection with the embodiments disclosed herein can be realized by electronic hardware, computer software, and a combination of electronic hardware and computer software. Whether the functions are executed in hardware or software depends on the specific application and design constraints of the technical solutions.

[0029] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, which can be located in one place or distributed on multiple network units. Part or all of the units can be selected to achieve the purpose of the embodiments according to actual needs.

[0030] The above is merely a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application.

Claims

1. A computer data management method, characterized in that: include: Read heterogeneous datasets and divide them into text, time series and binary data groups according to data type. Extract the core fields of each data group and generate deduplication features and index features. Perform standardization on the deduplication features and redundancy filtering on the index features. Set a sliding window based on data type and generate window index feature summary. The sliding window is used to calculate the comprehensive similarity score between data blocks, filter the set of similar data blocks and determine the benchmark data block and redundant data blocks, cluster the benchmark data blocks to generate initial clusters, calculate the global deduplication rate and the effectiveness of cluster preprocessing, and reinitialize the sliding window if the global deduplication rate and the effectiveness of cluster preprocessing do not meet the preset conditions. Based on the initial cluster search popularity prediction, the cluster structure is optimized and cluster feature summaries are generated. A three-layer index structure including a global index, an intra-cluster index, and a feature index is constructed. The search efficiency is tested and the index level is optimized until the performance indicators are met.

2. The computer data management method according to claim 1, characterized in that: The heterogeneous datasets are divided into text data groups, time-series data groups, and binary data groups based on data type. The core fields of the text data group include keywords, byte distribution, and topic tags. The core fields of the time-series data group include timestamps, numerical trends, and peak characteristics. The core fields of the binary data group include file header identifiers, data segment lengths, and feature codes. The dual-track feature consists of deduplication features and index features. The deduplication feature for text data blocks is a combination of keyword hash values ​​and byte sequence MD5 values, and the index feature is a combination of keyword frequency and topic tags. The deduplication feature for time-series data blocks is a combination of numerical trend curves and the difference sequence of adjacent data points, and the index feature is a combination of statistical values ​​within a preset time window and peak characteristics. The deduplication feature for binary data blocks is a combination of file header identifiers and data segment hash values, and the index feature is a combination of file type and data segment feature codes.

3. The computer data management method according to claim 2, characterized in that: The standardization of deduplication features includes: converting keywords in text data blocks to lowercase and deleting preset stop words; converting hash values ​​in binary data blocks to string format; and normalizing numerical features of time-series data blocks to the [0, 1] interval. Redundancy filtering of index features includes: deleting low-frequency keywords in text data blocks and deleting invalid features in time-series data blocks. When the deduplication feature or index feature missing rate of a data block is greater than or equal to the missing threshold, it is marked as an abnormal data block and stored in an independent temporary directory. The initial sliding window size is dynamically set according to the data type. For each sliding window, the index features of all data blocks in the window are summarized to generate a unique window index feature summary for that window.

4. The computer data management method according to claim 1, characterized in that: The sliding distance of the sliding window is 1 data block or the data block corresponding to 1 preset time window. Among them, the text and binary data group is 1 data block, and the time-series data group is the data block corresponding to 1 preset time window. The cosine similarity algorithm is used to calculate the deduplication feature similarity and the index feature similarity. After weighted summation of the deduplication feature similarity and the index feature similarity, the comprehensive similarity score of each pair of data blocks is calculated. The adjustment rules for the window size and the comprehensive similarity threshold T are as follows: If the proportion of data block pairs with a comprehensive similarity score ≥ T within the window ≥ M2, the window size is increased by 1 data block and T is increased by 0.05; if the proportion < M1, the window size is reduced by 1 data block and T is decreased by 0.05; if the proportion ≥ M1 and < M2, the window size and T remain unchanged.

5. A computer data management method according to claim 4, characterized in that: The set of similar data blocks is the data block pairs with a comprehensive similarity score ≥ the finally adjusted comprehensive similarity threshold T; the reference data block is the data block with the highest historical access frequency and a 100% index feature integrity in the set of similar data blocks, which is stored in the core storage node and the complete dual-track feature is retained; The redundant data block only stores the difference information from the reference data block and the core storage address. The text data block records the byte modification position and content. The time-series data block records the numerical difference and the corresponding timestamp. The binary data block records the data segment modification identifier and content. The complete data body of the redundant data block is deleted but the complete index feature is retained.

6. The computer data management method according to claim 5, characterized in that: The generation rule of the initial clustering cluster is as follows: The reference data blocks with an index feature similarity ≥ D2 are grouped into the same cluster, and the intersection of the index features of the reference data blocks within the cluster is extracted as the core index feature; if the index feature similarity between the reference data block and all the already generated clusters is < D1, a new cluster is generated separately. The global deduplication rate is the ratio of the total storage volume of the redundant data blocks of all data groups to the total storage volume of the original data set. The effectiveness of clustering preprocessing is the average value of the index feature similarities of the reference data blocks within all initial clustering clusters; When the global deduplication rate < the deduplication rate threshold and the effectiveness of clustering preprocessing < the effectiveness threshold, return to reinitialize the sliding window and adjust the index feature weight.

7. The computer data management method according to claim 1, characterized in that: The retrieval popularity prediction value is obtained by weighted summation of the historical access frequency of the reference data blocks within the cluster and the proportion of the historical hit frequency of the core index feature. The clustering cluster optimization rule is as follows: Small clusters with a retrieval popularity prediction value < Y1 and the number of reference data blocks within the cluster < 5 are merged with adjacent clusters with an index feature similarity ≥ D3; Large clusters with a retrieval popularity prediction value ≥ Y2 and the number of reference data blocks within the cluster ≥ 50 are subdivided according to the data type. The text data group is split according to the topic labels. The time-series data group is split according to the 24-hour time interval. The binary data group is split according to the file type.

8. A computer data management method according to claim 7, characterized in that: The cluster feature summary is generated by combining the core identification feature and the auxiliary retrieval feature. The core identification feature is the intersection of the index features of all reference data blocks within the cluster. The auxiliary retrieval features include: The top 5 keyword occurrence frequencies and the average byte distribution of the text data cluster. The mean / variance of the numerical features and the peak occurrence frequency of the time-series data cluster. The data segment length distribution range and the feature code repetition rate of the binary data cluster.

9. A computer data management method according to claim 8, characterized in that: In the three-level index structure, the global index adopts a hybrid structure of hash table and ordered linked list. The hash table key is the hash value of the cluster feature summary, and the value is the cluster identifier, the number of data blocks, the search popularity prediction value, and the pointer to the storage address of the intra-cluster index. The ordered linked list is arranged in descending order according to the search popularity prediction value. The intra-cluster index adopts inverted index, B+ tree index, and hash index according to the data type. If the base data block associated with any index item has been deleted and there is no redundant data block associated with the index item, the index item is directly deleted, the storage space is released, and the index structure is updated.

10. A computer data management system for implementing the method according to any one of claims 1 to 9, characterized in that: include: The data preprocessing module reads the heterogeneous dataset and divides it into text, time series and binary data groups according to data type. It extracts the core fields of each data group and generates deduplication features and index features. It performs standardization processing on the deduplication features and redundancy filtering on the index features. It sets a sliding window based on data type and generates a window index feature summary. The clustering module calculates the comprehensive similarity score between data blocks using a sliding window, filters the set of similar data blocks and determines the benchmark data block and redundant data blocks, clusters the benchmark data blocks to generate initial clusters, calculates the global deduplication rate and the effectiveness of cluster preprocessing, and reinitializes the sliding window if the global deduplication rate and the effectiveness of cluster preprocessing do not meet the preset conditions. The index optimization module optimizes the cluster structure and generates cluster feature summaries based on the initial cluster search popularity prediction values. It constructs a three-layer index structure including a global index, an intra-cluster index, and a feature index, tests search efficiency, and optimizes the index hierarchy until the performance indicators are met.

Images