Compression storage method and device for large-scale census data and medium

Abstract

The invention provides a compression storage method and device for large-scale census data and a medium, and relates to the technical field of data processing.The method comprises the steps that data structure analysis is conducted on a census data storage waiting set, and a data hierarchy node tree is constructed and obtained; performing node value feature analysis on the data level node tree to obtain a data node value topology; according to a data node value topology configuration compression strategy, adaptive compression processing is carried out on the census data storage waiting set, and a compressed data storage waiting set is obtained; performing hierarchical mapping construction of the graph database by taking the data hierarchical node tree as a hierarchical connection constraint to obtain a distributed graph database; and taking the data hierarchical node tree as a data storage guide, and performing storage node filling of the distributed graph database by adopting a compressed data storage waiting set. The technical problem that the storage resources cannot be fully utilized due to the fact that the data relation and the data value are difficult to analyze and evaluate in the existing general survey data storage process is solved, and the storage resource utilization rate and the storage efficiency are optimized.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device and medium for compressing and storing large-scale census data. Background Art

[0002] Large-scale census data is an important information resource in modern society. It covers information on multiple dimensions such as population, economy, and environment, and is of great significance for policy making, resource allocation, and academic research.

[0003] At present, large-scale census data storage mostly adopts traditional relational databases or simple file system storage. These systems usually use fixed data structures, such as tables or relational models, to store data. However, these storage methods have obvious shortcomings when facing large-scale, complex and related census data. In terms of compression, a unified compression standard is usually adopted, without fully considering the diversity and importance of data, which can easily cause the loss of high-value data information or insufficient compression of low-value data, affecting storage efficiency. The compression process may also introduce information loss, affecting the integrity and availability of data. In terms of data relationship management, relational databases and file systems lack in-depth analysis of the complex relationship structure within the census data, making it difficult to clearly present the relationship between data, resulting in low efficiency in data query and analysis. Summary of the invention

[0004] The present application provides a compression storage method, device and medium for large-scale census data, which solves the technical problem of insufficient utilization of storage resources due to the lack of effective analysis and evaluation of data relationships and data values ​​in the existing large-scale census data storage process, and achieves the technical effect of optimizing storage resource utilization and storage efficiency while ensuring data integrity and accuracy.

[0005] In view of the above problems, on the one hand, the present application provides a compression storage method for large-scale census data, the method comprising: constructing a data hierarchy node tree by performing data structure analysis on a candidate set of census data; performing node value feature analysis on the data hierarchy node tree to obtain a data node value topology; configuring a compression strategy according to the data node value topology, and adaptively compressing the candidate set of census data based on the configuration result to obtain a compressed data candidate set; constructing a hierarchical mapping of a graph database using the data hierarchy node tree as a hierarchical connection constraint to obtain a distributed graph database; using the data hierarchy node tree as a data storage guide, and using the compressed data candidate set to fill the storage nodes of the distributed graph database.

[0006] On the other hand, the present application also provides a compression storage device for large-scale census data, the device comprising: a data structure analysis module, the data structure analysis module is used to construct a data hierarchical node tree by performing data structure analysis on a candidate set of census data; a node value analysis module, the node value analysis module is used to perform node value feature analysis on the data hierarchical node tree to obtain a data node value topology; an adaptive compression module, the adaptive compression module is used to configure a compression strategy according to the data node value topology, and adaptively compress the candidate set of census data based on the configuration result to obtain a compressed data candidate set; a graph database construction module, the graph database construction module is used to perform hierarchical mapping construction of a graph database using the data hierarchical node tree as a hierarchical connection constraint to obtain a distributed graph database; a data storage module, the data storage module is used to use the data hierarchical node tree as a data storage guide, and use the compressed data candidate set to fill the storage nodes of the distributed graph database.

[0007] In a third aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps in any one of the above-mentioned methods when executed by a processor.

[0008] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0009] By analyzing the data structure of the candidate data set of the census data, a data hierarchical node tree is constructed to clarify the hierarchical relationship and logical structure between data, improve data organization, and lay the foundation for subsequent data value evaluation and storage management. The node value feature analysis of the data hierarchical node tree is performed to obtain the data node value topology, identify important data, so that it can be given priority when storing, and provide a basis for the subsequent configuration of storage resources. Compression strategy configuration is performed according to the data node value topology, and adaptive compression processing is performed on the candidate data set of the census data based on the configuration result to obtain a compressed data candidate set, effectively reducing the storage space occupied, while avoiding data loss caused by compression, and ensuring data integrity and availability. The hierarchical mapping of the graph database is constructed with the data hierarchical node tree as the hierarchical connection constraint to obtain a distributed graph database, so as to quickly access and process data with complex relationships, and improve query performance and flexibility. The data hierarchical node tree is used as a data storage guide, and the compressed data candidate set is used to fill the storage nodes of the distributed graph database to ensure data traceability and achieve effective management of census data.

[0010] In summary, this application provides an efficient solution for the storage and management of large-scale census data by constructing a hierarchical structure tree, combining data value evaluation for adaptive compression processing, and using graph database management for storage. This solution not only significantly improves storage efficiency and query performance, but also enhances data security and integrity, making the management of historical data more flexible, and ultimately achieving the maximum optimization of storage resource usage while ensuring data availability.

[0011] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 A flowchart of a method for compressing and storing large-scale census data provided in an embodiment of the present application.

[0013] Figure 2 A schematic diagram of a process for obtaining a data node value topology in a method for compressing and storing large-scale census data provided in an embodiment of the present application.

[0014] Figure 3 A schematic diagram of a process for obtaining a candidate set of compressed data in the method for compressing and storing large-scale census data provided in an embodiment of the present application.

[0015] Figure 4 A schematic diagram of the structure of a compression storage device for large-scale census data provided in an embodiment of the present application.

[0016] Explanation of the accompanying drawings: data structure analysis module 10, node value analysis module 20, adaptive compression module 30, graph database construction module 40, data storage module 50. DETAILED DESCRIPTION

[0017] The embodiments of the present application provide a compression storage method, device and medium for large-scale census data, thereby solving the technical problem of insufficient utilization of storage resources due to the lack of effective analysis and evaluation of data relationships and data values ​​in the existing large-scale census data storage process, and achieving the technical effect of optimizing storage resource utilization and storage efficiency while ensuring data integrity and accuracy.

[0018] Embodiment 1, as Figure 1 As shown, the embodiment of the present application provides a method for compressing and storing large-scale census data, the method comprising:

[0019] Step S1: Analyze the data structure of the candidate set of census data to construct a data hierarchical node tree.

[0020] Specifically, the census data candidate set refers to the collection of census data waiting to be stored and processed. For example, in a population census, the original data set of each person's age, gender, address and other information collected. The data hierarchy node tree is a tree-shaped data structure used to represent the hierarchical relationship between data. Each node of the tree represents a different part or element of the data, and the relationship between the parent node and the child node reflects the subordinate relationship of the data.

[0021] Obtain the candidate set of census data, import it into the data analysis tool for processing, and find the logical relationship within the data, including the classification relationship and dependency relationship of the data. You can classify the data fields (such as age, gender, income, etc.) to identify the hierarchical relationship of the data. Based on the analysis results, use the tree structure library to build a data hierarchical node tree according to the relationship between the data.

[0022] Step S1 analyzes the data structure of the candidate set of census data and uses data analysis tools and tree structure library to build a hierarchical node tree, which can clearly display the hierarchical relationship of the data and provide a basis for subsequent storage and management.

[0023] Step S2: Analyze the node value characteristics of the data hierarchical node tree to obtain the data node value topology.

[0024] Specifically, first, define the evaluation indicators of node value, which can include data call frequency, data size, data update frequency, etc. For the data in the node, calculate according to the set value evaluation criteria to obtain the corresponding score of each node. Based on these node value scores, use the image library to build the data node value topology and depict the value-based relationship structure between each node.

[0025] Step S2 obtains the data node value topology through node value feature analysis to determine the relative importance and connection relationship between data, so as to give priority to important data with high value in subsequent storage, thereby optimizing the configuration and utilization efficiency of storage resources.

[0026] Step S3: configuring a compression strategy according to the data node value topology, and adaptively compressing the census data candidate set based on the configuration result to obtain a compressed data candidate set.

[0027] Specifically, the compressed data candidate set refers to a set of census data that has been compressed and occupies less storage space than the census data candidate set.

[0028] Compression strategy configuration is performed according to the data node value topology obtained in step S2. This includes selecting a suitable compression algorithm, determining the compression ratio, setting the compression priority, etc. For data nodes with high value, lossless compression algorithm is preferred, and a lower compression ratio is set to ensure data accuracy; for data nodes with low value, lossy compression algorithm can be used, and a higher compression ratio can be set to save more storage space.

[0029] According to the compression strategy configuration results of different nodes, compress the data nodes in the data hierarchy node tree one by one. For example, when processing audio census data, if the sound of a certain period of time in the audio (corresponding to a data node) is background noise (low value), it is highly compressed according to the lossy compression strategy; if it is the main human voice part (high value), it is lightly compressed according to the lossless compression strategy. The compressed data is stored as a new compressed data candidate set.

[0030] Step S3 adaptively compresses the data according to its importance before storage, ensuring that important original information is retained, while effectively reducing storage space and ensuring data integrity and availability.

[0031] Step S4: construct a hierarchical mapping of the graph database using the data hierarchy node tree as a hierarchical connection constraint to obtain a distributed graph database.

[0032] Specifically, hierarchical connection constraints are restrictions when building a graph database, which stipulate the hierarchical relationship between nodes when building a graph database. Distributed graph database is a type of graph database whose data is stored on multiple nodes (which can be physical server nodes, etc.). It is suitable for processing large-scale, complex data relationships, and distributed storage can improve data processing efficiency, availability and scalability, and facilitate fast query and management.

[0033] The hierarchical mapping of the graph database is constructed using the data hierarchy node tree as the hierarchical connection constraint. That is, when constructing the graph database, the node connections in the graph database should be established according to the hierarchical relationships such as the parent-child relationship and the sibling relationship in the data hierarchy node tree, and the data should be mapped to the graph database according to the hierarchical structure. Starting from the root node of the data hierarchy node tree, it is mapped as an initial node (central node) in the graph database. Then, for each child node in the data hierarchy node tree, according to the hierarchical connection constraint, a corresponding node is created in the graph database, and an edge connection is established to represent the hierarchical relationship between the child node and the parent node. In this way, the nodes and relationships in the entire data hierarchy node tree are gradually mapped to the graph database, and the hierarchical mapping of the graph database is constructed to obtain a distributed graph database. This distributed graph database can be stored on multiple servers, and a distributed computing framework is used to manage data storage and access on different nodes, improving the data access speed and system scalability, so as to facilitate rapid access and processing of complex data relationships.

[0034] Step S5: Use the data hierarchy node tree as a data storage guide, and use the compressed data candidate set to fill the storage nodes of the distributed graph database.

[0035] Specifically, the data storage orientation is the rule or basis for guiding the direction and method of data storage. In an embodiment of the present application, the data hierarchy node tree is used as the data storage orientation, that is, when storing data, the storage location and organization of the data in the distributed graph database should be determined according to the structural relationship embodied by the data hierarchy node tree. For each part of the data in the compressed data candidate set, its storage node position in the distributed graph database is determined according to the storage orientation of the data hierarchy node tree. Use the write interface of the graph database to write the compressed data to the corresponding node. For example, the compressed data candidate set contains the sales data of the product. According to the hierarchical relationship between product classification and sales area in the data hierarchy node tree, the sales data is filled into the storage nodes related to the corresponding product category nodes and sales area nodes in the distributed graph database to complete the accurate storage of large-scale census data.

[0036] Further, such as Figure 2 As shown, step S2 of the embodiment of the present application also includes:

[0037] Step S21: Analyze the historical access behavior of multiple data category nodes constituting the data hierarchy node tree to obtain multiple access behavior characteristics, wherein the access behavior characteristics include data update change amount, data update cycle and historical access frequency.

[0038] Step S22: obtaining multiple hierarchical relationship features of the multiple data category nodes from the data hierarchical node tree, wherein the hierarchical relationship features include data relevance and data hierarchical position.

[0039] Step S23: using the multiple access behavior features and the multiple hierarchical relationship features to perform a joint analysis of the node value features of the multiple data category nodes to obtain multiple node value parameters.

[0040] Step S24: Based on the multiple data category nodes, project the multiple node value parameters to the data hierarchical node tree to obtain an initial node value topology.

[0041] Step S25: predefine an effective hierarchical radius, and accumulate the hierarchical value of the initial node value topology according to the effective hierarchical radius to obtain the data node value topology.

[0042] Specifically, first, the historical access behavior records of the data are obtained by analyzing the data access logs, and the access behavior characteristics such as data update change, data update cycle and historical access frequency are calculated from these historical access behavior records. Among them, the data update change indicates the degree of change of the data within a certain period of time. For example, for the returning farmland to forest data, the data update change is determined by calculating the ratio of the change between the current data and the previous data. The more changes, the higher the data value. For stock price data, the daily stock price fluctuation range is the data update change. The data update cycle refers to the time interval regularity of data updates, which can be determined by counting the mode of the time intervals between two adjacent updates. For example, the update cycle of monthly report data is one month. The historical access frequency refers to the number of times the data has been accessed in the past. The higher the historical access frequency, the higher the data value.

[0043] Use data mining algorithms to parse the structure of the data hierarchy node tree and obtain multiple hierarchical relationship features of multiple data category nodes. The hierarchical relationship features include data association and data hierarchical position. Data association reflects the closeness of the relationship between data category nodes. Algorithms based on graph theory can be used to analyze the connection relationship between data category nodes and the number of other nodes associated with each data category node can be determined by counting. The data hierarchical position represents the hierarchical position of the data category node in the data hierarchy node tree, which can be obtained by traversing the hierarchical structure of the data hierarchy node tree.

[0044] Comprehensively consider the access behavior characteristics and hierarchical relationship characteristics, and evaluate the value of each data category node. This process can be achieved by constructing a node value evaluation function to quantify the behavior characteristics and hierarchical relationship characteristics to obtain multiple node value parameters; or by combining the access behavior characteristics and hierarchical relationship characteristics through a weighted model, setting the weights of the access behavior characteristics and hierarchical relationship characteristics, and performing weighted summation on each node to obtain multiple node value parameters. Among them, the node value parameter is the quantitative result of the value of the data category node obtained after the joint analysis of the node value characteristics, which is used to measure its value in the data hierarchy node tree.

[0045] According to multiple data category nodes, the node value parameters are added as node attributes to the corresponding nodes in the data hierarchy node tree, so as to obtain the initial node value topology. The effective hierarchy radius is predefined to determine which levels of node values ​​in the data hierarchy node tree need to be accumulated. For example, if the effective hierarchy radius is 2, then the node values ​​of the target node and the 2 layers above and below it may only be accumulated. Then, the hierarchical value accumulation of the initial node value topology is performed according to the effective hierarchy radius. The accumulation process can use a recursive algorithm to traverse the nodes within the effective hierarchy radius in the data hierarchy node tree, accumulate these node value parameters from child nodes to parent nodes, realize the transfer and aggregation of the overall value, and finally obtain the data node value topology.

[0046] The above steps determine the node value parameters by comprehensively considering the access behavior characteristics and hierarchical relationship characteristics of each node, and implement hierarchical value accumulation according to the predefined effective hierarchical radius, and finally obtain the data node value topology, which fully evaluates the importance of the census data and provides a reference for the subsequent compression strategy configuration, so as to focus on preserving valuable data.

[0047] Furthermore, step S23 of the embodiment of the present application further includes:

[0048] A node value evaluation function is introduced, and the node value evaluation function is as follows:

[0049]

[0050] Among them, V is the node value parameter, A is the historical access frequency, U is the data update cycle, C is the data update change, L is the data level position, and N is the data association.

[0051] The node value evaluation function is input with the multiple access behavior features and the multiple hierarchical relationship features to perform a joint analysis of node value features, thereby obtaining the multiple node value parameters.

[0052] Specifically, the node value evaluation function combines multiple factors such as historical access frequency (A), data update cycle (U), data update change (C), data hierarchical position (L) and data correlation (N) to calculate the node value parameter (V). By inputting the characteristic value corresponding to each node obtained above into the node value evaluation function, the node value parameter corresponding to each node can be obtained, realizing a scientific assessment of the importance of census data and providing a specific basis for subsequent storage and management decisions.

[0053] Further, such as Figure 3 As shown, step S3 of the embodiment of the present application also includes:

[0054] Step S31: interactively obtain multiple sample compression configurations and multiple sample node value scales, and use the multiple sample compression configurations and multiple sample node value scales to tabulate and obtain a compression configuration table.

[0055] Step S32: using the data node value topology to traverse the compression configuration table to perform compression configuration mapping, and obtain an initial compression configuration topology.

[0056] Step S33: performing content perception on a plurality of census data candidate storage units in the census data candidate storage set, and classifying data based on the perception result to obtain a plurality of groups of census data contents.

[0057] Step S34: performing importance analysis on the contents of the multiple groups of census data, and performing local compression optimization based on the analysis results to obtain multiple groups of local compression configurations.

[0058] Step S35: Adopting the multiple groups of local compression configurations to adaptively correct the initial compression configuration topology to obtain a target compression configuration topology.

[0059] Step S36: Using the target compression configuration topology to perform compression processing on the multiple groups of census data content to obtain the compressed data candidate set.

[0060] Specifically, the sample compression configuration is a set of configuration parameters for data compression, including compression algorithm type, compression ratio, etc. The sample node value scale is a standard for measuring the importance of a node, which is usually related to the node value parameter and can be a grade standard corresponding to different node value parameter ranges. The compression configuration table is a table that lists all sample compression configurations and their corresponding node value scales, and is used to match the corresponding compression strategy for each data node. Collect multiple compression configurations and node value scales from historical data or through user input, and use spreadsheet software (such as Microsoft Excel) or data structures in programming languages ​​(such as list nested dictionary structures in Python) to tabulate these data to obtain a compression configuration table.

[0061] Using a loop traversal algorithm, the compression configuration table is traversed using the data node value topology. For each node, a matching item is searched in the compression configuration table according to its value parameter, and the corresponding compression configuration is determined to form an initial compression configuration topology. This initial compression configuration topology reflects the preliminary correspondence between the data node value and the compression configuration. Exemplarily, for high-value nodes, a lossless compression strategy (such as LZW, PNG format, etc.) is selected, and the compression ratio is set at a lower level, such as 1.2:1, to ensure the integrity and accuracy of the data. For medium-value nodes, a low-loss compression strategy is adopted to ensure that the amount of data is reduced with limited loss. For low-value nodes, a high-loss compression strategy is adopted to minimize the storage volume, such as using a JPEG compression algorithm based on discrete cosine transform (DCT) (if it is image data), and the compression ratio can be set to a higher 5:1 or higher.

[0062] Use data mining and machine learning techniques to perceive the content of multiple census data candidate units in the census data candidate set. For text data, the topic model in natural language processing (NLP) can be used to identify the topic content and thus classify the data; for image data, convolutional neural networks (CNN) can be used to perform operations such as object recognition to perceive the content and classify it. Based on the perception results, the census data candidate units are divided into multiple groups of census data content.

[0063] The importance analysis of multiple groups of census data content can be carried out using rule-based methods or machine learning algorithms. For example, for text data, if it contains key decision-making information, it is considered to be of high importance, and if it is a general description, it is considered to be of low importance. For image data, if it is the main part, it is of high importance, and if it is the background part, it is of low importance. According to the results of the importance analysis, a local compression configuration is formulated for each group of census data content.

[0064] Combine multiple groups of local compression configurations to adjust the relevant compression configurations in the initial compression configuration topology, optimize the overall compression strategy, and update the compression configuration topology. For example, if the compression configuration corresponding to a node in the initial compression configuration topology is a general compression method, after combining the local compression configuration, it is modified to a compression method that is more suitable for the content of the census data group, thereby obtaining the target compression configuration topology. According to the compression configuration information in the target compression configuration topology, the corresponding census data content is compressed, and finally the compressed data candidate set is obtained.

[0065] Furthermore, before step S36 in the embodiment of the present application, the following steps are also included:

[0066] Step 361: perform duplicate identification on the multiple groups of census data contents to obtain K groups of duplicate data contents corresponding to the K data intersections.

[0067] Step 362: Extract a first group of repeated data contents of a first data intersection from the K groups of repeated data contents, wherein the first group of repeated data contents includes N repeated data contents.

[0068] Step 363: After marking the N duplicate data contents in the multiple groups of census data contents, the N duplicate data contents are divided into first original data and N-1 duplicate data contents according to the data node value topology.

[0069] Step 364: In the multiple groups of census data contents, replace the N-1 repeated data contents with N-1 storage pointers.

[0070] By analogy, the plurality of groups of census data contents are pre-compressed based on the K groups of repeated data contents.

[0071] Specifically, before using the target compression configuration topology to perform compression processing on multiple sets of census data contents, it is also necessary to perform pre-compression processing on the multiple sets of census data contents to reduce data redundancy, thereby optimizing storage efficiency and preparing for the final compression processing.

[0072] Data intersection refers to the set of repeated parts in multiple census data sets. For example, if two census data sets contain some of the same user information, the same user information constitutes a data intersection.

[0073] First, analyze the content of multiple groups of census data, and find out the same or similar parts by comparing the features and content of the data. This process can use hash algorithms (such as MD5, SHA, etc.) to improve recognition efficiency. For text data, first convert the text content into a hash value, and then compare whether the hash values ​​are the same to determine whether it is duplicate data. For image data, you can first extract the feature vector of the image (such as using a pre-trained model in deep learning to extract features), and then identify duplicate images by comparing the similarity of the feature vectors. In this way, K groups of duplicate data content corresponding to K data intersections are obtained, where K is a positive integer representing the number of data intersections.

[0074] A group of repeated data contents is randomly extracted from the K groups of repeated data contents, recorded as the first group of repeated data contents, corresponding to the first data intersection. The first group of repeated data contents includes N repeated data contents, where N is a positive integer. N repeated data contents are marked in multiple groups of census data contents, and the node value information of the data nodes where the N repeated data contents are located is queried according to the data node value topology, and the data content with the highest data node value is marked as the first original data, and the rest are marked as N-1 repeated data contents.

[0075] Then, the N-1 duplicate data contents are replaced with N-1 storage pointers. A storage pointer is a mark used to point to the original data storage location. When processing duplicate data, replacing the duplicate data contents with storage pointers can reduce the storage space of the data, and at the same time, when the data needs to be accessed, the original data can be found through the pointers. The content replacement operation can be implemented in programming by modifying the data structure. For example, when using Python language to process data, if the data is stored in the form of a list or a dictionary, the element representing the duplicate data content can be replaced with a pointer pointing to the first original data storage location. By analogy, the K groups of duplicate data contents are processed in the same way to complete the pre-compression processing of multiple groups of census data contents, so as to prepare for the compression processing in step S36.

[0076] Furthermore, step S5 of the embodiment of the present application further includes:

[0077] Step S51: filling the compressed data candidate set into the distributed graph database according to the mapping relationship between the compressed data candidate set and the data hierarchy node tree.

[0078] Step S52: Optimize the query function of the distributed graph database by configuring storage pointers for the distributed graph database.

[0079] Specifically, first, the mapping relationship between the compressed data candidate set and the data hierarchical node tree is clarified, that is, the corresponding relationship between the compressed data candidate set and the data hierarchical node tree. From the above steps, it can be seen that the data hierarchical node tree is constructed according to the categories and hierarchical relationships of the data, and the various parts of the compressed data candidate set have a pre-defined corresponding relationship with these categories and hierarchies. According to this corresponding relationship, the compressed data candidate set is filled into the corresponding nodes and relationships to ensure that all data is successfully stored according to the mapping relationship, and a query is performed to confirm the data integrity.

[0080] Configure pointers for nodes or edges in the graph database to point to specific data content or other related nodes, and dynamically adjust the pointing of storage pointers by creating indexes or writing stored procedures to optimize the query function of the distributed graph database. When querying data, the query engine can quickly access and locate the required data through the link relationship of the pointer, avoiding redundant loading of duplicate data, thereby improving query efficiency. After optimization, run query tests, measure the performance improvement effect, and make further adjustments, thereby effectively improving the efficiency of data management and ensuring efficient access to data in a distributed environment.

[0081] Furthermore, after step S5, the embodiment of the present application further includes:

[0082] Step S53: configuring multiple update monitoring modules for multiple storage nodes in the distributed graph database.

[0083] Step S54: when the first update monitoring module monitors that data update occurs on the first storage node, the first initial compression configuration of the first storage node is called from the initial compression configuration topology.

[0084] Step S55: Perform compression weight allocation according to the M historical update behaviors of the first storage node to obtain M compression weight coefficients.

[0085] Step S56: Calculate and obtain M updated compression configurations based on the first initial compression configuration and M compression weight coefficients.

[0086] Step S57: Using the M update compression configurations to perform M-1 data compression on the data of the M historical update behaviors.

[0087] By analogy, the multiple update monitoring modules are applied to perform dynamic compression of the storage of the distributed graph database.

[0088] Specifically, the update monitoring module is a functional module used to monitor the data update of storage nodes in the distributed graph database, and can check in real time or regularly whether the data of the storage node has changed. The update monitoring module can use the database plug-in mechanism or a custom-developed monitoring program. An update monitoring module is configured for each storage node in the distributed graph database to accurately capture data update events and respond to them.

[0089] The compression weight coefficient is a coefficient assigned based on the historical update behavior of the storage node, which is used to measure the importance of different historical update behaviors in the compression configuration. For example, if a historical update behavior involves a large amount of data, it may be assigned a higher compression weight coefficient to give it more consideration in the update compression configuration. The update compression configuration is a compression configuration for data update calculated based on the initial compression configuration and the compression weight coefficient. The update compression configuration comprehensively considers the impact of the initial compression configuration and historical update behavior to more effectively compress the updated data.

[0090] One of the multiple update monitoring modules is selected as a first update monitoring module, corresponding to a first storage node. The first update monitoring module is taken as an example to illustrate the subsequent storage dynamic compression process.

[0091] When the first update monitoring module detects that the first storage node has a data update, the initial compression configuration of the first storage node is called from the initial compression configuration topology, which is recorded as the first initial compression configuration. The M historical update behavior records of the first storage node are collected, and the parameters such as the data change amount, update frequency, and data type change of each update are counted. A compression weight coefficient is assigned to each historical update behavior, and M compression weight coefficients are generated. Exemplarily, the percentage of the amount of data updated each time to the total amount of data is calculated, and then the compression weight coefficient is assigned according to the percentage. The higher the percentage, the greater the weight coefficient.

[0092] According to the compression weight coefficient, the initial compression configuration such as compression ratio, compression algorithm type, etc. is adjusted, and the weighted formula is used to generate the updated configuration to obtain M updated compression configurations. M updated compression configurations are used to perform M-1 data compressions, and each compression is performed according to the corresponding updated compression configuration, and the compressed data is stored back in the graph database, thereby realizing dynamic storage compression of the distributed graph database and improving data storage efficiency and flexibility.

[0093] In summary, the compression storage method for large-scale census data provided in the embodiments of the present application has the following technical effects:

[0094] By analyzing the data structure of the candidate data set of the census data, a data hierarchical node tree is constructed to clarify the hierarchical relationship and logical structure between data, improve data organization, and lay the foundation for subsequent data value evaluation and storage management. By introducing the node value evaluation function, the historical access frequency, data update cycle, data update change, data hierarchical position and data correlation are comprehensively considered, and multiple node value parameters are jointly analyzed to obtain the data node value topology, identify important data, so that they can be given priority when storing, and provide a basis for the subsequent configuration of storage resources. According to the data node value topology, the compression strategy is configured, the sample compression configuration and the sample node value scale are interactively obtained and tabulated to obtain the compression configuration table, and the initial compression configuration topology is obtained through compression configuration mapping. Then, the candidate data set of the census data is content-aware, importance analysis and local compression optimization are performed, and the initial compression configuration topology is corrected by using the local compression configuration to obtain the target compression configuration topology, and the candidate data set of the census data is adaptively compressed based on the target compression configuration topology to obtain the compressed data candidate set, effectively reducing the storage space occupied, while avoiding data loss caused by compression, and ensuring the integrity and availability of the data. In addition, before compression, pre-compression processing is completed by identifying and extracting duplicate data content and replacing the duplicate data with storage pointers to reduce data redundancy. The hierarchical mapping of the graph database is constructed using the data hierarchy node tree as a hierarchical connection constraint to obtain a distributed graph database, so as to quickly access and process data with complex relationships and improve query performance and flexibility. The data hierarchy node tree is used as a data storage guide, and the compressed data candidate set is used to fill the storage nodes of the distributed graph database. The storage node configuration of the distributed graph database is updated with a monitoring module. When data updates are monitored, the initial compression configuration is called, and the compression weight coefficient is assigned according to the historical update behavior to calculate the updated compression configuration to perform data compression, thereby ensuring data security and traceability, thereby improving the flexibility of data management.

[0095] Overall, the embodiment of the present application provides an efficient solution for the storage and management of large-scale census data by constructing a hierarchical structure tree, combining data value evaluation for adaptive compression processing, and using graph database management for storage. This method not only significantly improves storage efficiency and query performance, but also enhances data security and integrity, making the management of historical data more flexible, and ultimately achieving the maximum optimization of storage resource usage while ensuring data availability.

[0096] Embodiment 2, as Figure 4 As shown, the embodiment of the present application provides a compression storage device for large-scale census data, the device comprising:

[0097] The data structure analysis module 10 is used to construct a data hierarchical node tree by performing data structure analysis on the candidate collection of census data.

[0098] The node value analysis module 20 is used to perform node value feature analysis on the data hierarchical node tree to obtain a data node value topology.

[0099] The adaptive compression module 30 is used to configure the compression strategy according to the data node value topology, and to perform adaptive compression processing on the census data candidate set based on the configuration result to obtain the compressed data candidate set.

[0100] The graph database construction module 40 is used to construct a hierarchical mapping of the graph database using the data hierarchy node tree as a hierarchical connection constraint to obtain a distributed graph database.

[0101] The data storage module 50 is used to use the data hierarchy node tree as a data storage guide and use the compressed data candidate set to fill the storage nodes of the distributed graph database.

[0102] Furthermore, the node value analysis module 20 of the embodiment of the present application is also used to perform the following steps:

[0103] Performing historical access behavior analysis on multiple data category nodes constituting the data hierarchy node tree to obtain multiple access behavior features, wherein the access behavior features include data update variation, data update cycle and historical access frequency; obtaining multiple hierarchical relationship features of the multiple data category nodes by calling from the data hierarchy node tree, wherein the hierarchical relationship features include data correlation and data hierarchy position; performing a joint analysis of node value features of the multiple data category nodes using the multiple access behavior features and the multiple hierarchical relationship features to obtain multiple node value parameters; projecting the multiple node value parameters onto the data hierarchy node tree based on the multiple data category nodes to obtain an initial node value topology; predefining an effective hierarchy radius, and accumulating the hierarchy value of the initial node value topology based on the effective hierarchy radius to obtain the data node value topology.

[0104] Furthermore, the node value analysis module 20 of the embodiment of the present application is also used to perform the following steps:

[0105] A node value evaluation function is introduced, and the node value evaluation function is as follows:

[0106]

[0107] Among them, V is the node value parameter, A is the historical access frequency, U is the data update cycle, C is the data update change, L is the data hierarchical position, and N is the data association. The node value evaluation function is input with the multiple access behavior features and the multiple hierarchical relationship features to perform a joint analysis of the node value features to obtain the multiple node value parameters.

[0108] Furthermore, the adaptive compression module 30 of the embodiment of the present application uses the target compression configuration topology to perform compression processing on the plurality of groups of census data contents to obtain the compressed data candidate set, and before that, is also used to perform the following steps:

[0109] Interactively obtain multiple sample compression configurations and multiple sample node value scales, and use the multiple sample compression configurations and multiple sample node value scales to tabulate and obtain a compression configuration table; use the data node value topology to traverse the compression configuration table to perform compression configuration mapping and obtain an initial compression configuration topology; perform content perception on multiple census data candidate storage units in the census data candidate storage set, and perform data classification based on the perception results to obtain multiple groups of census data content; perform importance analysis on the multiple groups of census data content, and perform local compression optimization based on the analysis results to obtain multiple groups of local compression configurations; use the multiple groups of local compression configurations to adaptively correct the initial compression configuration topology to obtain a target compression configuration topology; use the target compression configuration topology to perform compression processing on the multiple groups of census data content to obtain the compressed data candidate storage set.

[0110] Furthermore, the adaptive compression module 30 in the embodiment of the present application is also used to perform the following steps:

[0111] Duplicate identification is performed on the multiple groups of census data contents to obtain K groups of duplicate data contents corresponding to the K data intersections; a first group of duplicate data contents of the first data intersection is extracted from the K groups of duplicate data contents, wherein the first group of duplicate data contents includes N duplicate data contents; after marking the N duplicate data contents in the multiple groups of census data contents, the N duplicate data contents are divided into first original data and N-1 duplicate data contents according to the data node value topology; in the multiple groups of census data contents, the N-1 duplicate data contents are replaced with N-1 storage pointers; and so on, pre-compression processing of the multiple groups of census data contents is performed according to the K groups of duplicate data contents.

[0112] Furthermore, the data storage module 50 in the embodiment of the present application is also used to perform the following steps:

[0113] According to the mapping relationship between the compressed data candidate set and the data hierarchical node tree, the compressed data candidate set is filled into the distributed graph database; by configuring the storage pointer of the distributed graph database, the query function of the distributed graph database is optimized.

[0114] Furthermore, the data storage module 50 of the embodiment of the present application uses the data hierarchy node tree as a data storage guide, uses the compressed data candidate set to fill the storage nodes of the distributed graph database, and then is used to perform the following steps:

[0115] Configure multiple update monitoring modules for multiple storage nodes in the distributed graph database; when the first update monitoring module monitors that data update occurs on the first storage node, call the first initial compression configuration of the first storage node from the initial compression configuration topology; perform compression weight allocation according to the M historical update behaviors of the first storage node to obtain M compression weight coefficients; obtain M update compression configurations based on the first initial compression configuration and the M compression weight coefficients; use the M update compression configurations to perform M-1 data compression of the data of the M historical update behaviors; and so on, apply the multiple update monitoring modules to perform dynamic storage compression of the distributed graph database.

[0116] Through the above-mentioned detailed description of the compression and storage method of large-scale census data in this specification, those skilled in the art can clearly understand the compression and storage device for large-scale census data in this embodiment. For the device disclosed in Example 2, since it corresponds to the method disclosed in Example 1 and has corresponding functional modules and beneficial effects, the relevant parts can be referred to the description of the method part.

[0117] In addition, the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the various steps of the above-mentioned large-scale census data compression storage method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0118] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for compressing and storing large-scale census data, characterized in that: The method comprises: By analyzing the data structure of the census data candidate set, a data hierarchical node tree is constructed; Performing node value feature analysis on the data hierarchical node tree to obtain data node value topology; Performing compression strategy configuration according to the data node value topology, and performing adaptive compression processing on the census data candidate set based on the configuration result to obtain a compressed data candidate set; Using the data hierarchical node tree as a hierarchical connection constraint, a hierarchical mapping construction of a graph database is performed to obtain a distributed graph database; The data hierarchy node tree is used as a data storage guide, and the compressed data candidate set is used to fill the storage nodes of the distributed graph database.

2. The method for compressing and storing large-scale census data according to claim 1, characterized in that: Performing node value feature analysis on the data hierarchical node tree to obtain data node value topology, the method comprising: Performing historical access behavior analysis on a plurality of data category nodes constituting the data hierarchical node tree to obtain a plurality of access behavior features, wherein the access behavior features include data update variation, data update cycle, and historical access frequency; Calling and obtaining a plurality of hierarchical relationship features of the plurality of data category nodes from the data hierarchical node tree, wherein the hierarchical relationship features include data association degree and data hierarchical position; Using the multiple access behavior features and the multiple hierarchical relationship features to jointly analyze the node value features of the multiple data category nodes, to obtain multiple node value parameters; According to the plurality of data category nodes, projecting the plurality of node value parameters to a data hierarchical node tree to obtain an initial node value topology; An effective hierarchical radius is predefined, and hierarchical value accumulation of the initial node value topology is performed according to the effective hierarchical radius to obtain the data node value topology.

3. The method for compressing and storing large-scale census data according to claim 2, characterized in that: The method comprises: using the plurality of access behavior features and the plurality of hierarchical relationship features to jointly analyze the node value features of the plurality of data category nodes to obtain a plurality of node value parameters; A node value evaluation function is introduced, and the node value evaluation function is as follows: Among them, V is the node value parameter, A is the historical access frequency, U is the data update cycle, C is the data update change, L is the data level position, and N is the data association degree; The node value evaluation function is input with the multiple access behavior features and the multiple hierarchical relationship features to perform a joint analysis of node value features, thereby obtaining the multiple node value parameters.

4. The method for compressing and storing large-scale census data according to claim 2, characterized in that: Compression strategy configuration is performed according to the data node value topology, and adaptive compression processing is performed on the census data candidate set based on the configuration result to obtain a compressed data candidate set. The method includes: Interactively obtain a plurality of sample compression configurations and a plurality of sample node value scales, and use the plurality of sample compression configurations and the plurality of sample node value scales to tabulate and obtain a compression configuration table; Using the data node value topology to traverse the compression configuration table to perform compression configuration mapping, and obtain an initial compression configuration topology; Performing content perception on a plurality of census data candidate storage units in the census data candidate storage set, and classifying data based on the perception result to obtain a plurality of groups of census data contents; Performing importance analysis on the contents of the multiple groups of census data, and performing local compression optimization based on the analysis results to obtain multiple groups of local compression configurations; Adopting the multiple groups of local compression configurations to adaptively correct the initial compression configuration topology to obtain a target compression configuration topology; The target compression configuration topology is used to perform compression processing on the multiple groups of census data contents to obtain the compressed data candidate set.

5. The method for compressing and storing large-scale census data according to claim 4, characterized in that: Before compressing the plurality of groups of census data contents using the target compression configuration topology to obtain the compressed data candidate set, the method includes: Performing duplicate identification on the multiple groups of census data contents to obtain K groups of duplicate data contents corresponding to the K data intersections; Extracting a first group of repeated data contents of a first data intersection from the K groups of repeated data contents, wherein the first group of repeated data contents includes N repeated data contents; After marking the N duplicate data contents on the multiple groups of survey data contents, dividing the N duplicate data contents into first original data and N-1 duplicate data contents according to the data node value topology; In the plurality of groups of census data contents, the N-1 repeated data contents are replaced with N-1 storage pointers; By analogy, the plurality of groups of census data contents are pre-compressed based on the K groups of repeated data contents.

6. The method for compressing and storing large-scale census data according to claim 5, characterized in that: The data hierarchy node tree is used as a data storage guide, and the compressed data candidate set is used to fill the storage nodes of the distributed graph database. The method includes: According to the mapping relationship between the compressed data candidate set and the data hierarchical node tree, the compressed data candidate set is filled into the distributed graph database; By configuring storage pointers for the distributed graph database, the query function of the distributed graph database is optimized.

7. The method for compressing and storing large-scale census data according to claim 4, characterized in that: The data hierarchy node tree is used as a data storage guide, and the compressed data candidate set is used to fill the storage nodes of the distributed graph database. After that, the method includes: Configure multiple update monitoring modules for multiple storage nodes in the distributed graph database; When the first update monitoring module monitors that data update occurs on the first storage node, calling the first initial compression configuration of the first storage node from the initial compression configuration topology; Perform compression weight allocation according to the M historical update behaviors of the first storage node to obtain M compression weight coefficients; Obtain M updated compression configurations based on the first initial compression configuration and the M compression weight coefficients; Using the M update compression configurations to perform M-1 data compression on the data of the M historical update behaviors; By analogy, the multiple update monitoring modules are applied to perform dynamic compression of the storage of the distributed graph database.

8. A compression storage device for large-scale census data, characterized in that: The device is used to execute the large-scale census data compression storage method according to any one of claims 1 to 7, comprising: A data structure analysis module, wherein the data structure analysis module is used to construct a data hierarchical node tree by performing data structure analysis on the candidate set of census data; A node value analysis module, the node value analysis module is used to analyze the node value characteristics of the data hierarchical node tree to obtain a data node value topology; An adaptive compression module, the adaptive compression module is used to configure a compression strategy according to the data node value topology, and adaptively compress the census data candidate set based on the configuration result to obtain a compressed data candidate set; A graph database construction module, wherein the graph database construction module is used to perform hierarchical mapping construction of a graph database using the data hierarchical node tree as a hierarchical connection constraint to obtain a distributed graph database; A data storage module, wherein the data storage module is used to use the data hierarchy node tree as a data storage guide and adopt the compressed data candidate set to fill the storage nodes of the distributed graph database.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for compressing and storing large-scale census data as described in any one of claims 1 to 7 are implemented.

Images