A method and system for direct calculation of compressed data based on non-volatile storage
Through the N-TADOC persistence strategy and pruning method of nonvolatile memory pool management, DAG data structures are constructed and graph traversal is optimized, which solves the data locality and redundant access problems of direct processing technology for compressed data on nonvolatile memory. , realizes efficient data analysis.
Patent Information
- Application Number
- CN202410399572.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-03
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-04-03
AI Technical Summary
Building compressed data direct processing technology on nonvolatile memory has problems such as poor data localization and redundant access, resulting in inefficiency and the existing technology cannot effectively utilize the hardware architecture.
Using N-TADOC persistence strategy and pruning method of nonvolatile memory pool management, DAG data structure is constructed, the data structure length is estimated through bottom-up summing method, redundant read and write operations are reduced, and metadata is designed to optimize the graph Traversal, use preset traversal strategies for text analysis.
Improve data locality, reduce redundant access to nonvolatile memory, reduce persistence costs and access latency, and significantly improve data analysis efficiency.
Smart Images

Figure CN118363528B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a method and system for directly calculating compressed data based on non-volatile storage, and belongs to the technical field of big data processing. Background Art
[0002] The advent of the big data era has magnified the demand for large-scale data analysis. Compressed data direct processing technology can directly process compressed data without decompression, which can significantly save storage space. In addition, compressed data direct processing technology can also optimize processing time by reusing input and intermediate data. However, traditional compressed data direct processing technology uses a large amount of dynamic random access memory (DRAM), resulting in high costs.
[0003] Non-volatile memory (NVM) is a next-generation storage technology with better density than DRAM and good performance in various practical applications, providing a potential solution. By putting data on NVM, it is possible to reduce the use of DRAM and process larger data sets. In addition, the lower device and power costs of NVM compared to DRAM can reduce the total cost of ownership for data analysts. In addition, the efficient implementation of compressed data direct processing technology on NVM is perfectly aligned with the unique characteristics of NVM. Compared with DRAM, NVM has a lower bandwidth, which is consistent with the ability of compressed data direct processing technology to process compressed data, thereby reducing bandwidth requirements. Compressed data direct processing technology can further enhance this consistency by exploiting text redundancy to reduce the update frequency during analysis, thereby minimizing NVM write operations and enhancing its durability.
[0004] Despite the many benefits of building DDP on NV memory devices, establishing a DDP framework on NV memory is a challenging task. DDP’s requirement for random access to compressed data sets can result in poor data locality when applied directly to NV memory. Additionally, when the capacity of traditional data structures is insufficient, brute force reconstruction is required. This results in a large number of redundant accesses to NV memory, further reducing the efficiency of DDP on NV memory. These challenges require the invention of new techniques to minimize the impact of poor data locality and redundant NV memory accesses on DDP processing on NV memory.
[0005] Currently, many scholars are conducting research on compressed data direct processing technology. However, due to differences in hardware architecture, they cannot be effectively utilized by non-volatile memories. Summary of the invention
[0006] In view of the above problems, an object of the present invention is to provide a method and system for direct calculation of compressed data based on non-volatile storage.
[0007] To achieve the above object, the present invention adopts the following technical solutions:
[0008] In a first aspect, the present invention provides a method for direct calculation of compressed data based on non-volatile storage, comprising the following steps:
[0009] 1) Obtaining compressed text data and an analysis application based on the compressed text data;
[0010] 2) Based on the N-TADOC persistence strategy, the compressed text data is initialized to construct a non-volatile storage DAG data structure, and the preset traversal strategy is used to traverse the non-volatile storage DAG data structure. The traversal results are statistically summarized according to the analysis application to obtain the text analysis results.
[0011] Further, in the step 2), based on the N-TADOC persistence strategy, the compressed text data is initialized to construct a non-volatile storage DAG data structure, and the DAG data structure of the non-volatile storage is traversed using a preset traversal strategy to obtain a text analysis result, including:
[0012] 2.1) Use N-TADOC to map the non-volatile storage space to the memory, initialize the compressed text data, and build the DAG data structure of non-volatile storage;
[0013] 2.2) Determine whether a fault occurs during step 2.1). If a fault occurs, N-TADOC returns to the previous checkpoint and reinitializes. Otherwise, proceed to step 2.3);
[0014] 2.3) Refresh the constructed DAG data structure back to non-volatile storage and proceed to step 2.4) to traverse the graph;
[0015] 2.4) Using the preset traversal strategy to traverse the DAG data structure in non-volatile storage, obtain the text analysis results, and use N-TADOC to persist the text analysis results to non-volatile storage;
[0016] 2.5) Determine whether a failure occurred during step 2.4). If a failure occurred, return to the previous checkpoint and traverse the graph again.
[0017] Further, in step 2.1), the compressed text data is initialized to construct a DAG data structure for non-volatile storage, including:
[0018] 2.1.1) A pruning method based on non-volatile memory pool management is used to eliminate some redundancy in the compressed text data, and a DAG pool is established on the non-volatile storage to obtain a DAG graph and its metadata;
[0019] 2.1.2) Based on the metadata, an upper bound on the length of the variable data structure on the non-volatile memory is estimated using a bottom-up summation method;
[0020] 2.1.3) Allocate space for the data object on the non-volatile storage based on the estimation result to obtain the DAG data structure of the non-volatile storage.
[0021] Furthermore, the pruning method based on non-volatile memory pool management eliminates part of the redundancy in the compressed text data, and establishes a DAG pool on the non-volatile storage to obtain a DAG graph and its metadata, including:
[0022] ① Traverse the rules to obtain the sub-rules and words contained in the rule and the sub-rules and word frequencies;
[0023] ②Write all sub-rules and their frequencies into the DAG pool;
[0024] ③Write all words and their frequencies into the DAG pool;
[0025] ④ Repeat steps ① to ③ to arrange the pruning representations of different rules in the compressed text data adjacently in the DAG pool;
[0026] ⑤Design metadata of rules in the DAG pool;
[0027] ⑥ Initialize the traversal queue and frequency counter in the DAG pool, where the traversal queue is used to record the progress of the task during the top-down traversal process, and the frequency counter is used to record the word or sequence frequency according to the requirements of the task.
[0028] Further, in step ⑤, the metadata includes:
[0029] The position of the sub-rule and word, which is used to indicate the offset where the rule is stored in the DAG pool;
[0030] The out / in degree is used to represent the number of out / in edges of the rule in the DAG representation of the compressed text data;
[0031] The word list size, which indicates the number of different word types included in the rule;
[0032] and the weight of the rule, which is used to represent the frequency or number of occurrences of the rule in the compressed text data.
[0033] Further, in step 2.1.2), based on the metadata, an upper limit of the length of the variable data structure on the non-volatile memory is estimated using a bottom-up summation method, including:
[0034] ① Randomly select a rule and determine whether the current rule contains sub-rules. If it does not contain any sub-rules, set the upper limit of the current rule to be equal to the length of its word list; if it contains sub-rules, proceed to step ②;
[0035] ② Randomly select a sub-rule to traverse from bottom to top, and when the upper limit of the current sub-rule is determined, set the sub-rule to "determined";
[0036] ③ Determine whether the current rule contains any sub-rules that have not yet been determined. If so, repeat step ② until all sub-rules in the current rule are "determined";
[0037] ④ Based on the upper limits of all determined sub-rules, get the upper limit of the current rule and set the current rule to "determined";
[0038] ⑤ Repeat steps ① to ④ to get the upper limit of all rules.
[0039] Furthermore, in step ④, the upper limit of the current rule is set to the sum of the upper limits of all sub-rules within it plus the length of the original word list of the current rule.
[0040] In a second aspect, the present invention provides a compressed data direct computing system based on non-volatile storage, comprising:
[0041] A data acquisition module, used to acquire compressed text data and an analysis application based on the compressed text data;
[0042] The data computing module is used to initialize the compressed text data to build a non-volatile storage DAG data structure based on the N-TADOC persistence strategy, and to traverse the non-volatile storage DAG data structure using a preset traversal strategy. The traversal results are statistically summarized according to the analysis application to obtain the text analysis results.
[0043] In a third aspect, the present invention provides a computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions, which, when executed by a computing device, cause the computing device to perform any method.
[0044] In a fourth aspect, the present invention provides a computing device, comprising: one or more processors and a memory, wherein the memory stores one or more programs and is configured to be executed by the one or more processors, and the one or more programs include instructions for executing any method.
[0045] The present invention adopts the above technical solution, which has the following advantages:
[0046] 1. The present invention designs an N-TADOC persistence strategy to isolate the data generated in different stages from the persistent data in the previous stage, making full use of the persistence characteristics of non-volatile memory and greatly reducing the persistence cost.
[0047] 2. The pruning method based on non-volatile memory pool management of the present invention eliminates redundancy and reconstructs compressed text data. The obtained DAG data structure can maintain good data locality during the DAG traversal process, thereby reducing redundant reading and writing of NVM due to frequent cache misses and improving the efficiency of data analysis.
[0048] 3. The present invention designs a bottom-up summation method to estimate the length of the variable-length data structure on the non-volatile storage and allocate space for the word list according to the upper limit of the word list, which helps to eliminate the overhead of redundant read and write operations during the reconstruction of the NVM data structure. This process improves the efficiency of the data structure reconstruction process and reduces the access delay of the NVM because it avoids unnecessary read and write operations that may have a negative impact on the overall performance.
[0049] Therefore, the present invention can be widely applied to the field of big data processing technology. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Throughout the accompanying drawings, the same reference numerals are used to represent the same components. In the accompanying drawings:
[0051] Figure 1 is a flow chart of a method for direct calculation of compressed data based on non-volatile storage provided by an embodiment of the present invention;
[0052] Figure 2 is a recovery flow chart of a failure occurring on a non-volatile storage provided by an embodiment of the present invention;
[0053] Figure 3 is a flow chart of allocating data structure space on non-volatile storage provided by an embodiment of the present invention;
[0054] Figure 4 is a flow chart of a method for calculating estimated space on non-volatile storage provided by an embodiment of the present invention;
[0055] Figure 5 This is an example of a method for calculating estimated space on non-volatile storage provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0056] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.
[0057] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.
[0058] In today's era of big data, processing large-scale text data poses great challenges in terms of spatial storage and processing time. In some embodiments of the present invention, a method for direct calculation of compressed data based on non-volatile storage is proposed, which mainly includes: first, organizing data on a non-volatile storage memory pool, that is, using a pruning method for the incoming compressed text data to eliminate part of the redundancy in the compressed text data, and reconstructing the compressed text data in the non-volatile memory; second, after the memory pool is built, a bottom-up method is used to estimate the upper limit of the length of the variable data structure on the non-volatile memory, and the space is extracted and allocated according to the upper limit value; finally, the compressed text data is traversed for text analysis. The present invention can improve data locality, reduce write operations on non-volatile storage, and thus improve efficiency.
[0059] Correspondingly, in some other embodiments of the present invention, a compressed data direct computing system, device and storage medium based on non-volatile storage are provided.
[0060] Example 1
[0061] like Figure 1 As shown, this embodiment provides a method for directly calculating compressed data based on non-volatile storage, specifically, comprising the following steps:
[0062] 1) Obtaining compressed text data and an analysis application based on the compressed text data;
[0063] 2) Based on the N-TADOC persistence strategy, the compressed text data is initialized to construct a non-volatile storage DAG data structure, and the preset traversal strategy is used to traverse the non-volatile storage DAG data structure. The traversal results are statistically summarized according to the analysis application to obtain the text analysis results.
[0064] Furthermore, in the above step 1), the compressed text data obtained in this embodiment is compressed text data represented by a context-free grammar (CFG), and the analysis application on this compressed text data may be word count or the like.
[0065] Furthermore, in the above step 2), in order to fully utilize the persistence characteristics of the non-volatile memory, the present invention develops N-TADOC persistence costs at the operation level and the stage level to meet different requirements.
[0066] For the operation-level persistence cost, the present invention utilizes the persistent memory development kit (PMDK) libpmemobj-cpp, which implements the transaction and log mechanism to ensure the atomicity of the operation. However, it should be noted that the transaction and log mechanism may cause significant write amplification problems, resulting in the degradation of N-TADOC data analysis efficiency. To solve this problem, the present invention proposes another persistence strategy, called stage-level persistence cost.
[0067] For the stage-level persistence cost, the present invention uses libpmem to directly map the space on NVM to memory and explicitly flushes it back to NVM at the end of each N-TADOC stage. Since data is only persisted at the end of each stage, the persistence cost is greatly reduced.
[0068] Specifically, Figure 2 As shown, the following steps are included:
[0069] 2.1) Use N-TADOC to map the non-volatile storage space to the memory, initialize the compressed text data, and build the DAG data structure of non-volatile storage;
[0070] 2.2) Determine whether a fault occurs during step 2.1). If a fault occurs, N-TADOC returns to the previous checkpoint and reinitializes. Otherwise, proceed to step 2.3);
[0071] 2.3) Refresh the constructed DAG data structure back to non-volatile storage and proceed to step 2.4) to traverse the graph;
[0072] 2.4) Using the preset traversal strategy to traverse the DAG data structure in non-volatile storage, obtain the text analysis results, and use N-TADOC to persist the text analysis results to non-volatile storage;
[0073] 2.5) Determine whether a failure occurred during step 2.4). If a failure occurred, return to the previous checkpoint and traverse the graph again.
[0074] The data produced in different stages is always isolated from the persistent data of the previous stage, which is only read in the next stage. This means that when a failure occurs in the current stage, the recovery process can directly overwrite the dirty data. This requires identifying and recovering to the previous checkpoint, discarding the intermediate results of the current stage. Therefore, the recovery process needs to recalculate some parts of the analysis, which may result in longer recovery time.
[0075] Furthermore, in the above step 2.1), the present invention proposes a technology for directly processing compressed data in non-volatile storage, and a data structure for allocating space on a memory pool, which is intended to effectively process compressed data in non-volatile storage. The most important data structure in the traditional technology for directly processing compressed data is a hash table, which can be used to store results locally or globally. The same data structure is also used in the technology for directly processing compressed data in non-volatile storage proposed by the present invention.
[0076] like Figure 3 As shown, when constructing a data structure for non-volatile storage, the following steps are included:
[0077] 2.1.1) A pruning method based on non-volatile memory pool management is used to eliminate some redundancy in the compressed text data, and a DAG pool is established on the non-volatile storage to obtain a DAG graph and its metadata;
[0078] 2.1.2) Based on the metadata, an upper bound on the length of the variable data structure on the non-volatile memory is estimated using a bottom-up summation method;
[0079] 2.1.3) Allocate space for the data object on the non-volatile storage based on the estimation result to obtain the DAG data structure of the non-volatile storage.
[0080] Furthermore, in the above step 2.1.1), the present invention directly calculates the compressed data, and needs to manage the DAG structure based on syntax compression on the non-volatile memory pool. The present invention redesigns the generated DAG structure by adjusting the compressed text data stored in the NVM. The structure proposed by the present invention can maintain good data locality during the DAG traversal process, thereby reducing the redundant reading and writing of the NVM caused by frequent cache misses and improving the efficiency of data analysis.
[0081] In order to illustrate the concepts of "rule", "sub-rule" and "word" that appear below, the present invention is described by an example:
[0082] Rule 1: text->sub-rule 1sub-rule 2word 1word 2
[0083] Rule 2: Sub-rule 1 -> word 3
[0084] Rule 3: Sub-rule 2 -> word 4 sub-rule 3
[0085] Rule 4: Sub-rule 3 -> word 5
[0086] In this example, rule 1 represents the entire compressed text, which consists of sub-rule 1, sub-rule 2, and two words, word 1 and word 2. Sub-rule 1 consists of a word 3, while sub-rule 2 consists of a word 4 and another sub-rule 3, which contains only one word.
[0087] After observing the DAG structure based on grammatical compression, the present invention found that there are two key results in the structure: one is that the generated rules may contain many repeated sub-rules, and the other is that the internal structure of the rules can be random, and the arrangement of sub-rules and words is disordered. This repetitive pattern is not further utilized, and the chaotic structure leads to DAG traversal that is not suitable for caching. In order to solve this problem, the present invention introduces a pruning method that can prune the grammatical representation of text analysis task rules. Specifically, it includes the following steps:
[0088] ① Traverse the rules to obtain the sub-rules and words contained in the rule and the sub-rules and word frequencies;
[0089] ②Write all sub-rules and their frequencies into the DAG pool;
[0090] ③Write all words and their frequencies into the DAG pool;
[0091] ④ Repeat steps ① to ③ to arrange the pruned representations of different rules in the compressed text data (i.e., sub-rules and their frequencies, words and their frequencies) adjacently in the DAG pool to maximize the use of the large read / write granularity on the NVM device;
[0092] ⑤ Design metadata for rules in the DAG pool, including the location of sub-rules and words (the offset where the rules are stored in the DAG pool), in / out degree (the number of in / out edges of the rules in the DAG representation of the compressed file), word list size (the number of different word types included in the rules), and rule weight (the frequency or number of occurrences of the rules in the compressed text data). The design of metadata enables efficient retrieval and traversal of the modified DAG graph during graph traversal.
[0093] ⑥The DAG pool also contains a traversal queue for recording the progress of the task during the top-down traversal and a frequency counter for recording the frequency of words or sequences according to the requirements of the task.
[0094] By using the traversal queue, you can take out the rule being traversed and add its sub-rules to the queue. For example, when you take rule R1 out of the queue, you can add its sub-rules R2, R3, and R4 to the queue. By using the frequency counter, you can record the frequency of words or sequences according to the requirements of the task.
[0095] The present invention uses the following example to demonstrate the pruning process of NVM pool management.
[0096] Consider the original grammar “R1→R2 w3 R4 w4 R3 R2 R4 w4”. After executing the pruning method of non-volatile memory pool management, R1 is pruned to a simplified version “R1→R2×2R3 R4×2w3 w4×2”.
[0097] It is worth noting that the representation of R1 is arranged adjacent to the next rule R2 in the NVM pool. This arrangement eliminates up to 50.2% of the grammatical redundancy on NVM. Such an arrangement not only saves NVM space, but also reduces internal fragmentation, making it easier to traverse the DAG to achieve good data locality. Overall, the pruning method effectively prunes the grammatical representation of the rules, making it more efficient for performing text analysis tasks.
[0098] Further, in the above step 2.1.2), the traditional compressed data direct processing technology traverses the DAG structure in a top-down or bottom-up manner for different data sets. In the top-down method, the weights of the nodes are propagated from top to bottom in topological order. In the bottom-up method, the nodes pass the word list from bottom to top in reverse topological order. In order to adapt to the growing word list in the bottom-up traversal, the compressed data direct processing technology uses a variable length data structure, such as unordered_map. On NVM, when the capacity of the variable length data structure is insufficient, the cost of reconstruction is very high. Therefore, the present invention designs a method capable of estimating the length of the data structure based on the bottom-up traversal method of the compressed data direct processing technology, which is called the "bottom-up summation" method. The bottom-up summation method is closely related to the bottom-up traversal method, in which the nodes pass the word list from bottom to top in reverse topological order.
[0099] like Figure 4 As shown, the method of the present invention focuses on estimating the upper limit of the word list length of each rule in a lightweight traversal from the bottom to the top of the DAG, including the following steps:
[0100] ① Randomly select a rule and determine whether the current rule contains sub-rules. If it does not contain any sub-rules, set the upper limit of the current rule to be equal to the length of its word list; if it contains sub-rules, proceed to step ②;
[0101] ② Randomly select a sub-rule to traverse from bottom to top, and when the upper limit of the current sub-rule is determined, set the sub-rule to "determined";
[0102] In this embodiment, once the upper limit of the word list length is determined, the rule is defined as "determined";
[0103] ③ Determine whether the current rule contains any sub-rules that have not yet been determined. If so, repeat step ② until all sub-rules in the current rule are "determined";
[0104] ④ Based on the upper limits of all determined sub-rules, get the upper limit of the current rule and set the current rule to "determined";
[0105] In this embodiment, the upper limit of the current rule is set to the sum of the upper limits of all sub-rules within it plus the length of the original word list of the current rule;
[0106] ⑤ Repeat steps ① to ④ to get the upper limit of all rules.
[0107] The present invention allocates space for the word list according to the upper limit of the word list, which helps to eliminate the overhead of redundant read and write operations during the NVM data structure reconstruction process. This process improves the efficiency of the data structure reconstruction process and reduces the access latency of the NVM because it avoids unnecessary read and write operations that may have a negative impact on overall performance.
[0108] The present invention uses the following example to demonstrate the bottom-up summation process.
[0109] like Figure 5 As shown, in order to obtain the upper limit of the word list length of R0, it is necessary to recursively determine the upper limits of R1 and R2 respectively. R2 does not contain any sub-rules, so the upper limit of R2 is equal to the length of its word list, which is 2. R1 only contains two words, the sub-rule R2 and itself, so its upper limit is 2+2, which is 4. Finally, through this bottom-up method, the upper limit of R0 is obtained, which is 6(4+2).
[0110] This embodiment was tested on several common compressed text data sets, including novel coronavirus infection data, NSF research award abstract data, and Wikipedia data. The test content includes six text analysis algorithms: vocabulary statistics, sorting, word vector, inverted index, sequence counting, and sequence inverted index. The experimental results show that on four real data sets, compared with directly processing uncompressed data, the present invention achieves a performance acceleration of 2.04 times, and saves 70.7% of dynamic random access memory space compared with the direct processing technology of original compressed data.
[0111] Example 2
[0112] The above-mentioned embodiment 1 provides a method for direct calculation of compressed data based on non-volatile storage. Correspondingly, this embodiment provides a system for direct calculation of compressed data based on non-volatile storage. The system provided in this embodiment can implement the method for direct calculation of compressed data based on non-volatile storage of embodiment 1, and the system can be implemented by software, hardware, or a combination of software and hardware. For example, the system may include integrated or separate functional modules or functional units to execute the corresponding steps in each method of embodiment 1. Since the system of this embodiment is basically similar to the method embodiment, the process described in this embodiment is relatively simple, and the relevant parts can refer to the partial description of embodiment 1. The embodiment of the system provided in this embodiment is only illustrative.
[0113] The compressed data direct computing system based on non-volatile storage provided in this embodiment includes:
[0114] A data acquisition module, used to acquire compressed text data and an analysis application based on the compressed text data;
[0115] The data processing module is used to initialize the compressed text data to build a non-volatile storage DAG data structure based on the N-TADOC persistence strategy, and to traverse the non-volatile storage DAG data structure using a preset traversal strategy, and to statistically summarize the traversal results according to the analysis application to obtain the text analysis results.
[0116] Example 3
[0117] This embodiment provides a processing device corresponding to the method for direct calculation of compressed data based on non-volatile storage provided in this embodiment 1. The processing device can be a processing device for a client, such as a mobile phone, a laptop computer, a tablet computer, a desktop computer, etc., to execute the method of embodiment 1.
[0118] The processing device includes a processor, a memory, a communication interface and a bus, and the processor, the memory and the communication interface are connected through the bus to complete mutual communication. The memory stores a computer program that can be run on the processor, and the processor executes the compressed data direct calculation method based on non-volatile storage provided in this embodiment 1 when running the computer program.
[0119] Preferably, the memory may be a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory.
[0120] Preferably, the processor may be a central processing unit (CPU), a digital signal processor (DSP) or other general-purpose processors of various types, which are not limited here.
[0121] Example 4
[0122] The direct calculation method of compressed data based on non-volatile storage of this embodiment 1 can be specifically implemented as a computer program product, which may include a computer-readable storage medium carrying computer-readable program instructions for executing the direct calculation method of compressed data based on non-volatile storage described in this embodiment 1.
[0123] Computer readable storage media can be tangible devices that hold and store instructions used by instruction execution devices. Computer readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any combination thereof.
[0124] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0125] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0126] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0127] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.
Claims
1. A method for direct computation of compressed data based on non-volatile storage, It is characterized in that The following steps are involved: 1) Obtaining compressed text data and an analysis application based on the compressed text data; 2) Based on the N-TADOC persistence strategy, the compressed text data is initialized to construct a non-volatile storage DAG data structure, and the DAG data structure of non-volatile storage is traversed using a preset traversal strategy. The traversal results are statistically summarized according to the analysis application to obtain the text analysis results; wherein, the N-TADOC persistence strategy is a stage-level persistence cost technology improved based on the operation-level persistence cost technology; In the step 2), based on the N-TADOC persistence strategy, the compressed text data is initialized to construct a non-volatile storage DAG data structure, and the non-volatile storage DAG data structure is traversed using a preset traversal strategy. The traversal results are statistically summarized according to the analysis application to obtain the text analysis results, including: 2.1) Use N-TADOC to map the non-volatile storage space to the memory, initialize the compressed text data, and build the DAG data structure of the non-volatile storage; 2.2) Determine whether a fault occurs during step 2.1). If a fault occurs, N-TADOC returns to the previous checkpoint and reinitializes. Otherwise, proceed to step 2.3); 2.3) Refresh the constructed DAG data structure back to non-volatile storage and proceed to step 2.4) to traverse the graph; 2.4) Use the preset traversal strategy to traverse the DAG data structure in non-volatile storage, summarize the traversal results according to the analysis application, obtain the text analysis results, and use N-TADOC to persist the text analysis results to non-volatile storage; 2.5) Determine whether a failure occurred during step 2.4). If a failure occurred, return to the previous checkpoint and traverse the graph again; In the step 2.1), the compressed text data is initialized to construct a DAG data structure for non-volatile storage, including: 2.1.1) A pruning method based on non-volatile memory pool management is used to eliminate some redundancy in the compressed text data, and a DAG pool is established on the non-volatile storage to obtain the DAG graph and its metadata; 2.1.2) Based on the metadata, an upper bound on the length of variable data structures on non-volatile memory is estimated using a bottom-up summation method; 2.1.3) Based on the estimation result, space is allocated for the data object on the non-volatile storage to obtain the DAG data structure of the non-volatile storage.
2. A method for direct calculation of compressed data based on non-volatile storage as claimed in claim 1, It is characterized in that The pruning method based on non-volatile memory pool management eliminates part of the redundancy in the compressed text data, and establishes a DAG pool on the non-volatile storage to obtain a DAG graph and its metadata, including: ① Traverse the rules to obtain the sub-rules and words contained in the rule and the sub-rules and word frequencies; ②Write all sub-rules and their frequencies into the DAG pool; ③Write all words and their frequencies into the DAG pool; ④ Repeat steps ① to ③ to arrange the pruning representations of different rules in the compressed text data adjacently in the DAG pool; ⑤ Design metadata for rules in the DAG pool; ⑥ Initialize the traversal queue and frequency counter in the DAG pool. The traversal queue is used to record the progress of the task during the top-down traversal process, and the frequency counter is used to record the word or sequence frequency according to the requirements of the task.
3. A method for direct calculation of compressed data based on non-volatile storage as claimed in claim 2, It is characterized in that In step ⑤, the metadata includes: The position of the sub-rule and word, which is used to indicate the offset where the rule is stored in the DAG pool; The out / in degree is used to represent the number of out / in edges of the rule in the DAG representation of the compressed text data; The word list size, which indicates the number of different word types included in the rule; and the weight of the rule, which is used to represent the frequency or number of occurrences of the rule in the compressed text data.
4. A method for direct calculation of compressed data based on non-volatile storage as claimed in claim 1, It is characterized in that In the step 2.1.2), based on the metadata, an upper limit of the length of the variable data structure on the non-volatile memory is estimated using a bottom-up summation method, including: ① Randomly select a rule and determine whether the current rule contains sub-rules. If it does not contain any sub-rules, set the upper limit of the current rule to be equal to the length of its word list; if it contains sub-rules, proceed to step ②; ② Randomly select a sub-rule to traverse from bottom to top, and when the upper limit of the current sub-rule is determined, set the sub-rule to "determined"; ③ Determine whether the current rule contains any sub-rules that have not yet been determined. If so, repeat step ② until all sub-rules in the current rule are "determined"; ④ Based on the upper limits of all determined sub-rules, get the upper limit of the current rule and set the current rule to "determined"; ⑤ Repeat steps ① to ④ to get the upper limit of all rules.
5. A method for direct calculation of compressed data based on non-volatile storage as claimed in claim 4, It is characterized in that In step ④, the upper limit of the current rule is set to the sum of the upper limits of all sub-rules within it plus the length of the original word list of the current rule.
6. A direct computing system for compressed data based on non-volatile storage, It is characterized in that include: A data acquisition module, used to acquire compressed text data and an analysis application based on the compressed text data; The data computing module is used to initialize the compressed text data to construct a non-volatile storage DAG data structure based on the N-TADOC persistence strategy, and traverse the non-volatile storage DAG data structure using a preset traversal strategy, and statistically summarize the traversal results according to the analysis application to obtain the text analysis results; Among them, the N-TADOC persistence strategy is a stage-level persistence cost technology improved based on the operation-level persistence cost technology; Based on the N-TADOC persistence strategy, the compressed text data is initialized to construct a non-volatile storage DAG data structure, and the non-volatile storage DAG data structure is traversed using a preset traversal strategy. The traversal results are statistically summarized according to the analysis application to obtain text analysis results, including: a. Use N-TADOC to map the non-volatile storage space to the memory, initialize the compressed text data, and build the DAG data structure of non-volatile storage; b. Determine whether a fault occurs during step a. If a fault occurs, N-TADOC returns to the previous checkpoint and reinitializes. Otherwise, proceed to step c. c. Refresh the constructed DAG data structure back to non-volatile storage and proceed to step d to traverse the graph; d. Use the preset traversal strategy to traverse the DAG data structure in non-volatile storage, summarize the traversal results according to the analysis application, obtain the text analysis results, and use N-TADOC to persist the text analysis results to non-volatile storage; e. Determine whether a failure occurs during step d. If a failure occurs, return to the previous checkpoint and traverse the graph again. In the step a, the compressed text data is initialized to construct a DAG data structure for non-volatile storage, including: a.
1. A pruning method based on non-volatile memory pool management is used to eliminate some redundancy in the compressed text data, and a DAG pool is established on the non-volatile storage to obtain a DAG graph and its metadata; a.
2. Based on metadata, use a bottom-up summation method to estimate the upper limit of the length of variable data structures on non-volatile memory; a.
3. Allocate space for the data object on the non-volatile storage based on the estimation result to obtain the DAG data structure of the non-volatile storage.
7. A computer-readable storage medium storing one or more programs, It is characterized in that The one or more programs include instructions, which, when executed by a computing device, cause the computing device to perform any one of the methods of claims 1 to 5.
8. A computing device, It is characterized in that include: One or more processors and a memory, wherein the memory stores one or more programs and is configured to be executed by the one or more processors, wherein the one or more programs include instructions for executing any one of the methods described in claims 1 to 5.
